Learning center

New ECS tasks fail load balancer health checks: read the reason code

Read describe-target-health for the new tasks and the old ones side by side. A failure on new targets only points at the release or the network the new deployment uses, a response-code mismatch means the application answered, and old targets failing too points at a shared dependency or the health check itself.

Two symptoms#

A rolling deployment of an ECS service behind an Application Load Balancer stalls, and you see one or both of these:

  • The service events repeat (service checkout) (task TASK_ID) (port 8080) is unhealthy in (target-group checkout-tg) due to (reason ...), each time for a task started from the new task definition, and new tasks are stopped and replaced every few minutes.
  • The target group's UnHealthyHostCount metric goes from 0 to 2 at 14:03 UTC, three minutes after the deployment began.

ECS stops tasks that fail load balancer health checks and starts replacements, so the failure keeps repeating until the deployment changes or the circuit breaker rolls it back. The reason code the load balancer recorded for each target tells you what kind of failure it saw. Service unhealthy event messages · Unhealthy task replacement · Why the circuit breaker rolled back

Read target health for old and new tasks together#

Start from the service. It names the target group and the health check grace period, and its deployments list shows the old and new task definitions while both run.

aws ecs describe-services \
  --profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
  --query 'services[0].{grace:healthCheckGracePeriodSeconds,targetGroups:loadBalancers[].targetGroupArn,deployments:deployments[].{status:status,definition:taskDefinition,created:createdAt}}' \
  --output json --no-cli-pager

In awsvpc network mode, which Fargate always uses, each task registers with its own private IP address. Map the running tasks to their task definitions and addresses, then read the target group's view of those addresses. Network mode · describe-tasks · describe-target-health

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 TASK_ID_3 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,ip:containers[0].networkInterfaces[0].privateIpv4Address}' \
  --output table --no-cli-pager

aws elbv2 describe-target-health \
  --profile PROFILE --region REGION --target-group-arn TARGET_GROUP_ARN \
  --query 'TargetHealthDescriptions[].{target:Target.Id,port:Target.Port,state:TargetHealth.State,reason:TargetHealth.Reason,description:TargetHealth.Description}' \
  --output table --no-cli-pager

aws ecs list-tasks --service-name SERVICE gives the task IDs.

On EC2 with bridge networking the target is the instance ID and the host port the task's container is bound to, and tasks on the same instance differ only by that port. Map each task to its instance and host port instead of its address, then match the pairs against Target.Id and Target.Port. NetworkBinding fields · describe-container-instances

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 TASK_ID_3 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,instance:containerInstanceArn,bindings:containers[?name==`CONTAINER`].networkBindings[].{container:containerPort,host:hostPort}}' \
  --output json --no-cli-pager

aws ecs describe-container-instances \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --container-instances CONTAINER_INSTANCE_ARN_1 CONTAINER_INSTANCE_ARN_2 \
  --query 'containerInstances[].{arn:containerInstanceArn,ec2:ec2InstanceId}' \
  --output table --no-cli-pager

Then read the health check itself, because a failure on every target can be a setting rather than a fault in the tasks. Target group fields

aws elbv2 describe-target-groups \
  --profile PROFILE --region REGION --target-group-arns TARGET_GROUP_ARN \
  --query 'TargetGroups[0].{path:HealthCheckPath,port:HealthCheckPort,protocol:HealthCheckProtocol,timeout:HealthCheckTimeoutSeconds,interval:HealthCheckIntervalSeconds,unhealthyAfter:UnhealthyThresholdCount,codes:Matcher.HttpCode}' \
  --output table --no-cli-pager

For ip targets the defaults are a 5-second timeout, a 30-second interval, two consecutive failures to mark a target unhealthy, and success code 200. Health check settings

What each reason code records#

Codes that begin with Elb come from the load balancer and codes that begin with Target come from the target side. The descriptions below are the ones the API and console return. Health check reason codes · TargetHealth reference

Reason codeDescriptionWhat it tells you
Elb.RegistrationInProgressTarget registration is in progressNot a failure yet
Elb.InitialHealthCheckingInitial health checks in progressNot a failure yet; a new target needs one passing check to receive traffic
Target.TimeoutRequest timed outNo answer within HealthCheckTimeoutSeconds: blocked traffic, a process not yet listening, or a handler slower than the timeout
Target.ResponseCodeMismatchHealth checks failed with these codes: [code]The application answered with a status outside the matcher
Target.FailedHealthChecksHealth checks failedThe load balancer could not connect or got a malformed response
Elb.InternalErrorHealth checks failed due to an internal errorA load balancer-side failure

A timeout has two very different causes. A security group or network ACL that drops the load balancer's traffic and a /health handler that waits 8 seconds on a database both produce Target.Timeout. The AWS troubleshooting procedure separates them by connecting to the target's private IP on the health check port from inside the VPC. A registered target is not in service

The check that splits release from dependency#

The old tasks are registered in the same target group, checked by the same load balancer nodes, and call the same dependencies. Compare their reason codes with the new tasks' codes.

New targetsOld targetsMost likely causeNext check
Target.ResponseCodeMismatch with 404HealthyThe release moved or removed the health pathCompare the path with the new code's routes
Target.ResponseCodeMismatch with 5xxHealthyThe new release's health handler fails on something the old one does not check, or on new configurationRead the new tasks' logs for the health requests
Target.Timeout or Target.FailedHealthChecksHealthyThe new deployment's network path, a changed container port, or a slow startCompare the deployments' subnets and security groups; check the grace period
Any failureFailing tooA dependency the health endpoint calls, or the health check settings or load balancer pathRead the health endpoint's logs and the target group's settings

For a failure on new targets only, compare the network configuration of the two deployments. A new security group that the load balancer's group cannot reach gives Target.Timeout on new tasks while old tasks stay healthy. Deployment fields · Timeouts only from new tasks

aws ecs describe-services \
  --profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
  --query 'services[0].deployments[].{definition:taskDefinition,subnets:networkConfiguration.awsvpcConfiguration.subnets,securityGroups:networkConfiguration.awsvpcConfiguration.securityGroups}' \
  --output json --no-cli-pager

The load balancer sends health checks with the User-Agent header ELB-HealthChecker/2.0. If the application logs request headers, this Logs Insights query separates health check requests per task, which shows whether the handler answered, how long it took and what it called. Health check request attributes · Logs Insights filter

fields @timestamp, @logStream, @message
| filter @message like /ELB-HealthChecker/
| sort @timestamp desc
| limit 50

A worked fixture: same symptom, two causes#

This fixture is illustrative. A fictional service checkout runs task definition checkout:41; the deployment of checkout:42 started at 14:00 UTC. The target group checkout-tg checks /health on port 8080 with the default 5-second timeout. Both readings were taken at 14:06.

In the release case, the old targets are healthy and the new ones answer 503:

TargetTask definitionStateReasonDescription
10.0.1.17:8080checkout:41healthy
10.0.2.45:8080checkout:41healthy
10.0.1.88:8080checkout:42unhealthyTarget.ResponseCodeMismatchHealth checks failed with these codes: [503]
10.0.2.91:8080checkout:42unhealthyTarget.ResponseCodeMismatchHealth checks failed with these codes: [503]

The new tasks are up and answering. Their health handler returns 503 while the old handler, against the same database, returns 200. In this fixture checkout:42 added a check of a new cache endpoint that only its configuration names. Let the deployment roll back, then diff the task definitions. Diff the revisions

In the dependency case, every target times out, including tasks that had been healthy for days:

TargetTask definitionStateReasonDescription
10.0.1.17:8080checkout:41unhealthyTarget.TimeoutRequest timed out
10.0.2.45:8080checkout:41unhealthyTarget.TimeoutRequest timed out
10.0.1.88:8080checkout:42unhealthyTarget.TimeoutRequest timed out
10.0.2.91:8080checkout:42initialElb.InitialHealthCheckingInitial health checks in progress

UnHealthyHostCount first rose at 13:57, before the deployment, and the health requests in the old tasks' logs wait on a database query for longer than 5 seconds. The release is not what changed. From 13:45 to 14:06 checkout served about 3,000 requests and sent about 9,000 queries a minute, and its connections, rows and time per query held steady in two pg_stat_statements snapshots either side of 14:00. The pipeline ran no migration and changed no security group, role or secret the old tasks use. checkout:42 is not adding to the database's load. A rollback would start more tasks that fail the same check against the same database. With every target unhealthy, the load balancer routes to all of them anyway. Health checks that call a dependency

Where the reading stops#

describe-target-health is a snapshot of the targets registered now. ECS stops failing tasks and the load balancer deregisters them, so a task you missed has left the output. The service keeps its 100 most recent events, and the UnHealthyHostCount metric keeps the count over time without the reason. Health check logs, if enabled before the incident, record each check's time, latency, status code and reason per target, delivered to S3 about every 5 minutes. They are off by default. describe-services events · Health check logs · ALB metrics

The reason code is the load balancer's view. It cannot tell a dropped packet from a slow handler, and a deep health check can return 503 because of a dependency or because of a bug in the new handler. Read the health endpoint's own logs before choosing.

healthCheckGracePeriodSeconds changes what ECS acts on, not what the load balancer records. For that many seconds after a task first starts, ECS ignores failing load balancer and container health checks; the default is 0. A grace period shorter than the application's start time gets slow-starting tasks replaced before they could pass, which looks like a failed release. A longer grace period does not fix a 503 from a dependency. healthCheckGracePeriodSeconds

ECS also replaces tasks whose container health check fails, which the target group does not show. If targets are healthy and tasks still stop, read the task's healthStatus and the containers' health. For the general method of comparing revisions in the same minutes, see the collection's guide. Container health checks

Questions#

What does Target.ResponseCodeMismatch mean on new ECS tasks?

The load balancer reached the task and the application answered, with a status code outside the target group's success codes. The description lists the codes it received. A 404 usually means the health path changed; a 5xx means the health handler itself reported a failure, often from a dependency it checks.

Should I raise healthCheckGracePeriodSeconds when new tasks fail health checks?

Only if the tasks need longer to start than the grace period allows and would pass afterwards. If the reason is a response-code mismatch from a dependency check, or old tasks fail the same check, a longer grace period only delays the replacement.

Why does the load balancer still send traffic to unhealthy tasks?

When every registered target in a target group is unhealthy, an Application Load Balancer fails open and routes requests to all of them. ECS keeps replacing the tasks that fail health checks while that happens.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading