Two symptoms#
A rolling deployment of an ECS service behind an Application Load Balancer stalls, and you see one or both of these:
- The service events repeat
(service checkout) (task TASK_ID) (port 8080) is unhealthy in (target-group checkout-tg) due to (reason ...), each time for a task started from the new task definition, and new tasks are stopped and replaced every few minutes. - The target group's
UnHealthyHostCountmetric goes from 0 to 2 at 14:03 UTC, three minutes after the deployment began.
ECS stops tasks that fail load balancer health checks and starts replacements, so the failure keeps repeating until the deployment changes or the circuit breaker rolls it back. The reason code the load balancer recorded for each target tells you what kind of failure it saw. Service unhealthy event messages · Unhealthy task replacement · Why the circuit breaker rolled back
Read target health for old and new tasks together#
Start from the service. It names the target group and the health check grace period, and its deployments list shows the old and new task definitions while both run.
aws ecs describe-services \
--profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
--query 'services[0].{grace:healthCheckGracePeriodSeconds,targetGroups:loadBalancers[].targetGroupArn,deployments:deployments[].{status:status,definition:taskDefinition,created:createdAt}}' \
--output json --no-cli-pager
In awsvpc network mode, which Fargate always uses, each task registers with its own private IP address. Map the running tasks to their task definitions and addresses, then read the target group's view of those addresses. Network mode · describe-tasks · describe-target-health
aws ecs describe-tasks \
--profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 TASK_ID_3 \
--query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,ip:containers[0].networkInterfaces[0].privateIpv4Address}' \
--output table --no-cli-pager
aws elbv2 describe-target-health \
--profile PROFILE --region REGION --target-group-arn TARGET_GROUP_ARN \
--query 'TargetHealthDescriptions[].{target:Target.Id,port:Target.Port,state:TargetHealth.State,reason:TargetHealth.Reason,description:TargetHealth.Description}' \
--output table --no-cli-pager
aws ecs list-tasks --service-name SERVICE gives the task IDs.
On EC2 with bridge networking the target is the instance ID and the host port the task's container is bound to, and tasks on the same instance differ only by that port. Map each task to its instance and host port instead of its address, then match the pairs against Target.Id and Target.Port. NetworkBinding fields · describe-container-instances
aws ecs describe-tasks \
--profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 TASK_ID_3 \
--query 'tasks[].{task:taskArn,definition:taskDefinitionArn,instance:containerInstanceArn,bindings:containers[?name==`CONTAINER`].networkBindings[].{container:containerPort,host:hostPort}}' \
--output json --no-cli-pager
aws ecs describe-container-instances \
--profile PROFILE --region REGION --cluster CLUSTER \
--container-instances CONTAINER_INSTANCE_ARN_1 CONTAINER_INSTANCE_ARN_2 \
--query 'containerInstances[].{arn:containerInstanceArn,ec2:ec2InstanceId}' \
--output table --no-cli-pager
Then read the health check itself, because a failure on every target can be a setting rather than a fault in the tasks. Target group fields
aws elbv2 describe-target-groups \
--profile PROFILE --region REGION --target-group-arns TARGET_GROUP_ARN \
--query 'TargetGroups[0].{path:HealthCheckPath,port:HealthCheckPort,protocol:HealthCheckProtocol,timeout:HealthCheckTimeoutSeconds,interval:HealthCheckIntervalSeconds,unhealthyAfter:UnhealthyThresholdCount,codes:Matcher.HttpCode}' \
--output table --no-cli-pager
For ip targets the defaults are a 5-second timeout, a 30-second interval, two consecutive failures to mark a target unhealthy, and success code 200. Health check settings
What each reason code records#
Codes that begin with Elb come from the load balancer and codes that begin with Target come from the target side. The descriptions below are the ones the API and console return. Health check reason codes · TargetHealth reference
| Reason code | Description | What it tells you |
|---|---|---|
Elb.RegistrationInProgress | Target registration is in progress | Not a failure yet |
Elb.InitialHealthChecking | Initial health checks in progress | Not a failure yet; a new target needs one passing check to receive traffic |
Target.Timeout | Request timed out | No answer within HealthCheckTimeoutSeconds: blocked traffic, a process not yet listening, or a handler slower than the timeout |
Target.ResponseCodeMismatch | Health checks failed with these codes: [code] | The application answered with a status outside the matcher |
Target.FailedHealthChecks | Health checks failed | The load balancer could not connect or got a malformed response |
Elb.InternalError | Health checks failed due to an internal error | A load balancer-side failure |
A timeout has two very different causes. A security group or network ACL that drops the load balancer's traffic and a /health handler that waits 8 seconds on a database both produce Target.Timeout. The AWS troubleshooting procedure separates them by connecting to the target's private IP on the health check port from inside the VPC. A registered target is not in service
The check that splits release from dependency#
The old tasks are registered in the same target group, checked by the same load balancer nodes, and call the same dependencies. Compare their reason codes with the new tasks' codes.
| New targets | Old targets | Most likely cause | Next check |
|---|---|---|---|
Target.ResponseCodeMismatch with 404 | Healthy | The release moved or removed the health path | Compare the path with the new code's routes |
Target.ResponseCodeMismatch with 5xx | Healthy | The new release's health handler fails on something the old one does not check, or on new configuration | Read the new tasks' logs for the health requests |
Target.Timeout or Target.FailedHealthChecks | Healthy | The new deployment's network path, a changed container port, or a slow start | Compare the deployments' subnets and security groups; check the grace period |
| Any failure | Failing too | A dependency the health endpoint calls, or the health check settings or load balancer path | Read the health endpoint's logs and the target group's settings |
For a failure on new targets only, compare the network configuration of the two deployments. A new security group that the load balancer's group cannot reach gives Target.Timeout on new tasks while old tasks stay healthy. Deployment fields · Timeouts only from new tasks
aws ecs describe-services \
--profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
--query 'services[0].deployments[].{definition:taskDefinition,subnets:networkConfiguration.awsvpcConfiguration.subnets,securityGroups:networkConfiguration.awsvpcConfiguration.securityGroups}' \
--output json --no-cli-pager
The load balancer sends health checks with the User-Agent header ELB-HealthChecker/2.0. If the application logs request headers, this Logs Insights query separates health check requests per task, which shows whether the handler answered, how long it took and what it called. Health check request attributes · Logs Insights filter
fields @timestamp, @logStream, @message
| filter @message like /ELB-HealthChecker/
| sort @timestamp desc
| limit 50
A worked fixture: same symptom, two causes#
This fixture is illustrative. A fictional service checkout runs task definition checkout:41; the deployment of checkout:42 started at 14:00 UTC. The target group checkout-tg checks /health on port 8080 with the default 5-second timeout. Both readings were taken at 14:06.
In the release case, the old targets are healthy and the new ones answer 503:
| Target | Task definition | State | Reason | Description |
|---|---|---|---|---|
| 10.0.1.17:8080 | checkout:41 | healthy | ||
| 10.0.2.45:8080 | checkout:41 | healthy | ||
| 10.0.1.88:8080 | checkout:42 | unhealthy | Target.ResponseCodeMismatch | Health checks failed with these codes: [503] |
| 10.0.2.91:8080 | checkout:42 | unhealthy | Target.ResponseCodeMismatch | Health checks failed with these codes: [503] |
The new tasks are up and answering. Their health handler returns 503 while the old handler, against the same database, returns 200. In this fixture checkout:42 added a check of a new cache endpoint that only its configuration names. Let the deployment roll back, then diff the task definitions. Diff the revisions
In the dependency case, every target times out, including tasks that had been healthy for days:
| Target | Task definition | State | Reason | Description |
|---|---|---|---|---|
| 10.0.1.17:8080 | checkout:41 | unhealthy | Target.Timeout | Request timed out |
| 10.0.2.45:8080 | checkout:41 | unhealthy | Target.Timeout | Request timed out |
| 10.0.1.88:8080 | checkout:42 | unhealthy | Target.Timeout | Request timed out |
| 10.0.2.91:8080 | checkout:42 | initial | Elb.InitialHealthChecking | Initial health checks in progress |
UnHealthyHostCount first rose at 13:57, before the deployment, and the health requests in the old tasks' logs wait on a database query for longer than 5 seconds. The release is not what changed. From 13:45 to 14:06 checkout served about 3,000 requests and sent about 9,000 queries a minute, and its connections, rows and time per query held steady in two pg_stat_statements snapshots either side of 14:00. The pipeline ran no migration and changed no security group, role or secret the old tasks use. checkout:42 is not adding to the database's load. A rollback would start more tasks that fail the same check against the same database. With every target unhealthy, the load balancer routes to all of them anyway. Health checks that call a dependency
Where the reading stops#
describe-target-health is a snapshot of the targets registered now. ECS stops failing tasks and the load balancer deregisters them, so a task you missed has left the output. The service keeps its 100 most recent events, and the UnHealthyHostCount metric keeps the count over time without the reason. Health check logs, if enabled before the incident, record each check's time, latency, status code and reason per target, delivered to S3 about every 5 minutes. They are off by default. describe-services events · Health check logs · ALB metrics
The reason code is the load balancer's view. It cannot tell a dropped packet from a slow handler, and a deep health check can return 503 because of a dependency or because of a bug in the new handler. Read the health endpoint's own logs before choosing.
healthCheckGracePeriodSeconds changes what ECS acts on, not what the load balancer records. For that many seconds after a task first starts, ECS ignores failing load balancer and container health checks; the default is 0. A grace period shorter than the application's start time gets slow-starting tasks replaced before they could pass, which looks like a failed release. A longer grace period does not fix a 503 from a dependency. healthCheckGracePeriodSeconds
ECS also replaces tasks whose container health check fails, which the target group does not show. If targets are healthy and tasks still stop, read the task's healthStatus and the containers' health. For the general method of comparing revisions in the same minutes, see the collection's guide. Container health checks
Questions#
What does Target.ResponseCodeMismatch mean on new ECS tasks?
The load balancer reached the task and the application answered, with a status code outside the target group's success codes. The description lists the codes it received. A 404 usually means the health path changed; a 5xx means the health handler itself reported a failure, often from a dependency it checks.
Should I raise healthCheckGracePeriodSeconds when new tasks fail health checks?
Only if the tasks need longer to start than the grace period allows and would pass afterwards. If the reason is a response-code mismatch from a dependency check, or old tasks fail the same check, a longer grace period only delays the replacement.
Why does the load balancer still send traffic to unhealthy tasks?
When every registered target in a target group is unhealthy, an Application Load Balancer fails open and routes requests to all of them. ECS keeps replacing the tasks that fail health checks while that happens.