---
title: "New ECS tasks fail load balancer health checks: read the reason code | Proofline"
description: "Read describe-target-health for the new tasks and the old ones side by side."
url: https://proofline.sh/learn/ecs-target-health-check-reason-codes/
---

[Learning center](/learn/)

# New ECS tasks fail load balancer health checks: read the reason code

Read describe-target-health for the new tasks and the old ones side by side. A failure on new targets only points at the release or the network the new deployment uses, a response-code mismatch means the application answered, and old targets failing too points at a shared dependency or the health check itself.

Deployment failures The Proofline team November 20, 2026 8 min read

## Two symptoms

A rolling deployment of an ECS service behind an Application Load Balancer stalls, and you see one or both of these:

-   The service events repeat `(service checkout) (task TASK_ID) (port 8080) is unhealthy in (target-group checkout-tg) due to (reason ...)`, each time for a task started from the new task definition, and new tasks are stopped and replaced every few minutes.
-   The target group's `UnHealthyHostCount` metric goes from 0 to 2 at 14:03 UTC, three minutes after the deployment began.

ECS stops tasks that fail load balancer health checks and starts replacements, so the failure keeps repeating until the deployment changes or the circuit breaker rolls it back. The reason code the load balancer recorded for each target tells you what kind of failure it saw. [Service unhealthy event messages](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-unhealthy-event-messages.html) · [Unhealthy task replacement](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html) · [Why the circuit breaker rolled back](/learn/ecs-circuit-breaker-rollback-cause/)

## Read target health for old and new tasks together

Start from the service. It names the target group and the health check grace period, and its `deployments` list shows the old and new task definitions while both run.

```
aws ecs describe-services \
  --profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
  --query 'services[0].{grace:healthCheckGracePeriodSeconds,targetGroups:loadBalancers[].targetGroupArn,deployments:deployments[].{status:status,definition:taskDefinition,created:createdAt}}' \
  --output json --no-cli-pager
```

In `awsvpc` network mode, which Fargate always uses, each task registers with its own private IP address. Map the running tasks to their task definitions and addresses, then read the target group's view of those addresses. [Network mode](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#network_mode) · [describe-tasks](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html) · [describe-target-health](https://docs.aws.amazon.com/cli/latest/reference/elbv2/describe-target-health.html)

```
aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 TASK_ID_3 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,ip:containers[0].networkInterfaces[0].privateIpv4Address}' \
  --output table --no-cli-pager

aws elbv2 describe-target-health \
  --profile PROFILE --region REGION --target-group-arn TARGET_GROUP_ARN \
  --query 'TargetHealthDescriptions[].{target:Target.Id,port:Target.Port,state:TargetHealth.State,reason:TargetHealth.Reason,description:TargetHealth.Description}' \
  --output table --no-cli-pager
```

`aws ecs list-tasks --service-name SERVICE` gives the task IDs.

On EC2 with `bridge` networking the target is the instance ID and the host port the task's container is bound to, and tasks on the same instance differ only by that port. Map each task to its instance and host port instead of its address, then match the pairs against `Target.Id` and `Target.Port`. [NetworkBinding fields](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_NetworkBinding.html) · [describe-container-instances](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-container-instances.html)

```
aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 TASK_ID_3 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,instance:containerInstanceArn,bindings:containers[?name==`CONTAINER`].networkBindings[].{container:containerPort,host:hostPort}}' \
  --output json --no-cli-pager

aws ecs describe-container-instances \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --container-instances CONTAINER_INSTANCE_ARN_1 CONTAINER_INSTANCE_ARN_2 \
  --query 'containerInstances[].{arn:containerInstanceArn,ec2:ec2InstanceId}' \
  --output table --no-cli-pager
```

Then read the health check itself, because a failure on every target can be a setting rather than a fault in the tasks. [Target group fields](https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_TargetGroup.html)

```
aws elbv2 describe-target-groups \
  --profile PROFILE --region REGION --target-group-arns TARGET_GROUP_ARN \
  --query 'TargetGroups[0].{path:HealthCheckPath,port:HealthCheckPort,protocol:HealthCheckProtocol,timeout:HealthCheckTimeoutSeconds,interval:HealthCheckIntervalSeconds,unhealthyAfter:UnhealthyThresholdCount,codes:Matcher.HttpCode}' \
  --output table --no-cli-pager
```

For `ip` targets the defaults are a 5-second timeout, a 30-second interval, two consecutive failures to mark a target unhealthy, and success code 200. [Health check settings](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html)

## What each reason code records

Codes that begin with `Elb` come from the load balancer and codes that begin with `Target` come from the target side. The descriptions below are the ones the API and console return. [Health check reason codes](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html#target-health-reason-codes) · [TargetHealth reference](https://docs.aws.amazon.com/elasticloadbalancing/latest/APIReference/API_TargetHealth.html)

| Reason code | Description | What it tells you |
| --- | --- | --- |
| `Elb.RegistrationInProgress` | Target registration is in progress | Not a failure yet |
| `Elb.InitialHealthChecking` | Initial health checks in progress | Not a failure yet; a new target needs one passing check to receive traffic |
| `Target.Timeout` | Request timed out | No answer within `HealthCheckTimeoutSeconds`: blocked traffic, a process not yet listening, or a handler slower than the timeout |
| `Target.ResponseCodeMismatch` | Health checks failed with these codes: \[code\] | The application answered with a status outside the matcher |
| `Target.FailedHealthChecks` | Health checks failed | The load balancer could not connect or got a malformed response |
| `Elb.InternalError` | Health checks failed due to an internal error | A load balancer-side failure |

A timeout has two very different causes. A security group or network ACL that drops the load balancer's traffic and a `/health` handler that waits 8 seconds on a database both produce `Target.Timeout`. The AWS troubleshooting procedure separates them by connecting to the target's private IP on the health check port from inside the VPC. [A registered target is not in service](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#target-not-inservice)

## The check that splits release from dependency

The old tasks are registered in the same target group, checked by the same load balancer nodes, and call the same dependencies. Compare their reason codes with the new tasks' codes.

| New targets | Old targets | Most likely cause | Next check |
| --- | --- | --- | --- |
| `Target.ResponseCodeMismatch` with 404 | Healthy | The release moved or removed the health path | Compare the path with the new code's routes |
| `Target.ResponseCodeMismatch` with 5xx | Healthy | The new release's health handler fails on something the old one does not check, or on new configuration | Read the new tasks' logs for the health requests |
| `Target.Timeout` or `Target.FailedHealthChecks` | Healthy | The new deployment's network path, a changed container port, or a slow start | Compare the deployments' subnets and security groups; check the grace period |
| Any failure | Failing too | A dependency the health endpoint calls, or the health check settings or load balancer path | Read the health endpoint's logs and the target group's settings |

For a failure on new targets only, compare the network configuration of the two deployments. A new security group that the load balancer's group cannot reach gives `Target.Timeout` on new tasks while old tasks stay healthy. [Deployment fields](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Deployment.html) · [Timeouts only from new tasks](/learn/dependency-timeouts-only-from-new-tasks/)

```
aws ecs describe-services \
  --profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
  --query 'services[0].deployments[].{definition:taskDefinition,subnets:networkConfiguration.awsvpcConfiguration.subnets,securityGroups:networkConfiguration.awsvpcConfiguration.securityGroups}' \
  --output json --no-cli-pager
```

The load balancer sends health checks with the `User-Agent` header `ELB-HealthChecker/2.0`. If the application logs request headers, this Logs Insights query separates health check requests per task, which shows whether the handler answered, how long it took and what it called. [Health check request attributes](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#target-not-inservice) · [Logs Insights filter](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-Filter.html)

```
fields @timestamp, @logStream, @message
| filter @message like /ELB-HealthChecker/
| sort @timestamp desc
| limit 50
```

## A worked fixture: same symptom, two causes

This fixture is illustrative. A fictional service `checkout` runs task definition `checkout:41`; the deployment of `checkout:42` started at 14:00 UTC. The target group `checkout-tg` checks `/health` on port 8080 with the default 5-second timeout. Both readings were taken at 14:06.

In the release case, the old targets are healthy and the new ones answer 503:

| Target | Task definition | State | Reason | Description |
| --- | --- | --- | --- | --- |
| 10.0.1.17:8080 | `checkout:41` | healthy |  |  |
| 10.0.2.45:8080 | `checkout:41` | healthy |  |  |
| 10.0.1.88:8080 | `checkout:42` | unhealthy | `Target.ResponseCodeMismatch` | Health checks failed with these codes: \[503\] |
| 10.0.2.91:8080 | `checkout:42` | unhealthy | `Target.ResponseCodeMismatch` | Health checks failed with these codes: \[503\] |

The new tasks are up and answering. Their health handler returns 503 while the old handler, against the same database, returns 200. In this fixture `checkout:42` added a check of a new cache endpoint that only its configuration names. Let the deployment roll back, then diff the task definitions. [Diff the revisions](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

In the dependency case, every target times out, including tasks that had been healthy for days:

| Target | Task definition | State | Reason | Description |
| --- | --- | --- | --- | --- |
| 10.0.1.17:8080 | `checkout:41` | unhealthy | `Target.Timeout` | Request timed out |
| 10.0.2.45:8080 | `checkout:41` | unhealthy | `Target.Timeout` | Request timed out |
| 10.0.1.88:8080 | `checkout:42` | unhealthy | `Target.Timeout` | Request timed out |
| 10.0.2.91:8080 | `checkout:42` | initial | `Elb.InitialHealthChecking` | Initial health checks in progress |

`UnHealthyHostCount` first rose at 13:57, before the deployment, and the health requests in the old tasks' logs wait on a database query for longer than 5 seconds. The release is not what changed. From 13:45 to 14:06 `checkout` served about 3,000 requests and sent about 9,000 queries a minute, and its connections, rows and time per query held steady in two `pg_stat_statements` snapshots either side of 14:00. The pipeline ran no migration and changed no security group, role or secret the old tasks use. `checkout:42` is not adding to the database's load. A rollback would start more tasks that fail the same check against the same database. With every target unhealthy, the load balancer routes to all of them anyway. [Health checks that call a dependency](/learn/health-checks-that-call-dependencies/)

## Where the reading stops

`describe-target-health` is a snapshot of the targets registered now. ECS stops failing tasks and the load balancer deregisters them, so a task you missed has left the output. The service keeps its 100 most recent events, and the `UnHealthyHostCount` metric keeps the count over time without the reason. Health check logs, if enabled before the incident, record each check's time, latency, status code and reason per target, delivered to S3 about every 5 minutes. They are off by default. [describe-services events](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-services.html) · [Health check logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-health-check-logs.html) · [ALB metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html)

The reason code is the load balancer's view. It cannot tell a dropped packet from a slow handler, and a deep health check can return 503 because of a dependency or because of a bug in the new handler. Read the health endpoint's own logs before choosing.

`healthCheckGracePeriodSeconds` changes what ECS acts on, not what the load balancer records. For that many seconds after a task first starts, ECS ignores failing load balancer and container health checks; the default is 0. A grace period shorter than the application's start time gets slow-starting tasks replaced before they could pass, which looks like a failed release. A longer grace period does not fix a 503 from a dependency. [healthCheckGracePeriodSeconds](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_CreateService.html#ECS-CreateService-request-healthCheckGracePeriodSeconds)

ECS also replaces tasks whose container health check fails, which the target group does not show. If targets are healthy and tasks still stop, read the task's `healthStatus` and the containers' health. For the general method of comparing revisions in the same minutes, see the [collection's guide](/learn/bad-release-or-unhealthy-dependency/#split-by-revision). [Container health checks](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_HealthCheck.html)

## Questions

### What does Target.ResponseCodeMismatch mean on new ECS tasks?

The load balancer reached the task and the application answered, with a status code outside the target group's success codes. The description lists the codes it received. A 404 usually means the health path changed; a 5xx means the health handler itself reported a failure, often from a dependency it checks.

### Should I raise healthCheckGracePeriodSeconds when new tasks fail health checks?

Only if the tasks need longer to start than the grace period allows and would pass afterwards. If the reason is a response-code mismatch from a dependency check, or old tasks fail the same check, a longer grace period only delays the replacement.

### Why does the load balancer still send traffic to unhealthy tasks?

When every registered target in a target group is unhealthy, an Application Load Balancer fails open and routes requests to all of them. ECS keeps replacing the tasks that fail health checks while that happens.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Health checks that call a dependency turn its outage into a failed rollout](/learn/health-checks-that-call-dependencies/)
    
    Broken dependencies The Proofline team November 20, 2026 8 min read
    
-   ### [Dependency timeouts only from new ECS tasks or Cloud Run revisions](/learn/dependency-timeouts-only-from-new-tasks/)
    
    Network outages The Proofline team November 20, 2026 9 min read
