---
title: "Why do 502 and 503 errors spike during every ECS deployment? | Proofline"
description: "ELB 5xx errors that cluster as tasks register or drain, and recur on a same-image redeploy, come from task turnover: draining, registration or new connections."
url: https://proofline.sh/learn/alb-5xx-during-ecs-deployments/
---

[Learning center](/learn/)

# Why do 502 and 503 errors spike during every ECS deployment?

ELB 5xx errors that cluster as tasks register or drain, and recur on a same-image redeploy, come from task turnover: draining, registration or new connections. Compare HTTPCode\_ELB\_5XX\_Count with HTTPCode\_Target\_5XX\_Count, line both up with task start and stop times, and check earlier deployments before rolling back.

Network outages The Proofline team November 20, 2026 8 min read

## Two symptoms that repeat at every deploy

Some error spikes belong to the act of deploying. Two patterns are common on ECS services behind an Application Load Balancer:

-   **A burst of 502s while the old tasks drain.** `HTTPCode_ELB_502_Count` jumps to a few dozen in the two minutes after the new tasks become healthy, then returns to zero. `HTTPCode_Target_5XX_Count` stays flat. The bursts started after someone shortened the target group's deregistration delay.
-   **A minute of 503s on a small service.** A service with one task, `minimumHealthyPercent` at 0 and `maximumPercent` at 100 returns `HTTPCode_ELB_503_Count` errors between the old task stopping and the new task passing its first health check.

Each looks like a bad release in an incident channel, because it starts at the deploy. The checks below tell the rollout's mechanics apart from the release and from a dependency.

## Check 1: did the load balancer or the task produce the error?

`HTTPCode_ELB_5XX_Count` counts 5xx responses that originate from the load balancer and excludes any response a target generated; `HTTPCode_ELB_502_Count`, `HTTPCode_ELB_503_Count` and `HTTPCode_ELB_504_Count` split it by code. `HTTPCode_Target_5XX_Count` counts 5xx responses your tasks returned. The `ELB` metrics have only the `LoadBalancer` and `AvailabilityZone` dimensions; the target metric can be read per target group. [Application Load Balancer metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html)

```
for METRIC in HTTPCode_ELB_502_Count HTTPCode_ELB_503_Count HTTPCode_ELB_504_Count; do
  echo "$METRIC"
  aws cloudwatch get-metric-statistics \
    --profile PROFILE --region REGION --namespace AWS/ApplicationELB \
    --metric-name "$METRIC" --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID \
    --start-time START_TIME --end-time END_TIME --period 60 --statistics Sum \
    --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text --no-cli-pager
done

aws cloudwatch get-metric-statistics \
  --profile PROFILE --region REGION --namespace AWS/ApplicationELB \
  --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID Name=TargetGroup,Value=targetgroup/TG_NAME/TG_ID \
  --start-time START_TIME --end-time END_TIME --period 60 --statistics Sum \
  --query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text --no-cli-pager
```

A release that breaks a code path usually shows up as target 5xx from the new tasks for as long as they serve. Rollout mechanics usually show up as load balancer 5xx for a minute or two. Each load balancer code has a short list of causes, and several of them are rollout events. [Application Load Balancer troubleshooting](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html) · [Deregistration delay](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-target-group-attributes.html)

| Code | Causes that a rollout can trigger |
| --- | --- |
| 502 | A request was still running on a deregistered target when the deregistration delay elapsed. The target closed a connection while the load balancer had a request outstanding, for example because its keep-alive timeout is shorter than the load balancer's idle timeout. |
| 503 | The target group had no registered targets, or all of them were unused. |
| 504 | The target accepted the connection but did not respond within the load balancer's idle timeout, or no connection was established within 10 seconds. |

## Check 2: line the errors up with task starts and stops

ECS takes a service task out of a target group in a fixed order. During `DEACTIVATING` it deregisters the target, and the load balancer stops sending it new requests and waits for in-flight requests, for up to the deregistration delay of 300 seconds by default. If the target has no in-flight requests and no active connections, deregistration completes at once. Only then, during `STOPPING`, does the container get `SIGTERM` (or the image's `STOPSIGNAL`), followed by `SIGKILL` after `stopTimeout`, which defaults to 30 seconds. [Task lifecycle](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-lifecycle-explanation.html) · [Container stopTimeout](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html)

Two consequences follow. A short deregistration delay cuts off long requests: AWS suggests 5 seconds for services whose responses take under a second, and warns against it for slow uploads and streaming. And if a deregistering target closes its connections before the delay elapses, clients receive 500-level errors. [Connection draining parameters for ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-connection-draining.html)

On the way in, ECS registers the target while the task is activating, and the load balancer sends it requests after it passes one health check. A health endpoint that answers before the application has warmed its caches or pools lets the first requests reach a task that cannot serve them yet. Slow start, which ramps a new target's share of traffic, is off by default. [Health checks for target groups](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html) · [Health check parameters for ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-healthcheck.html)

To line these up, take `startedAt`, `stoppingAt` and `stoppedAt` for the old and new tasks with the `describe-tasks` query in [Find when the rollout started serving](/learn/when-did-the-rollout-start-serving/). For per-request detail, the access logs record `elb_status_code` and `target_status_code` for each request with the target's address. `target_status_code` is `-` when the target sent no response, and `target_processing_time` is `-1` when the target closed the connection or did not respond before the idle timeout. Matching those addresses to the old tasks' IPs shows whether the 502s hit draining tasks. [Access log fields](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html)

## Check 3: do the errors come back without a code change?

Start with history, which costs nothing. ECS keeps 90 days of service deployments with each one's `startedAt` and `finishedAt`. Pull the load balancer 5xx for each of those windows. If deployments that changed only configuration, or only a dependency version, show the same burst, the burst is likely the rollout's, which supports clearing the release but does not prove it: confirm in the access logs that today's errors land on draining or registering targets, since a release defect can arrive in the same minutes. [Service deployment history](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-deployment.html)

```
aws ecs list-service-deployments \
  --profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
  --output json --no-cli-pager
```

For a controlled test, force a new deployment of the task definition the service already runs. This command changes the service: it replaces every task, so run it deliberately and at a time when the test traffic resembles the incident's. [update-service](https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html)

```
aws ecs update-service \
  --profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
  --force-new-deployment --no-cli-pager
```

A forced deployment resolves the task definition's image tag again, so a tag such as `latest` can start a different image. Compare the image digests of the tasks before and after (`describe-tasks` returns each container's `imageDigest`) to confirm the code did not change. [Task container fields](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Container.html) · [Diff the two deployments](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

If the same burst appears with the same digests and the same configuration, and the access logs put the errors on draining or registering targets' addresses, that supports the rollout's mechanics. It does not exclude a dependency: the redeploy also starts tasks with fresh connection pools, so a dependency that refuses new connections produces errors on the new tasks at the same minutes. Errors the load balancer generates (`HTTPCode_ELB_5XX_Count`, `target_status_code` `-`) on old tasks point to draining; target 5xx on new tasks point back to the tasks or what they call.

## A worked fixture: the same burst, two causes

The fixture is illustrative. Service `checkout` runs 4 tasks behind an Application Load Balancer, with a deregistration delay of 5 seconds and a report export that can run for 40 seconds. A deployment from `checkout:41` to `checkout:42` starts at 14:00 UTC.

**Case A, rollout mechanics.** Per-minute sums:

| Minute | ELB 502 | Target 5xx | Tasks |
| --- | --- | --- | --- |
| 14:02 | 0 | 0 | 4 `checkout:42` tasks start and register |
| 14:03 | 0 | 0 | New targets healthy |
| 14:04 | 17 | 0 | `checkout:41` targets deregister |
| 14:05 | 9 | 0 | `checkout:41` tasks stop |
| 14:06 | 0 | 0 | Steady state |

Every 502 in the access logs went to an address of a `checkout:41` task, on an export request, with `target_status_code` `-`. A forced redeploy of `checkout:42` at 16:30 produced 14 and 6 in the same two minutes. The deregistration delay is shorter than the longest request; raise it above the export's duration, and keep the release.

**Case B, the release.** Same deployment, same minutes:

| Minute | ELB 502 | Target 5xx | Tasks |
| --- | --- | --- | --- |
| 14:03 | 0 | 22 | New targets healthy |
| 14:04 | 0 | 38 | `checkout:41` targets deregister |
| 14:08 | 0 | 41 | Steady state |

The errors come from the tasks, start when `checkout:42` begins serving and do not fade when the old tasks are gone. At 14:05 a `checkout:41` task opened a fresh connection to the database and ran the failing request's query successfully, so the dependency was accepting new connections and the new tasks' failures are their own. Roll back and compare the task definitions. Had the target 5xx started in the old tasks before the first `checkout:42` task ran any code (its `pullStoppedAt`, 14:02 here), the new code would not have started them. A dependency is then the likely cause, unless the pipeline changed shared configuration before 14:02. Even then, check whether `checkout:42` adds calls or connections to the failing dependency before keeping it, since a rollback still helps when the release amplifies the outage. [Compare both revisions in the same minutes](/learn/bad-release-or-unhealthy-dependency/#split-by-revision) · [Diff the task definitions](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

## Where these checks mislead

`HTTPCode_ELB_5XX_Count` covers the whole load balancer. If other services share it, a burst from their deployments lands in yours; the access logs' `target_group_arn` field separates them.

A keep-alive mismatch produces 502s at any time, not only during deployments. Compare an hour without a deployment before attributing a low, steady 502 rate to rollouts. The load balancer's idle timeout defaults to 60 seconds, and AWS recommends an application idle timeout longer than that. [Load balancer attributes](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-load-balancer-attributes.html)

If every target fails health checks at once, the load balancer fails open and routes to all of them, so a dependency outage that fails every health check shows up as target errors or timeouts rather than 503s. [Health checks that call dependencies](/learn/health-checks-that-call-dependencies/)

A quiet-hour redeploy with no long requests in flight can come out clean even when draining is the cause. A redeploy also starts fresh tasks with new connection pools, so a dependency that rejects new connections can make it fail for reasons unrelated to draining. [Connection budgets during rolling deployments](/learn/connection-limits/)

## Questions

### What are the default deregistration delay and stop timeout for ECS tasks behind an ALB?

The target group's deregistration delay defaults to 300 seconds. A container's `stopTimeout` defaults to 30 seconds; on Fargate the maximum is 120 seconds, and on EC2 an unset value falls back to the agent's `ECS_CONTAINER_STOP_TIMEOUT`.

### Why do I see 502 errors when ECS stops old tasks?

Usually because requests outlive the deregistration delay, or because a draining task closes connections while the load balancer still has requests on them. Access log entries with `target_status_code` `-` on the old tasks' addresses confirm it.

### Does a clean forced redeploy prove the release caused the errors?

No. It shows the burst did not repeat under the test's traffic. Compare earlier deployments over the same windows as well, and read whether the errors came from the load balancer or the tasks.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Find when an ECS deployment or Cloud Run revision started serving](/learn/when-did-the-rollout-start-serving/)
    
    Deployment failures The Proofline team November 20, 2026 9 min read
    
-   ### [How rolling deployments change your PostgreSQL connection budget](/learn/connection-limits/)
    
    Capacity overruns The Proofline team November 5, 2026 8 min read
