Two symptoms that repeat at every deploy#
Some error spikes belong to the act of deploying. Two patterns are common on ECS services behind an Application Load Balancer:
- A burst of 502s while the old tasks drain.
HTTPCode_ELB_502_Countjumps to a few dozen in the two minutes after the new tasks become healthy, then returns to zero.HTTPCode_Target_5XX_Countstays flat. The bursts started after someone shortened the target group's deregistration delay. - A minute of 503s on a small service. A service with one task,
minimumHealthyPercentat 0 andmaximumPercentat 100 returnsHTTPCode_ELB_503_Counterrors between the old task stopping and the new task passing its first health check.
Each looks like a bad release in an incident channel, because it starts at the deploy. The checks below tell the rollout's mechanics apart from the release and from a dependency.
Check 1: did the load balancer or the task produce the error?#
HTTPCode_ELB_5XX_Count counts 5xx responses that originate from the load balancer and excludes any response a target generated; HTTPCode_ELB_502_Count, HTTPCode_ELB_503_Count and HTTPCode_ELB_504_Count split it by code. HTTPCode_Target_5XX_Count counts 5xx responses your tasks returned. The ELB metrics have only the LoadBalancer and AvailabilityZone dimensions; the target metric can be read per target group. Application Load Balancer metrics
for METRIC in HTTPCode_ELB_502_Count HTTPCode_ELB_503_Count HTTPCode_ELB_504_Count; do
echo "$METRIC"
aws cloudwatch get-metric-statistics \
--profile PROFILE --region REGION --namespace AWS/ApplicationELB \
--metric-name "$METRIC" --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID \
--start-time START_TIME --end-time END_TIME --period 60 --statistics Sum \
--query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text --no-cli-pager
done
aws cloudwatch get-metric-statistics \
--profile PROFILE --region REGION --namespace AWS/ApplicationELB \
--metric-name HTTPCode_Target_5XX_Count \
--dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID Name=TargetGroup,Value=targetgroup/TG_NAME/TG_ID \
--start-time START_TIME --end-time END_TIME --period 60 --statistics Sum \
--query 'sort_by(Datapoints,&Timestamp)[].[Timestamp,Sum]' --output text --no-cli-pager
A release that breaks a code path usually shows up as target 5xx from the new tasks for as long as they serve. Rollout mechanics usually show up as load balancer 5xx for a minute or two. Each load balancer code has a short list of causes, and several of them are rollout events. Application Load Balancer troubleshooting · Deregistration delay
| Code | Causes that a rollout can trigger |
|---|---|
| 502 | A request was still running on a deregistered target when the deregistration delay elapsed. The target closed a connection while the load balancer had a request outstanding, for example because its keep-alive timeout is shorter than the load balancer's idle timeout. |
| 503 | The target group had no registered targets, or all of them were unused. |
| 504 | The target accepted the connection but did not respond within the load balancer's idle timeout, or no connection was established within 10 seconds. |
Check 2: line the errors up with task starts and stops#
ECS takes a service task out of a target group in a fixed order. During DEACTIVATING it deregisters the target, and the load balancer stops sending it new requests and waits for in-flight requests, for up to the deregistration delay of 300 seconds by default. If the target has no in-flight requests and no active connections, deregistration completes at once. Only then, during STOPPING, does the container get SIGTERM (or the image's STOPSIGNAL), followed by SIGKILL after stopTimeout, which defaults to 30 seconds. Task lifecycle · Container stopTimeout
Two consequences follow. A short deregistration delay cuts off long requests: AWS suggests 5 seconds for services whose responses take under a second, and warns against it for slow uploads and streaming. And if a deregistering target closes its connections before the delay elapses, clients receive 500-level errors. Connection draining parameters for ECS
On the way in, ECS registers the target while the task is activating, and the load balancer sends it requests after it passes one health check. A health endpoint that answers before the application has warmed its caches or pools lets the first requests reach a task that cannot serve them yet. Slow start, which ramps a new target's share of traffic, is off by default. Health checks for target groups · Health check parameters for ECS
To line these up, take startedAt, stoppingAt and stoppedAt for the old and new tasks with the describe-tasks query in Find when the rollout started serving. For per-request detail, the access logs record elb_status_code and target_status_code for each request with the target's address. target_status_code is - when the target sent no response, and target_processing_time is -1 when the target closed the connection or did not respond before the idle timeout. Matching those addresses to the old tasks' IPs shows whether the 502s hit draining tasks. Access log fields
Check 3: do the errors come back without a code change?#
Start with history, which costs nothing. ECS keeps 90 days of service deployments with each one's startedAt and finishedAt. Pull the load balancer 5xx for each of those windows. If deployments that changed only configuration, or only a dependency version, show the same burst, the burst is likely the rollout's, which supports clearing the release but does not prove it: confirm in the access logs that today's errors land on draining or registering targets, since a release defect can arrive in the same minutes. Service deployment history
aws ecs list-service-deployments \
--profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
--output json --no-cli-pager
For a controlled test, force a new deployment of the task definition the service already runs. This command changes the service: it replaces every task, so run it deliberately and at a time when the test traffic resembles the incident's. update-service
aws ecs update-service \
--profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
--force-new-deployment --no-cli-pager
A forced deployment resolves the task definition's image tag again, so a tag such as latest can start a different image. Compare the image digests of the tasks before and after (describe-tasks returns each container's imageDigest) to confirm the code did not change. Task container fields · Diff the two deployments
If the same burst appears with the same digests and the same configuration, and the access logs put the errors on draining or registering targets' addresses, that supports the rollout's mechanics. It does not exclude a dependency: the redeploy also starts tasks with fresh connection pools, so a dependency that refuses new connections produces errors on the new tasks at the same minutes. Errors the load balancer generates (HTTPCode_ELB_5XX_Count, target_status_code -) on old tasks point to draining; target 5xx on new tasks point back to the tasks or what they call.
A worked fixture: the same burst, two causes#
The fixture is illustrative. Service checkout runs 4 tasks behind an Application Load Balancer, with a deregistration delay of 5 seconds and a report export that can run for 40 seconds. A deployment from checkout:41 to checkout:42 starts at 14:00 UTC.
Case A, rollout mechanics. Per-minute sums:
| Minute | ELB 502 | Target 5xx | Tasks |
|---|---|---|---|
| 14:02 | 0 | 0 | 4 checkout:42 tasks start and register |
| 14:03 | 0 | 0 | New targets healthy |
| 14:04 | 17 | 0 | checkout:41 targets deregister |
| 14:05 | 9 | 0 | checkout:41 tasks stop |
| 14:06 | 0 | 0 | Steady state |
Every 502 in the access logs went to an address of a checkout:41 task, on an export request, with target_status_code -. A forced redeploy of checkout:42 at 16:30 produced 14 and 6 in the same two minutes. The deregistration delay is shorter than the longest request; raise it above the export's duration, and keep the release.
Case B, the release. Same deployment, same minutes:
| Minute | ELB 502 | Target 5xx | Tasks |
|---|---|---|---|
| 14:03 | 0 | 22 | New targets healthy |
| 14:04 | 0 | 38 | checkout:41 targets deregister |
| 14:08 | 0 | 41 | Steady state |
The errors come from the tasks, start when checkout:42 begins serving and do not fade when the old tasks are gone. At 14:05 a checkout:41 task opened a fresh connection to the database and ran the failing request's query successfully, so the dependency was accepting new connections and the new tasks' failures are their own. Roll back and compare the task definitions. Had the target 5xx started in the old tasks before the first checkout:42 task ran any code (its pullStoppedAt, 14:02 here), the new code would not have started them. A dependency is then the likely cause, unless the pipeline changed shared configuration before 14:02. Even then, check whether checkout:42 adds calls or connections to the failing dependency before keeping it, since a rollback still helps when the release amplifies the outage. Compare both revisions in the same minutes · Diff the task definitions
Where these checks mislead#
HTTPCode_ELB_5XX_Count covers the whole load balancer. If other services share it, a burst from their deployments lands in yours; the access logs' target_group_arn field separates them.
A keep-alive mismatch produces 502s at any time, not only during deployments. Compare an hour without a deployment before attributing a low, steady 502 rate to rollouts. The load balancer's idle timeout defaults to 60 seconds, and AWS recommends an application idle timeout longer than that. Load balancer attributes
If every target fails health checks at once, the load balancer fails open and routes to all of them, so a dependency outage that fails every health check shows up as target errors or timeouts rather than 503s. Health checks that call dependencies
A quiet-hour redeploy with no long requests in flight can come out clean even when draining is the cause. A redeploy also starts fresh tasks with new connection pools, so a dependency that rejects new connections can make it fail for reasons unrelated to draining. Connection budgets during rolling deployments
Questions#
What are the default deregistration delay and stop timeout for ECS tasks behind an ALB?
The target group's deregistration delay defaults to 300 seconds. A container's stopTimeout defaults to 30 seconds; on Fargate the maximum is 120 seconds, and on EC2 an unset value falls back to the agent's ECS_CONTAINER_STOP_TIMEOUT.
Why do I see 502 errors when ECS stops old tasks?
Usually because requests outlive the deregistration delay, or because a draining task closes connections while the load balancer still has requests on them. Access log entries with target_status_code - on the old tasks' addresses confirm it.
Does a clean forced redeploy prove the release caused the errors?
No. It shows the burst did not repeat under the test's traffic. Compare earlier deployments over the same windows as well, and read whether the errors came from the load balancer or the tasks.