Two symptoms after an automatic rollback#
ECS rolls a service back on its own when the deployment circuit breaker or a deployment alarm marks the new deployment FAILED and rollback is turned on. Two versions of this show up:
- The circuit breaker fired a few minutes into the deployment. The service events say the deployment failed because tasks failed to start, and EventBridge delivered a
SERVICE_DEPLOYMENT_FAILEDevent with the reasonECS deployment circuit breaker: task failed to start.followed by aSERVICE_DEPLOYMENT_IN_PROGRESSevent whose reason beginsECS deployment circuit breaker: rolling back to deploymentId. - An alarm rolled back a deployment that looked finished. All the new tasks were healthy and the old ones gone when an alarm on
HTTPCode_ELB_5XX_Countwent intoALARM, and ECS redeployed the previous task definition. The errors continued after the rollback.
Either way, the rollback is a verdict on the new deployment's tasks, reached without knowing why they failed. Deployment state change events · Service event messages
What the circuit breaker counts#
The circuit breaker watches the new deployment's tasks in two stages. Until one of them reaches RUNNING, it counts tasks that fail to get there. Once one is running, it counts tasks replaced because an Elastic Load Balancing, AWS Cloud Map or container health check failed. When the count reaches the threshold, the deployment becomes FAILED. How the circuit breaker detects failures
By default the threshold is half the service's desired count, rounded up and held between 3 and 200, and the count resets whenever a task becomes healthy, so only consecutive failures add up. Both are configurable: thresholdConfiguration can set an unbounded percentage or a fixed count, and resetOnHealthyTask set to false accumulates every failure. Read the threshold the deployment actually used instead of computing it. DeploymentCircuitBreaker
With rollback on, ECS redeploys the most recent COMPLETED deployment. That deployment moves back to IN_PROGRESS and cannot be rolled back again until it completes; if no deployment is COMPLETED, the circuit breaker launches no new tasks and the deployment stalls.
Two details make the old tasks a useful comparison. Failures among the old tasks do not count: the circuit breaker evaluates the tasks in the deployment being rolled out, and ECS replaces an unhealthy old task from its own service revision, independently of failures in the new one. And a health check that fails because a database is down counts exactly like one that fails because of new code. Rolling update behavior
Step 1: find which mechanism fired#
Service deployments record which detector triggered, its failure count and threshold, any alarms that fired, and the rollback. The history covers the last 90 days. Service deployment history · ServiceDeployment fields
aws ecs list-service-deployments \
--profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
--output json --no-cli-pager
aws ecs describe-service-deployments \
--profile PROFILE --region REGION \
--service-deployment-arns SERVICE_DEPLOYMENT_ARN \
--query 'serviceDeployments[].{status:status,reason:statusReason,started:startedAt,finished:finishedAt,breaker:deploymentCircuitBreaker,alarms:alarms,rollback:rollback,target:targetServiceRevision.arn}' \
--output json --no-cli-pager
A breaker with status TRIGGERED gives failureCount and threshold. An alarms block with triggeredAlarmNames means an alarm failed the deployment. If both are configured, whichever meets its criteria first fails it. Service deployment circuit breaker fields · Service deployment alarm fields
Step 2: read how the new tasks stopped#
List the service's stopped tasks and describe them. startedBy holds the deployment ID, which separates the failed deployment's tasks from the old ones. describe-tasks returns a stopped task for at least an hour after it stops, so capture these first. ECS Task fields · describe-tasks
aws ecs list-tasks \
--profile PROFILE --region REGION --cluster CLUSTER \
--service-name SERVICE --desired-status STOPPED \
--query 'taskArns' --output json --no-cli-pager
aws ecs describe-tasks \
--profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ARN_1 TASK_ARN_2 \
--query 'tasks[].{deployment:startedBy,definition:taskDefinitionArn,created:createdAt,started:startedAt,stopped:stoppedAt,code:stopCode,reason:stoppedReason,containers:containers[].{name:name,exit:exitCode,reason:reason}}' \
--output json --no-cli-pager
Sort the failed deployment's tasks into three groups:
- No
startedAt. The task never reachedRUNNING: an image pull, a secret, a network interface or a volume failed during startup. The stopped reason begins with a category such asTaskFailedToStartorResourceInitializationError. For Secrets Manager failures, see ECS ResourceInitializationError when pulling secrets. Stopped task error messages startedAtset, stop codeEssentialContainerExited. The process started and exited. The exit code and the container's last log lines say why.startedAtset, stopped after failing a health check. The stopped reason says whether the load balancer or the container health check failed. The target's reason code tells a timeout from a wrong response code. Read the target health reason code
Step 3: check the old tasks in the same minutes#
The old deployment's tasks kept running against the same database, secrets and network while the new ones failed. Run the same describe-tasks query over the stopped tasks whose startedBy is the old deployment's ID. Old tasks stopped for failed health checks during the rollout mean something both share was failing: usually a dependency, but first confirm that the target group's health check path, port and matcher did not change with the rollout; the logs below say whether it started before the deployment or after the new tasks arrived. Old tasks that ran untouched through the window point at the release, unless the new tasks failed on a dependency the health check calls: then check that an old task can still open a fresh connection to it, because old tasks can survive on connections they opened earlier. Timeouts only from new tasks
Then search the application logs for the error the new tasks hit, by task, starting well before the deployment. With the awslogs driver and a stream prefix, each task logs to its own stream. awslogs stream names · Logs Insights parse · Logs Insights stats functions
fields @timestamp, @logStream, @message
| filter strcontains(@message, "ERROR_TEXT")
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as matches, min(@timestamp) as first_seen by task_id
| sort first_seen asc
If the old tasks logged the same error before the deployment's createdAt, the dependency was already unhealthy. If their first errors came only after new tasks started, the release may have caused them by adding load the dependency could not carry, such as more connections from the extra tasks or a new query; compare this service's calls and connections per request before and after. Did our deploy overload a dependency? If the health check endpoint itself calls that dependency, the new tasks failed their checks for the same reason, and the circuit breaker turned an outage into a failed rollout. Health checks that call dependencies
A worked fixture: the same rollback, two causes#
The fixture is illustrative. Service checkout runs 4 tasks of checkout:41. At 14:00 UTC a deployment of checkout:42 starts with the default circuit breaker settings, so the threshold is 3. In both cases the new tasks reach RUNNING, fail their load balancer health checks, and the breaker rolls back at about 14:05.
| Observation | Case A | Case B |
|---|---|---|
checkout:42 tasks stopped | 3, each after failing health checks | 3, each after failing health checks |
Target reason code for checkout:42 | Target.ResponseCodeMismatch (404) | Target.Timeout |
checkout:41 tasks stopped 13:30–14:05 | None | 2, at 13:56 and 14:02, after failing health checks |
First database timeout in checkout:41 logs | None | 13:48 |
| After the rollback | Errors stop | checkout:41 tasks keep failing checks; errors continue |
In case A the old tasks served cleanly the whole time, and the new tasks' health path returned 404. The release changed something the health check depends on, such as a route or a port. Compare the task definitions and the health check path. Diff the task definitions
In case B the old tasks were failing the same checks eight minutes before the deployment, and ECS had already replaced two of them. The rollback restored checkout:41 but not the database, and the deployment it restored cannot be rolled back again until it completes. Work on the database first; redeploying either task definition will not help until it recovers. Did our deploy overload a dependency?
Where this check falls short#
The old tasks do not repeat startup. If the new tasks failed before RUNNING, for example because a rotated secret or an exhausted subnet only affects tasks that start now, the old tasks keep running and look healthy even though the release is not the cause. Compare a fresh task of the old definition instead. A secret rotated during the rollout · Timeouts only from new tasks
An alarm-based rollback can fire with no old tasks to compare. The bake time starts after all new tasks are healthy and the old revision is scaled to zero; by default it is under five minutes. HTTPCode_ELB_5XX_Count has no target group dimension, so an alarm on it also counts errors from other services behind the same load balancer. And if the alarm was already in ALARM when the deployment started, ECS ignores alarms for that deployment, so a dependency outage under way at deploy time does not trigger an alarm rollback at all. How CloudWatch alarms detect deployment failures · Application Load Balancer metrics
A rollback that cleared the errors still restarted tasks and moved traffic. Does a rollback prove the release caused it?
Questions#
How many failed tasks does the ECS circuit breaker need before it rolls back?
By default, half the desired task count rounded up, with a minimum of 3 and a maximum of 200, counting consecutive failures. A service can change both the threshold type and whether a healthy task resets the count, so read threshold from describe-service-deployments.
Do failures of the old tasks count toward the circuit breaker?
No. The circuit breaker counts the new deployment's tasks, and ECS replaces unhealthy old tasks from their own revision. That separation is why old tasks failing in the same minutes point at a shared dependency.
Can the circuit breaker roll back a rollback?
No. The deployment ECS rolled back to moves from COMPLETED to IN_PROGRESS and is not eligible for another rollback until it completes. A deployment that fails while no other deployment is COMPLETED has nothing to roll back to, and stalls. If the restored tasks fail the way the new ones did, look outside both task definitions first.