---
title: "Why the ECS deployment circuit breaker rolled back | Proofline"
description: "Find out which check failed the deployment and how its tasks stopped, then look at the old deployment's tasks over the same minutes."
url: https://proofline.sh/learn/ecs-circuit-breaker-rollback-cause/
---

[Learning center](/learn/)

# Why the ECS deployment circuit breaker rolled back

Find out which check failed the deployment and how its tasks stopped, then look at the old deployment's tasks over the same minutes. If the old tasks also failed health checks or were replaced, a dependency they share was failing. When those failures began before the deployment, the dependency was already unhealthy and the rollback will not clear the errors, unless the new tasks also added load to it; when they began only after the new tasks started, check whether the new tasks overloaded the dependency first. Either way, compare this service's calls and connections per request before and after.

Broken dependencies The Proofline team November 20, 2026 8 min read

## Two symptoms after an automatic rollback

ECS rolls a service back on its own when the deployment circuit breaker or a deployment alarm marks the new deployment `FAILED` and rollback is turned on. Two versions of this show up:

-   **The circuit breaker fired a few minutes into the deployment.** The service events say the deployment failed because tasks failed to start, and EventBridge delivered a `SERVICE_DEPLOYMENT_FAILED` event with the reason `ECS deployment circuit breaker: task failed to start.` followed by a `SERVICE_DEPLOYMENT_IN_PROGRESS` event whose reason begins `ECS deployment circuit breaker: rolling back to deploymentId`.
-   **An alarm rolled back a deployment that looked finished.** All the new tasks were healthy and the old ones gone when an alarm on `HTTPCode_ELB_5XX_Count` went into `ALARM`, and ECS redeployed the previous task definition. The errors continued after the rollback.

Either way, the rollback is a verdict on the new deployment's tasks, reached without knowing why they failed. [Deployment state change events](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_service_deployment_events.html) · [Service event messages](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-event-messages-list.html)

## What the circuit breaker counts

The circuit breaker watches the new deployment's tasks in two stages. Until one of them reaches `RUNNING`, it counts tasks that fail to get there. Once one is running, it counts tasks replaced because an Elastic Load Balancing, AWS Cloud Map or container health check failed. When the count reaches the threshold, the deployment becomes `FAILED`. [How the circuit breaker detects failures](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-circuit-breaker.html)

By default the threshold is half the service's desired count, rounded up and held between 3 and 200, and the count resets whenever a task becomes healthy, so only consecutive failures add up. Both are configurable: `thresholdConfiguration` can set an unbounded percentage or a fixed count, and `resetOnHealthyTask` set to `false` accumulates every failure. Read the threshold the deployment actually used instead of computing it. [DeploymentCircuitBreaker](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DeploymentCircuitBreaker.html)

With rollback on, ECS redeploys the most recent `COMPLETED` deployment. That deployment moves back to `IN_PROGRESS` and cannot be rolled back again until it completes; if no deployment is `COMPLETED`, the circuit breaker launches no new tasks and the deployment stalls.

Two details make the old tasks a useful comparison. Failures among the old tasks do not count: the circuit breaker evaluates the tasks in the deployment being rolled out, and ECS replaces an unhealthy old task from its own service revision, independently of failures in the new one. And a health check that fails because a database is down counts exactly like one that fails because of new code. [Rolling update behavior](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html)

## Step 1: find which mechanism fired

Service deployments record which detector triggered, its failure count and threshold, any alarms that fired, and the rollback. The history covers the last 90 days. [Service deployment history](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-deployment.html) · [ServiceDeployment fields](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceDeployment.html)

```
aws ecs list-service-deployments \
  --profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
  --output json --no-cli-pager

aws ecs describe-service-deployments \
  --profile PROFILE --region REGION \
  --service-deployment-arns SERVICE_DEPLOYMENT_ARN \
  --query 'serviceDeployments[].{status:status,reason:statusReason,started:startedAt,finished:finishedAt,breaker:deploymentCircuitBreaker,alarms:alarms,rollback:rollback,target:targetServiceRevision.arn}' \
  --output json --no-cli-pager
```

A `breaker` with status `TRIGGERED` gives `failureCount` and `threshold`. An `alarms` block with `triggeredAlarmNames` means an alarm failed the deployment. If both are configured, whichever meets its criteria first fails it. [Service deployment circuit breaker fields](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceDeploymentCircuitBreaker.html) · [Service deployment alarm fields](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceDeploymentAlarms.html)

## Step 2: read how the new tasks stopped

List the service's stopped tasks and describe them. `startedBy` holds the deployment ID, which separates the failed deployment's tasks from the old ones. `describe-tasks` returns a stopped task for at least an hour after it stops, so capture these first. [ECS Task fields](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Task.html) · [describe-tasks](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html)

```
aws ecs list-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --service-name SERVICE --desired-status STOPPED \
  --query 'taskArns' --output json --no-cli-pager

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ARN_1 TASK_ARN_2 \
  --query 'tasks[].{deployment:startedBy,definition:taskDefinitionArn,created:createdAt,started:startedAt,stopped:stoppedAt,code:stopCode,reason:stoppedReason,containers:containers[].{name:name,exit:exitCode,reason:reason}}' \
  --output json --no-cli-pager
```

Sort the failed deployment's tasks into three groups:

-   **No `startedAt`.** The task never reached `RUNNING`: an image pull, a secret, a network interface or a volume failed during startup. The stopped reason begins with a category such as `TaskFailedToStart` or `ResourceInitializationError`. For Secrets Manager failures, see [ECS ResourceInitializationError when pulling secrets](/learn/ecs-secrets-startup-errors/). [Stopped task error messages](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/stopped-task-error-codes.html)
-   **`startedAt` set, stop code `EssentialContainerExited`.** The process started and exited. The exit code and the container's last log lines say why.
-   **`startedAt` set, stopped after failing a health check.** The stopped reason says whether the load balancer or the container health check failed. The target's reason code tells a timeout from a wrong response code. [Read the target health reason code](/learn/ecs-target-health-check-reason-codes/)

## Step 3: check the old tasks in the same minutes

The old deployment's tasks kept running against the same database, secrets and network while the new ones failed. Run the same `describe-tasks` query over the stopped tasks whose `startedBy` is the old deployment's ID. Old tasks stopped for failed health checks during the rollout mean something both share was failing: usually a dependency, but first confirm that the target group's health check path, port and matcher did not change with the rollout; the logs below say whether it started before the deployment or after the new tasks arrived. Old tasks that ran untouched through the window point at the release, unless the new tasks failed on a dependency the health check calls: then check that an old task can still open a fresh connection to it, because old tasks can survive on connections they opened earlier. [Timeouts only from new tasks](/learn/dependency-timeouts-only-from-new-tasks/)

Then search the application logs for the error the new tasks hit, by task, starting well before the deployment. With the `awslogs` driver and a stream prefix, each task logs to its own stream. [awslogs stream names](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html) · [Logs Insights `parse`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-Parse.html) · [Logs Insights stats functions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-Stats.html)

```
fields @timestamp, @logStream, @message
| filter strcontains(@message, "ERROR_TEXT")
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as matches, min(@timestamp) as first_seen by task_id
| sort first_seen asc
```

If the old tasks logged the same error before the deployment's `createdAt`, the dependency was already unhealthy. If their first errors came only after new tasks started, the release may have caused them by adding load the dependency could not carry, such as more connections from the extra tasks or a new query; compare this service's calls and connections per request before and after. [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/) If the health check endpoint itself calls that dependency, the new tasks failed their checks for the same reason, and the circuit breaker turned an outage into a failed rollout. [Health checks that call dependencies](/learn/health-checks-that-call-dependencies/)

## A worked fixture: the same rollback, two causes

The fixture is illustrative. Service `checkout` runs 4 tasks of `checkout:41`. At 14:00 UTC a deployment of `checkout:42` starts with the default circuit breaker settings, so the threshold is 3. In both cases the new tasks reach `RUNNING`, fail their load balancer health checks, and the breaker rolls back at about 14:05.

| Observation | Case A | Case B |
| --- | --- | --- |
| `checkout:42` tasks stopped | 3, each after failing health checks | 3, each after failing health checks |
| Target reason code for `checkout:42` | `Target.ResponseCodeMismatch` (404) | `Target.Timeout` |
| `checkout:41` tasks stopped 13:30–14:05 | None | 2, at 13:56 and 14:02, after failing health checks |
| First database timeout in `checkout:41` logs | None | 13:48 |
| After the rollback | Errors stop | `checkout:41` tasks keep failing checks; errors continue |

In case A the old tasks served cleanly the whole time, and the new tasks' health path returned 404. The release changed something the health check depends on, such as a route or a port. Compare the task definitions and the health check path. [Diff the task definitions](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

In case B the old tasks were failing the same checks eight minutes before the deployment, and ECS had already replaced two of them. The rollback restored `checkout:41` but not the database, and the deployment it restored cannot be rolled back again until it completes. Work on the database first; redeploying either task definition will not help until it recovers. [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/)

## Where this check falls short

The old tasks do not repeat startup. If the new tasks failed before `RUNNING`, for example because a rotated secret or an exhausted subnet only affects tasks that start now, the old tasks keep running and look healthy even though the release is not the cause. Compare a fresh task of the old definition instead. [A secret rotated during the rollout](/learn/secret-rotation-during-rollout/) · [Timeouts only from new tasks](/learn/dependency-timeouts-only-from-new-tasks/)

An alarm-based rollback can fire with no old tasks to compare. The bake time starts after all new tasks are healthy and the old revision is scaled to zero; by default it is under five minutes. `HTTPCode_ELB_5XX_Count` has no target group dimension, so an alarm on it also counts errors from other services behind the same load balancer. And if the alarm was already in `ALARM` when the deployment started, ECS ignores alarms for that deployment, so a dependency outage under way at deploy time does not trigger an alarm rollback at all. [How CloudWatch alarms detect deployment failures](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-alarm-failure.html) · [Application Load Balancer metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html)

A rollback that cleared the errors still restarted tasks and moved traffic. [Does a rollback prove the release caused it?](/learn/does-a-rollback-prove-the-release-caused-it/)

## Questions

### How many failed tasks does the ECS circuit breaker need before it rolls back?

By default, half the desired task count rounded up, with a minimum of 3 and a maximum of 200, counting consecutive failures. A service can change both the threshold type and whether a healthy task resets the count, so read `threshold` from `describe-service-deployments`.

### Do failures of the old tasks count toward the circuit breaker?

No. The circuit breaker counts the new deployment's tasks, and ECS replaces unhealthy old tasks from their own revision. That separation is why old tasks failing in the same minutes point at a shared dependency.

### Can the circuit breaker roll back a rollback?

No. The deployment ECS rolled back to moves from `COMPLETED` to `IN_PROGRESS` and is not eligible for another rollback until it completes. A deployment that fails while no other deployment is `COMPLETED` has nothing to roll back to, and stalls. If the restored tasks fail the way the new ones did, look outside both task definitions first.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [New ECS tasks fail load balancer health checks: read the reason code](/learn/ecs-target-health-check-reason-codes/)
    
    Deployment failures The Proofline team November 20, 2026 8 min read
    
-   ### [Health checks that call a dependency turn its outage into a failed rollout](/learn/health-checks-that-call-dependencies/)
    
    Broken dependencies The Proofline team November 20, 2026 8 min read
