Two symptoms of a deployment that never finishes#
A rolling deployment stays IN_PROGRESS until the service reaches a steady state. Without a deployment circuit breaker or deployment alarms, nothing marks it FAILED, so a broken rollout can sit there for hours. It usually looks like one of these:
- The new deployment has no tasks at all. Forty minutes after the update,
describe-servicesshows thePRIMARYdeployment withrunningCount0 andpendingCount0, while theACTIVEdeployment still runs four tasks. A service event reports that the servicewas unable to place a task. Reason: Capacity is unavailable at this time. Please try again later or in a different availability zone. - The new deployment's tasks keep coming and going. Its
runningCountrises to 1 or 2 and drops back every minute or two, and the list of stopped tasks grows.
The first is a placement problem and the second is a crash loop. They have different causes and different owners, and the check that separates them takes a minute. ECS Deployment fields · Service event messages
Why ECS keeps trying#
When tasks repeatedly fail to move from PENDING to RUNNING, the scheduler retries indefinitely and stretches the interval between attempts, up to 27 minutes, and reports is unable to consistently start tasks successfully. Tasks that stop after reaching RUNNING, including those that fail load balancer health checks, are replaced without that slowdown. Updating the service to a new task definition resets the throttle. Service throttle logic
A circuit breaker does not always end it either. By default it counts only consecutive failures and resets whenever a task becomes healthy, so a deployment where some tasks succeed can stay in progress. With deployment alarms, the state stays IN_PROGRESS through the bake time even after the new tasks are healthy. If the circuit breaker did fire, see why the circuit breaker rolled back. Circuit breaker failure threshold · Deployment alarms and bake time
The check: placed or never placed#
Read both deployments and the recent events. The event list keeps the latest 100 events and omits a repeated message until its cause is resolved or six hours pass, so a placement failure may appear only once, near the start of the deployment. Viewing service events
aws ecs describe-services \
--profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
--query 'services[0].{deployments:deployments[].{id:id,status:status,definition:taskDefinition,created:createdAt,rollout:rolloutState,reason:rolloutStateReason,running:runningCount,pending:pendingCount,failed:failedTasks,subnets:networkConfiguration.awsvpcConfiguration.subnets},events:events[].{at:createdAt,message:message}}' \
--output json --no-cli-pager
Then list the service's tasks, stopped and current, and describe them. list-tasks returns tasks whose desired status is RUNNING by default, which includes tasks still PENDING. A service task's startedBy holds its deployment ID, lastStatus gives its current lifecycle state, and startedAt is set only once the task reached RUNNING. ECS Task fields
aws ecs list-tasks \
--profile PROFILE --region REGION --cluster CLUSTER \
--service-name SERVICE --desired-status STOPPED \
--query 'taskArns' --output json --no-cli-pager
aws ecs list-tasks \
--profile PROFILE --region REGION --cluster CLUSTER \
--service-name SERVICE \
--query 'taskArns' --output json --no-cli-pager
aws ecs describe-tasks \
--profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ARN_1 TASK_ARN_2 \
--query 'tasks[].{deployment:startedBy,definition:taskDefinitionArn,status:lastStatus,created:createdAt,started:startedAt,stopped:stoppedAt,code:stopCode,reason:stoppedReason,exits:containers[].exitCode}' \
--output json --no-cli-pager
| What you find for the new deployment | What it means | Where to look |
|---|---|---|
"unable to place a task" events; no stopped or current task carries its ID; the deployment's runningCount and pendingCount are 0; it started less than an hour ago | Never placed | Capacity, quotas, placement constraints |
| "unable to place a task" events, but the deployment is older than an hour and no task carries its ID | Inconclusive: stopped tasks may have expired | The deployment's failedTasks, the events' full history and the application logs |
Tasks with its ID, no startedAt, stopped during startup | Placed, failed before running | Subnet addresses, image pull, secrets |
Tasks with startedAt, then EssentialContainerExited or a failed health check | Started, then stopped | The release or a dependency used at startup |
An event about minimumHealthyPercent or maximumPercent | The deployment settings allow no task to start or stop | The service's deployment configuration |
If tasks are never placed#
Placement failures come from the cluster, not from the code. The event text names the constraint: no container instance with enough CPU, memory, free ports or network interface attachment points; Fargate capacity unavailable in the subnets' zones; the account's concurrent task or vCPU quota; or no container instances in the capacity provider. For the Fargate capacity message, AWS suggests waiting or giving the service additional subnets.
A release can still cause a placement failure by asking for more. Compare the CPU, memory, port mappings and placement constraints of the two task definitions: a larger task may fit nowhere the old one did, or push the service over its vCPU quota. Diff the task definitions
Each task in awsvpc mode gets its own network interface and private IP address in one of the service's subnets. With the default maximumPercent of 200, a service of four tasks can start four new tasks before stopping the old ones, so the subnets need that many free addresses beside the running tasks. A Fargate task in a subnet with public IP assignment that runs out of addresses stops with TaskFailedToStart and a message about creating the network interface. Service definition parameters · TaskFailedToStart errors
aws ec2 describe-subnets \
--profile PROFILE --region REGION --subnet-ids SUBNET_ID_1 SUBNET_ID_2 \
--query 'Subnets[].{subnet:SubnetId,zone:AvailabilityZone,free:AvailableIpAddressCount}' \
--output table --no-cli-pager
AvailableIpAddressCount is the number of unused private IPv4 addresses in the subnet. Other workloads in the same subnets consume them too, which is why an unchanged release can stop fitting. Subnet fields
If tasks start and stop#
A crash loop is either the new code or something it needs at startup, and the old deployment's tasks separate the two. They run against the same database, secrets and network right now. Search the application logs for the new tasks' last error, per task, over the last hour; with the awslogs driver each task writes to its own stream. awslogs stream names · Logs Insights parse
fields @timestamp, @logStream, @message
| filter strcontains(@message, "ERROR_TEXT")
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as matches, min(@timestamp) as first_seen, max(@timestamp) as last_seen by task_id
| sort first_seen asc
If only tasks of the new task definition log the error, while old tasks keep using the same dependency without it, check that an old task can still open a fresh connection to the dependency. Old tasks may be living on connections they opened before the dependency stopped accepting new ones, and then a rollback starts tasks that fail the same way. If the old task's fresh connection succeeds and it can run the operation the new tasks fail on at startup, such as the same query, the release is the likely cause: a changed endpoint, credential reference, timeout or code path. An old task still holds the secret values injected when it started, so first check that no secret it reads changed since then; otherwise new tasks of either definition may read a rotated credential. A secret rotated during the rollout If the old tasks log it too, or an old task that ECS replaced in the same window failed the same way, the dependency was failing for everyone, and the new tasks are the ones that must reach it to start. Check when the old tasks' errors began: before the deployment, the dependency was already failing, though new tasks can still add to its load, so compare the connections and attempts each task definition makes on it before ruling out a rollback; only after new tasks started, the new tasks may have overloaded it, for example with a larger connection pool. Did our deploy overload a dependency? If the stopped reason names OutOfMemoryError instead, the tasks ran out of memory. Timeouts only from new tasks · Containers killed for memory after a deploy
A worked fixture: three stuck deployments#
The fixture is illustrative. Service checkout runs 4 tasks of checkout:41 on Fargate with no circuit breaker. At 14:00 UTC a deployment of checkout:42 starts and is still IN_PROGRESS at 14:40.
| Observation at 14:40 | Case A | Case B | Case C |
|---|---|---|---|
| Events | One capacity message at 14:00:20 | None about placement | None about placement |
checkout:42 tasks stopped | None | 11, each with startedAt set | 11, each with startedAt set |
| How they stopped | EssentialContainerExited, exit 1 | EssentialContainerExited, exit 1 | |
Last log line of a checkout:42 task | Missing setting PAYMENTS_URL | Database connection refused | |
checkout:41 tasks | 4 running, no errors | 4 running, no errors | 4 running, logging database errors since 13:51 |
Case A never placed a task: both subnets are in one Availability Zone. The old tasks serve normally, so nothing is broken for users yet; adding a subnet in another zone lets the deployment proceed. Case B is the release: the new tasks fail on a setting the old ones never read. Roll the service back to checkout:41 and fix the task definition. Case C started failing before the deployment, in the old tasks. The database's connections from checkout held at the old tasks' 40 from 13:45 to 14:40. Held connections cannot show refused attempts, so count those too: each checkout:42 task logs one refused attempt and exits, so the 11 starts in 40 minutes added about one attempt every four minutes beside the old tasks' steady retries. The crashing tasks add no load worth stopping. Deploying either task definition again will loop the same way until the database accepts connections. Had connections or attempts climbed with each checkout:42 start, the release would be adding to the outage, and rolling back would cut that load. Compare both revisions in the same minutes
Where the check misleads#
The old tasks do not repeat startup. A dependency used only at startup, such as a secret fetched by ECS or a registry for the image pull, can fail for new tasks while old tasks look healthy. For those failures, read the stopped reason instead of the logs. ECS ResourceInitializationError when pulling secrets
Stopped tasks stay visible to describe-tasks for at least an hour, so in a deployment stuck for longer, the earliest failures are gone. The latest ones usually fail the same way, but check the first ones' logs if the cause might have changed. describe-tasks
With minimumHealthyPercent below 100, ECS can stop old tasks before new ones run, and a stuck deployment then leaves the service short of capacity. Check the ACTIVE deployment's runningCount before treating a stuck deployment as harmless.
Questions#
Why doesn't ECS fail a deployment whose tasks keep crashing?
Without a circuit breaker or deployment alarms, nothing marks a rolling deployment FAILED. The scheduler keeps replacing tasks, slowing down only for tasks that never reach RUNNING.
How do I tell whether ECS ever placed a task for the new deployment?
Look for stopped, pending or running tasks whose startedBy matches the new deployment's ID, and read the deployment's runningCount, pendingCount and failedTasks. If there are none, the deployment started within the last hour, and the service events report that ECS was unable to place a task, the deployment never got capacity. For an older deployment, stopped tasks may have expired, so the absence of tasks does not settle it; read failedTasks and the logs.
Is the service down while its deployment is stuck?
Not necessarily. With minimumHealthyPercent at 100, the old deployment's tasks keep serving until new ones are healthy. Its runningCount in describe-services shows how many still run.