Learning center

An ECS deployment stuck in progress: placement or crashing tasks

First establish whether ECS ever placed a task for the new deployment. Tasks that are never placed leave service events and no task records, and point at capacity, quotas or subnet addresses. Tasks that are placed and then stop leave stopped reasons, and the old deployment's tasks tell you whether the release or a shared dependency stops them.

Two symptoms of a deployment that never finishes#

A rolling deployment stays IN_PROGRESS until the service reaches a steady state. Without a deployment circuit breaker or deployment alarms, nothing marks it FAILED, so a broken rollout can sit there for hours. It usually looks like one of these:

  • The new deployment has no tasks at all. Forty minutes after the update, describe-services shows the PRIMARY deployment with runningCount 0 and pendingCount 0, while the ACTIVE deployment still runs four tasks. A service event reports that the service was unable to place a task. Reason: Capacity is unavailable at this time. Please try again later or in a different availability zone.
  • The new deployment's tasks keep coming and going. Its runningCount rises to 1 or 2 and drops back every minute or two, and the list of stopped tasks grows.

The first is a placement problem and the second is a crash loop. They have different causes and different owners, and the check that separates them takes a minute. ECS Deployment fields · Service event messages

Why ECS keeps trying#

When tasks repeatedly fail to move from PENDING to RUNNING, the scheduler retries indefinitely and stretches the interval between attempts, up to 27 minutes, and reports is unable to consistently start tasks successfully. Tasks that stop after reaching RUNNING, including those that fail load balancer health checks, are replaced without that slowdown. Updating the service to a new task definition resets the throttle. Service throttle logic

A circuit breaker does not always end it either. By default it counts only consecutive failures and resets whenever a task becomes healthy, so a deployment where some tasks succeed can stay in progress. With deployment alarms, the state stays IN_PROGRESS through the bake time even after the new tasks are healthy. If the circuit breaker did fire, see why the circuit breaker rolled back. Circuit breaker failure threshold · Deployment alarms and bake time

The check: placed or never placed#

Read both deployments and the recent events. The event list keeps the latest 100 events and omits a repeated message until its cause is resolved or six hours pass, so a placement failure may appear only once, near the start of the deployment. Viewing service events

aws ecs describe-services \
  --profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
  --query 'services[0].{deployments:deployments[].{id:id,status:status,definition:taskDefinition,created:createdAt,rollout:rolloutState,reason:rolloutStateReason,running:runningCount,pending:pendingCount,failed:failedTasks,subnets:networkConfiguration.awsvpcConfiguration.subnets},events:events[].{at:createdAt,message:message}}' \
  --output json --no-cli-pager

Then list the service's tasks, stopped and current, and describe them. list-tasks returns tasks whose desired status is RUNNING by default, which includes tasks still PENDING. A service task's startedBy holds its deployment ID, lastStatus gives its current lifecycle state, and startedAt is set only once the task reached RUNNING. ECS Task fields

aws ecs list-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --service-name SERVICE --desired-status STOPPED \
  --query 'taskArns' --output json --no-cli-pager

aws ecs list-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --service-name SERVICE \
  --query 'taskArns' --output json --no-cli-pager

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ARN_1 TASK_ARN_2 \
  --query 'tasks[].{deployment:startedBy,definition:taskDefinitionArn,status:lastStatus,created:createdAt,started:startedAt,stopped:stoppedAt,code:stopCode,reason:stoppedReason,exits:containers[].exitCode}' \
  --output json --no-cli-pager
What you find for the new deploymentWhat it meansWhere to look
"unable to place a task" events; no stopped or current task carries its ID; the deployment's runningCount and pendingCount are 0; it started less than an hour agoNever placedCapacity, quotas, placement constraints
"unable to place a task" events, but the deployment is older than an hour and no task carries its IDInconclusive: stopped tasks may have expiredThe deployment's failedTasks, the events' full history and the application logs
Tasks with its ID, no startedAt, stopped during startupPlaced, failed before runningSubnet addresses, image pull, secrets
Tasks with startedAt, then EssentialContainerExited or a failed health checkStarted, then stoppedThe release or a dependency used at startup
An event about minimumHealthyPercent or maximumPercentThe deployment settings allow no task to start or stopThe service's deployment configuration

If tasks are never placed#

Placement failures come from the cluster, not from the code. The event text names the constraint: no container instance with enough CPU, memory, free ports or network interface attachment points; Fargate capacity unavailable in the subnets' zones; the account's concurrent task or vCPU quota; or no container instances in the capacity provider. For the Fargate capacity message, AWS suggests waiting or giving the service additional subnets.

A release can still cause a placement failure by asking for more. Compare the CPU, memory, port mappings and placement constraints of the two task definitions: a larger task may fit nowhere the old one did, or push the service over its vCPU quota. Diff the task definitions

Each task in awsvpc mode gets its own network interface and private IP address in one of the service's subnets. With the default maximumPercent of 200, a service of four tasks can start four new tasks before stopping the old ones, so the subnets need that many free addresses beside the running tasks. A Fargate task in a subnet with public IP assignment that runs out of addresses stops with TaskFailedToStart and a message about creating the network interface. Service definition parameters · TaskFailedToStart errors

aws ec2 describe-subnets \
  --profile PROFILE --region REGION --subnet-ids SUBNET_ID_1 SUBNET_ID_2 \
  --query 'Subnets[].{subnet:SubnetId,zone:AvailabilityZone,free:AvailableIpAddressCount}' \
  --output table --no-cli-pager

AvailableIpAddressCount is the number of unused private IPv4 addresses in the subnet. Other workloads in the same subnets consume them too, which is why an unchanged release can stop fitting. Subnet fields

If tasks start and stop#

A crash loop is either the new code or something it needs at startup, and the old deployment's tasks separate the two. They run against the same database, secrets and network right now. Search the application logs for the new tasks' last error, per task, over the last hour; with the awslogs driver each task writes to its own stream. awslogs stream names · Logs Insights parse

fields @timestamp, @logStream, @message
| filter strcontains(@message, "ERROR_TEXT")
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as matches, min(@timestamp) as first_seen, max(@timestamp) as last_seen by task_id
| sort first_seen asc

If only tasks of the new task definition log the error, while old tasks keep using the same dependency without it, check that an old task can still open a fresh connection to the dependency. Old tasks may be living on connections they opened before the dependency stopped accepting new ones, and then a rollback starts tasks that fail the same way. If the old task's fresh connection succeeds and it can run the operation the new tasks fail on at startup, such as the same query, the release is the likely cause: a changed endpoint, credential reference, timeout or code path. An old task still holds the secret values injected when it started, so first check that no secret it reads changed since then; otherwise new tasks of either definition may read a rotated credential. A secret rotated during the rollout If the old tasks log it too, or an old task that ECS replaced in the same window failed the same way, the dependency was failing for everyone, and the new tasks are the ones that must reach it to start. Check when the old tasks' errors began: before the deployment, the dependency was already failing, though new tasks can still add to its load, so compare the connections and attempts each task definition makes on it before ruling out a rollback; only after new tasks started, the new tasks may have overloaded it, for example with a larger connection pool. Did our deploy overload a dependency? If the stopped reason names OutOfMemoryError instead, the tasks ran out of memory. Timeouts only from new tasks · Containers killed for memory after a deploy

A worked fixture: three stuck deployments#

The fixture is illustrative. Service checkout runs 4 tasks of checkout:41 on Fargate with no circuit breaker. At 14:00 UTC a deployment of checkout:42 starts and is still IN_PROGRESS at 14:40.

Observation at 14:40Case ACase BCase C
EventsOne capacity message at 14:00:20None about placementNone about placement
checkout:42 tasks stoppedNone11, each with startedAt set11, each with startedAt set
How they stoppedEssentialContainerExited, exit 1EssentialContainerExited, exit 1
Last log line of a checkout:42 taskMissing setting PAYMENTS_URLDatabase connection refused
checkout:41 tasks4 running, no errors4 running, no errors4 running, logging database errors since 13:51

Case A never placed a task: both subnets are in one Availability Zone. The old tasks serve normally, so nothing is broken for users yet; adding a subnet in another zone lets the deployment proceed. Case B is the release: the new tasks fail on a setting the old ones never read. Roll the service back to checkout:41 and fix the task definition. Case C started failing before the deployment, in the old tasks. The database's connections from checkout held at the old tasks' 40 from 13:45 to 14:40. Held connections cannot show refused attempts, so count those too: each checkout:42 task logs one refused attempt and exits, so the 11 starts in 40 minutes added about one attempt every four minutes beside the old tasks' steady retries. The crashing tasks add no load worth stopping. Deploying either task definition again will loop the same way until the database accepts connections. Had connections or attempts climbed with each checkout:42 start, the release would be adding to the outage, and rolling back would cut that load. Compare both revisions in the same minutes

Where the check misleads#

The old tasks do not repeat startup. A dependency used only at startup, such as a secret fetched by ECS or a registry for the image pull, can fail for new tasks while old tasks look healthy. For those failures, read the stopped reason instead of the logs. ECS ResourceInitializationError when pulling secrets

Stopped tasks stay visible to describe-tasks for at least an hour, so in a deployment stuck for longer, the earliest failures are gone. The latest ones usually fail the same way, but check the first ones' logs if the cause might have changed. describe-tasks

With minimumHealthyPercent below 100, ECS can stop old tasks before new ones run, and a stuck deployment then leaves the service short of capacity. Check the ACTIVE deployment's runningCount before treating a stuck deployment as harmless.

Questions#

Why doesn't ECS fail a deployment whose tasks keep crashing?

Without a circuit breaker or deployment alarms, nothing marks a rolling deployment FAILED. The scheduler keeps replacing tasks, slowing down only for tasks that never reach RUNNING.

How do I tell whether ECS ever placed a task for the new deployment?

Look for stopped, pending or running tasks whose startedBy matches the new deployment's ID, and read the deployment's runningCount, pendingCount and failedTasks. If there are none, the deployment started within the last hour, and the service events report that ECS was unable to place a task, the deployment never got capacity. For an older deployment, stopped tasks may have expired, so the absence of tasks does not settle it; read failedTasks and the logs.

Is the service down while its deployment is stuck?

Not necessarily. With minimumHealthyPercent at 100, the old deployment's tasks keep serving until new ones are healthy. Its runningCount in describe-services shows how many still run.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading