All posts

Fix ECS secret startup errors

Fix ECS ResourceInitializationError during secret injection by checking the failed task's revision, execution role, secret reference and network path.

Identify which startup dependency failed#

ResourceInitializationError covers several initialization failures. The phrase unable to pull secrets or registry auth can involve Secrets Manager, Systems Manager Parameter Store or image-registry authentication. Read the rest of the message before choosing a fix. AWS lists the error categories and their causes.

Signal in the stopped-task reasonStart investigating here
Secrets Manager request times outNetwork path from the task to Secrets Manager
Access denied while fetching a secretExecution-role permissions and applicable resource policies
Secret or version cannot be foundRegion, secret ARN and version selection
Parameter Store is namedParameter reference, Systems Manager access and its network path
Registry authentication is namedECR or the configured private-registry credentials

This guide follows Secrets Manager injection through a task definition's secrets field. If the application starts and later calls Secrets Manager itself, investigate that application's credentials and request separately.

1. Capture the failed task and its exact revision#

Use an AWS CLI profile authorized to inspect the affected account. Replace every uppercase placeholder below with the affected task's values. These commands read metadata; none retrieves a secret value.

aws ecs describe-tasks \
  --profile PROFILE \
  --region REGION \
  --cluster CLUSTER \
  --tasks TASK_ARN \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,platform:platformVersion,reason:stoppedReason,containers:containers[].{name:name,reason:reason,exitCode:exitCode},attachments:attachments}' \
  --output json \
  --no-cli-pager

Record the stopped reason, task definition ARN and task network attachment. Inspect a recent failure promptly: AWS says stopped tasks remain available through this API for at least one hour.

Use the returned task definition ARN in the next command. A family name without a revision can resolve to a newer definition than the task that failed.

aws ecs describe-task-definition \
  --profile PROFILE \
  --region REGION \
  --task-definition TASK_DEFINITION_ARN \
  --query 'taskDefinition.{definition:taskDefinitionArn,executionRole:executionRoleArn,taskRole:taskRoleArn,containers:containerDefinitions[].{name:name,secrets:secrets}}' \
  --output json \
  --no-cli-pager

The task-definition API accepts an exact revision or full ARN. The projected output above keeps literal environment values out of the result. Secret names and ARNs can still identify internal systems, so redact them before sharing an incident publicly.

2. Check the execution role used for injection#

For a task-definition secret, ECS retrieves the value using executionRoleArn. The taskRoleArn gives credentials to application code running inside the container. Granting the application role access does not grant the ECS agent access. AWS documents the execution role and application task role separately.

Inspect the execution role's policies against the secret referenced by the failed revision. Secrets Manager injection needs secretsmanager:GetSecretValue. If the secret uses a customer-managed KMS key, it also needs the required kms:Decrypt access. AWS describes these additional execution-role permissions.

Check the KMS key and secret access policies and any applicable permissions boundary or organization restriction when an allow on the role is insufficient. An explicit denial needs to be resolved at the layer that applies it. IAM's policy evaluation rules explain how those limits interact.

Scope a correction to the intended secret and key. A broad policy change obscures which permission was missing and can grant access the service does not need. If the error names Parameter Store, use its documented execution-role permissions, including ssm:GetParameters, for that branch of the investigation.

3. Verify the secret reference without printing its value#

Read the valueFrom reference in the failed task definition. Confirm the account, Region and secret ARN, then inspect the secret's metadata. Supply the base secret ARN here, excluding any ECS JSON-key or version suffix.

aws secretsmanager describe-secret \
  --profile PROFILE \
  --region REGION \
  --secret-id SECRET_ARN \
  --query '{arn:ARN,key:KmsKeyId,deleted:DeletedDate,versions:VersionIdsToStages}' \
  --output json \
  --no-cli-pager

describe-secret reads metadata without returning the secret value. Success uses your CLI identity, so it establishes existence and metadata visible to you, not the execution role's ability to retrieve or decrypt the value.

Check any JSON-key, version-stage or version-ID suffix separately. Without a version selector, ECS uses AWSCURRENT; an explicit selector can keep pointing to another version after rotation. A JSON-key reference requires that key in the selected secret version. For Linux Fargate tasks, injecting the full secret requires platform 1.3.0 or later; selecting a JSON key or version requires 1.4.0 or later. AWS specifies the selector syntax and platform requirements.

4. Follow the network path from the task#

For a timeout, inspect the failed task's subnet and network configuration. A successful Secrets Manager request from a laptop or another service does not establish connectivity from that task.

A private subnet can reach Secrets Manager through an appropriate outbound route or a Secrets Manager interface VPC endpoint. For the endpoint path, check private DNS, the endpoint's security groups and the endpoint policy. The service name is com.amazonaws.REGION.secretsmanager. An ECS API endpoint serves a different API. AWS explains Secrets Manager endpoints and their policies.

Inspect outbound HTTPS from the task, inbound HTTPS on the interface endpoint's security group, and network ACL rules for the request and return traffic. Check subnet route tables for the path you intend to use. The dependency named in the error determines which endpoint matters; fetching an image and injecting a secret can require different services. Follow AWS's stopped-task connectivity procedure for the affected task.

A worked example: the new role can read the secret, but ECS cannot#

In this illustrative example, an API deployment changes both its task definition and IAM policies. The application task role has access to the new database secret. New tasks still stop before the API starts.

The failed task points to revision 42. That revision references the new secret but retains an execution role whose policy allows only the previous secret. The stopped reason reports an access denial during secret retrieval. The application role's new grant does not authorize ECS's startup request.

The correction belongs in the execution role used by revision 42, scoped to the new secret and its encryption key where required. A reviewer can then inspect the task-definition reference and permission change together. If the stopped reason instead reports a timeout, that evidence points to a different investigation; changing the role alone would leave the network path unresolved.

Verify the correction on replacement tasks#

After the responsible operator applies the correction through the team's deployment process, inspect the replacement task and its exact revision. Check that it passes initialization, reaches the expected container state and satisfies the service's health checks. For a secret rotation, include a newly started task in the check: an existing container keeps the value injected at startup. AWS documents that injection behavior.

Keep the failed task, the correction and the replacement task together in the incident record. These observations establish what changed and what the new task did. A successful startup still needs the application's ordinary functional checks.

Bring the startup dependency into code review#

Review a secret change together with its task-definition reference, execution-role policy, encryption key requirements and network configuration. This makes the startup dependency visible before the next rollout. The same habit helps with connection limits during an ECS deployment, where independently valid settings can fail when combined.

Proofline's GitHub code review can help your team inspect changes and act on supported findings. You can start a hosted review without a cloud connection. That code review is available separately from AWS environment assessment, which is currently disabled; connecting AWS inventory does not activate it. Check current product availability before expecting an automated environment check of this failure.

Questions#

Which IAM role needs access to an ECS startup secret?

For a secret injected through the task definition, check the execution role. For an AWS SDK call made by the application after startup, check the application task role and the credentials that call actually uses.

Why does the error remain after I add Secrets Manager permissions?

Confirm that you changed the execution role used by the failed task's exact revision. Then check the secret and KMS policies and the task's network path. A request timeout can remain even when permissions allow retrieval.

Does rotating a secret update a running ECS container?

A secret injected as an environment variable is read at container startup. Existing containers do not automatically receive a rotated value; verify the intended version on newly started tasks.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

More from the blog