Learning center

What changed between two ECS task definitions or Cloud Run revisions?

Compare the image digest each revision actually ran, then environment variable names, secret references, roles, resources, probes and network settings. Only a difference on the failing dependency's path, such as its endpoint, credentials, timeout, pool size or egress route, can explain a dependency symptom; a diff with none of those leaves the dependency or the new code, which a comparison of the service's load on the dependency before and after separates.

Two symptoms that send you to the diff#

  • An ECS service moved from task definition checkout:41 to checkout:42. The new tasks log connection timed out to the payments API, the old tasks did not, and the pull request touched only a template file.
  • A Cloud Run service shifted 10% of traffic to checkout-00042. Its 500s come with timeouts calling the same payments API, while checkout-00041 keeps serving cleanly, and the deploy pipeline says the configuration did not change.

In both cases the split by revision already points at the new revision. The question left is what the new revision runs or uses differently. A code diff answers part of it; the revision records the rest: which image digest started, what configuration it received, and how it reaches the network. Split errors by revision first

What a revision records, and what it leaves out#

SettingECSCloud Run
Imageimage in each container definition, as written; the digest that ran is in describe-tasks and the service revisionspec.containers[].image as written; status.imageDigest holds the digest resolved when the revision was created
Environmentenvironment, plus environmentFiles stored in S3env[].value
Secret referencessecrets[].valueFromenv[].valueFrom.secretKeyRef (secret name and version), secret volumes
IdentitytaskRoleArn for the application, executionRoleArn for the agentserviceAccountName
CPU and memorycpu, memory on the task and each containerresources.limits
Ports and healthportMappings, container healthCheckports, startupProbe, livenessProbe
NetworkService revision networkConfiguration: subnets, security groups, public IPAnnotations run.googleapis.com/vpc-access-egress, run.googleapis.com/network-interfaces, run.googleapis.com/vpc-access-connector
Concurrency and scaleOutside the task definitioncontainerConcurrency, timeoutSeconds, autoscaling.knative.dev/minScale and maxScale

ECS resolves the image tag in a task definition to a digest when a deployment starts its first task, and a forced new deployment resolves it again, so one task definition can run different images in two deployments if the tag moved. A Cloud Run revision resolves its digest once, when the revision is created. ECS container image resolution · Cloud Run revision imageDigest · Cloud Run YAML reference

Some settings that affect a dependency call live outside both revisions: the ALB target group's health check, the rules inside a security group whose ID did not change, the policies attached to a role whose ARN did not change, Cloud Run ingress and service-level minimum instances on the service's own metadata, and the value behind a secret reference.

Diff two ECS task definitions#

Pass full family:revision identifiers, because a family name alone resolves to the latest active revision. The function drops the fields that always differ, sorts keys and environment entries, and writes one normalized file per revision. describe-task-definition

td() {
  aws ecs describe-task-definition \
    --profile PROFILE --region REGION --task-definition "$1" \
    --query taskDefinition --output json --no-cli-pager |
  jq -S 'del(.taskDefinitionArn, .revision, .registeredAt, .registeredBy)
    | .containerDefinitions |= (sort_by(.name)
        | map(.environment |= (. // [] | sort_by(.name))))'
}
td FAMILY:OLD_REVISION > old.json
td FAMILY:NEW_REVISION > new.json
diff -u old.json new.json

These files contain literal environment values, which often include internal hostnames, account IDs and sometimes credentials that should have been secrets. Keep them on your machine. To share the result in a ticket, list only the names whose values changed:

jq -n --slurpfile a old.json --slurpfile b new.json '
  def env($t): [$t.containerDefinitions[] | .name as $c
    | .environment[] | {key: "\($c)/\(.name)", value}] | from_entries;
  env($a[0]) as $x | env($b[0]) as $y
  | [$x + $y | keys[] | select($x[.] != $y[.])]'

Secret references print as ARNs, never values. environmentFiles prints only the S3 object ARN, so a changed file under the same ARN will not show up. environmentFiles

The task definition does not hold the service's subnets, security groups or load balancer. Service revisions do. ECS records one whenever a service update starts a deployment, and you can describe those created on or after October 25, 2024. List the deployments to find the revision ARNs, then describe both. Service revisions · list-service-deployments · describe-service-revisions

aws ecs list-service-deployments \
  --profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
  --query 'serviceDeployments[].{created:createdAt,status:status,revision:targetServiceRevisionArn}' \
  --output table --no-cli-pager

aws ecs describe-service-revisions \
  --profile PROFILE --region REGION \
  --service-revision-arns OLD_SERVICE_REVISION_ARN NEW_SERVICE_REVISION_ARN \
  --query 'serviceRevisions[].{revision:serviceRevisionArn,definition:taskDefinition,
    images:containerImages,network:networkConfiguration,
    loadBalancers:loadBalancers,platform:platformVersion}' \
  --output json --no-cli-pager

containerImages gives each container's image and the imageDigest ECS resolved for that deployment. To confirm what a particular failing task ran, read its containers' imageDigest with describe-tasks. ServiceRevision fields · Task container fields

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,
    containers:containers[].{name:name,image:image,digest:imageDigest}}' \
  --output json --no-cli-pager

Diff two Cloud Run revisions#

A revision is immutable, so describing it later returns the configuration it served with. --format export writes YAML without status and without automatically generated labels, which keeps the diff to configuration. It also drops status.imageDigest, so read the digests separately. Manage revisions · View service and revision details · gcloud run revisions describe

gcloud run revisions describe OLD_REVISION --project PROJECT --region REGION --format export > old.yaml
gcloud run revisions describe NEW_REVISION --project PROJECT --region REGION --format export > new.yaml
diff -u old.yaml new.yaml

gcloud run revisions describe OLD_REVISION --project PROJECT --region REGION --format 'value(status.imageDigest)'
gcloud run revisions describe NEW_REVISION --project PROJECT --region REGION --format 'value(status.imageDigest)'

The exported YAML also holds literal env values, so treat it like the ECS files. A secret reference shows the secret name and the version in key. If the key is latest, both revisions can show the same reference while their instances hold different values, because Cloud Run resolves secrets passed as environment variables when each instance starts. Cloud Run secrets

Ingress is set by the run.googleapis.com/ingress annotation on the service, outside the revision template, and so is service-level minimum instances. Read them with gcloud run services describe SERVICE --project PROJECT --region REGION --format export. That shows only the current value. Ingress settings

Which differences can explain a dependency symptom#

DifferenceCan it explain timeouts or errors to one dependency?
Dependency endpoint, host or port in environment or envYes; the new revision calls a different place
Client timeout, retry count, pool size, AWS_MAX_ATTEMPTSYes; shorter timeouts fail sooner, and retries and larger pools add load AWS SDK retry settings
Secret reference: different secret, version or JSON keyYes, for authentication failures
Task role, execution role or service accountYes, for permission errors on the dependency; the execution role is used by the ECS agent for image pulls, secret injection and logs, not by application calls Execution role
Subnets, security groups, VPC egress setting or connectorYes, for connection timeouts that start in the new revision only
CPU, memory, containerConcurrency, maxScaleIndirectly: CPU starvation slows calls, and more concurrency or instances multiply connections
Image digestOnly through the code it contains; diff the commits
Health check, port, labels, log configurationRarely; they explain failed rollouts, not dependency errors

A dependency-shaped symptom with none of the "yes" rows in the diff points back at the dependency, at something outside the revision, or at the code itself: a new query or a bypassed cache changes only the image digest. Compare this service's calls and work per request on the dependency before and after before you clear the code. Did our deploy overload a dependency?

A worked fixture: two diffs for the same timeout#

The fixture is illustrative: a fictional ECS service checkout deployed from checkout:41 to checkout:42, with new tasks timing out against the payments API. Digests are shortened.

What the diff showsCase ACase B
Image digest, old → newsha256:3b9e… → sha256:c71d…sha256:3b9e… → sha256:c71d…
Environment names with changed valuesapp/PAYMENTS_TIMEOUT_MSapp/BANNER_TEXT
Secret referencesunchangedunchanged
Task role, execution roleunchangedunchanged
Service revision networkConfigurationsecurityGroups changed: sg-0a1… → sg-0f4…unchanged
Commits between the digeststemplate change onlytemplate change only

In case A, the local diff shows PAYMENTS_TIMEOUT_MS going from 5000 to 1500, and the new security group may lack the egress rule the payments path needs. Both sit on the failing path, and either can explain timeouts that only the new tasks see. Roll back, then test each change on its own.

In case B, nothing on the payments path changed. If the old tasks were still serving and clean, look at what differs between old and new tasks that no revision records: their start time, their fresh connections, the secret value they read. Timeouts only from new tasks

Where the diff misleads#

An empty configuration diff does not mean the same thing ran. A forced redeployment of the same task definition can resolve a moved tag to a new digest, so compare digests even when the definition ARN did not change. Forcing a new deployment re-resolves digests

An unchanged secret reference can still deliver a new value. Rotation behind latest or AWSCURRENT changes what new tasks read without touching either revision. A secret rotated during the rollout · ECS secrets are injected at container start

The diff cannot see changes made in place: new rules on the same security group, a policy edit on the same role, a target group health check, DNS. Look for those in CloudTrail or Cloud Audit Logs for the rollout window.

Questions#

Why compare image digests instead of tags?

A tag can point to a different image tomorrow. ECS resolves the tag to a digest at each deployment, and Cloud Run resolves it when a revision is created, so the digest is the only record of the code that ran.

Can I paste a task definition diff into an incident channel?

Not as it is. describe-task-definition and --format export print literal environment values. Share the list of changed variable names and keep the values on your machine. Secret references print as ARNs or secret names, not values.

Where do ECS subnets and security groups appear if not in the task definition?

In the service revision ECS records for each deployment. describe-service-revisions returns its networkConfiguration with subnets, security groups and the public IP setting, for revisions created on or after October 25, 2024.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading