Two symptoms that send you to the diff#
- An ECS service moved from task definition
checkout:41tocheckout:42. The new tasks logconnection timed outto the payments API, the old tasks did not, and the pull request touched only a template file. - A Cloud Run service shifted 10% of traffic to
checkout-00042. Its 500s come with timeouts calling the same payments API, whilecheckout-00041keeps serving cleanly, and the deploy pipeline says the configuration did not change.
In both cases the split by revision already points at the new revision. The question left is what the new revision runs or uses differently. A code diff answers part of it; the revision records the rest: which image digest started, what configuration it received, and how it reaches the network. Split errors by revision first
What a revision records, and what it leaves out#
| Setting | ECS | Cloud Run |
|---|---|---|
| Image | image in each container definition, as written; the digest that ran is in describe-tasks and the service revision | spec.containers[].image as written; status.imageDigest holds the digest resolved when the revision was created |
| Environment | environment, plus environmentFiles stored in S3 | env[].value |
| Secret references | secrets[].valueFrom | env[].valueFrom.secretKeyRef (secret name and version), secret volumes |
| Identity | taskRoleArn for the application, executionRoleArn for the agent | serviceAccountName |
| CPU and memory | cpu, memory on the task and each container | resources.limits |
| Ports and health | portMappings, container healthCheck | ports, startupProbe, livenessProbe |
| Network | Service revision networkConfiguration: subnets, security groups, public IP | Annotations run.googleapis.com/vpc-access-egress, run.googleapis.com/network-interfaces, run.googleapis.com/vpc-access-connector |
| Concurrency and scale | Outside the task definition | containerConcurrency, timeoutSeconds, autoscaling.knative.dev/minScale and maxScale |
ECS resolves the image tag in a task definition to a digest when a deployment starts its first task, and a forced new deployment resolves it again, so one task definition can run different images in two deployments if the tag moved. A Cloud Run revision resolves its digest once, when the revision is created. ECS container image resolution · Cloud Run revision imageDigest · Cloud Run YAML reference
Some settings that affect a dependency call live outside both revisions: the ALB target group's health check, the rules inside a security group whose ID did not change, the policies attached to a role whose ARN did not change, Cloud Run ingress and service-level minimum instances on the service's own metadata, and the value behind a secret reference.
Diff two ECS task definitions#
Pass full family:revision identifiers, because a family name alone resolves to the latest active revision. The function drops the fields that always differ, sorts keys and environment entries, and writes one normalized file per revision. describe-task-definition
td() {
aws ecs describe-task-definition \
--profile PROFILE --region REGION --task-definition "$1" \
--query taskDefinition --output json --no-cli-pager |
jq -S 'del(.taskDefinitionArn, .revision, .registeredAt, .registeredBy)
| .containerDefinitions |= (sort_by(.name)
| map(.environment |= (. // [] | sort_by(.name))))'
}
td FAMILY:OLD_REVISION > old.json
td FAMILY:NEW_REVISION > new.json
diff -u old.json new.json
These files contain literal environment values, which often include internal hostnames, account IDs and sometimes credentials that should have been secrets. Keep them on your machine. To share the result in a ticket, list only the names whose values changed:
jq -n --slurpfile a old.json --slurpfile b new.json '
def env($t): [$t.containerDefinitions[] | .name as $c
| .environment[] | {key: "\($c)/\(.name)", value}] | from_entries;
env($a[0]) as $x | env($b[0]) as $y
| [$x + $y | keys[] | select($x[.] != $y[.])]'
Secret references print as ARNs, never values. environmentFiles prints only the S3 object ARN, so a changed file under the same ARN will not show up. environmentFiles
The task definition does not hold the service's subnets, security groups or load balancer. Service revisions do. ECS records one whenever a service update starts a deployment, and you can describe those created on or after October 25, 2024. List the deployments to find the revision ARNs, then describe both. Service revisions · list-service-deployments · describe-service-revisions
aws ecs list-service-deployments \
--profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
--query 'serviceDeployments[].{created:createdAt,status:status,revision:targetServiceRevisionArn}' \
--output table --no-cli-pager
aws ecs describe-service-revisions \
--profile PROFILE --region REGION \
--service-revision-arns OLD_SERVICE_REVISION_ARN NEW_SERVICE_REVISION_ARN \
--query 'serviceRevisions[].{revision:serviceRevisionArn,definition:taskDefinition,
images:containerImages,network:networkConfiguration,
loadBalancers:loadBalancers,platform:platformVersion}' \
--output json --no-cli-pager
containerImages gives each container's image and the imageDigest ECS resolved for that deployment. To confirm what a particular failing task ran, read its containers' imageDigest with describe-tasks. ServiceRevision fields · Task container fields
aws ecs describe-tasks \
--profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 \
--query 'tasks[].{task:taskArn,definition:taskDefinitionArn,
containers:containers[].{name:name,image:image,digest:imageDigest}}' \
--output json --no-cli-pager
Diff two Cloud Run revisions#
A revision is immutable, so describing it later returns the configuration it served with. --format export writes YAML without status and without automatically generated labels, which keeps the diff to configuration. It also drops status.imageDigest, so read the digests separately. Manage revisions · View service and revision details · gcloud run revisions describe
gcloud run revisions describe OLD_REVISION --project PROJECT --region REGION --format export > old.yaml
gcloud run revisions describe NEW_REVISION --project PROJECT --region REGION --format export > new.yaml
diff -u old.yaml new.yaml
gcloud run revisions describe OLD_REVISION --project PROJECT --region REGION --format 'value(status.imageDigest)'
gcloud run revisions describe NEW_REVISION --project PROJECT --region REGION --format 'value(status.imageDigest)'
The exported YAML also holds literal env values, so treat it like the ECS files. A secret reference shows the secret name and the version in key. If the key is latest, both revisions can show the same reference while their instances hold different values, because Cloud Run resolves secrets passed as environment variables when each instance starts. Cloud Run secrets
Ingress is set by the run.googleapis.com/ingress annotation on the service, outside the revision template, and so is service-level minimum instances. Read them with gcloud run services describe SERVICE --project PROJECT --region REGION --format export. That shows only the current value. Ingress settings
Which differences can explain a dependency symptom#
| Difference | Can it explain timeouts or errors to one dependency? |
|---|---|
Dependency endpoint, host or port in environment or env | Yes; the new revision calls a different place |
Client timeout, retry count, pool size, AWS_MAX_ATTEMPTS | Yes; shorter timeouts fail sooner, and retries and larger pools add load AWS SDK retry settings |
| Secret reference: different secret, version or JSON key | Yes, for authentication failures |
| Task role, execution role or service account | Yes, for permission errors on the dependency; the execution role is used by the ECS agent for image pulls, secret injection and logs, not by application calls Execution role |
| Subnets, security groups, VPC egress setting or connector | Yes, for connection timeouts that start in the new revision only |
CPU, memory, containerConcurrency, maxScale | Indirectly: CPU starvation slows calls, and more concurrency or instances multiply connections |
| Image digest | Only through the code it contains; diff the commits |
| Health check, port, labels, log configuration | Rarely; they explain failed rollouts, not dependency errors |
A dependency-shaped symptom with none of the "yes" rows in the diff points back at the dependency, at something outside the revision, or at the code itself: a new query or a bypassed cache changes only the image digest. Compare this service's calls and work per request on the dependency before and after before you clear the code. Did our deploy overload a dependency?
A worked fixture: two diffs for the same timeout#
The fixture is illustrative: a fictional ECS service checkout deployed from checkout:41 to checkout:42, with new tasks timing out against the payments API. Digests are shortened.
| What the diff shows | Case A | Case B |
|---|---|---|
| Image digest, old → new | sha256:3b9e… → sha256:c71d… | sha256:3b9e… → sha256:c71d… |
| Environment names with changed values | app/PAYMENTS_TIMEOUT_MS | app/BANNER_TEXT |
| Secret references | unchanged | unchanged |
| Task role, execution role | unchanged | unchanged |
Service revision networkConfiguration | securityGroups changed: sg-0a1… → sg-0f4… | unchanged |
| Commits between the digests | template change only | template change only |
In case A, the local diff shows PAYMENTS_TIMEOUT_MS going from 5000 to 1500, and the new security group may lack the egress rule the payments path needs. Both sit on the failing path, and either can explain timeouts that only the new tasks see. Roll back, then test each change on its own.
In case B, nothing on the payments path changed. If the old tasks were still serving and clean, look at what differs between old and new tasks that no revision records: their start time, their fresh connections, the secret value they read. Timeouts only from new tasks
Where the diff misleads#
An empty configuration diff does not mean the same thing ran. A forced redeployment of the same task definition can resolve a moved tag to a new digest, so compare digests even when the definition ARN did not change. Forcing a new deployment re-resolves digests
An unchanged secret reference can still deliver a new value. Rotation behind latest or AWSCURRENT changes what new tasks read without touching either revision. A secret rotated during the rollout · ECS secrets are injected at container start
The diff cannot see changes made in place: new rules on the same security group, a policy edit on the same role, a target group health check, DNS. Look for those in CloudTrail or Cloud Audit Logs for the rollout window.
Questions#
Why compare image digests instead of tags?
A tag can point to a different image tomorrow. ECS resolves the tag to a digest at each deployment, and Cloud Run resolves it when a revision is created, so the digest is the only record of the code that ran.
Can I paste a task definition diff into an incident channel?
Not as it is. describe-task-definition and --format export print literal environment values. Share the list of changed variable names and keep the values on your machine. Secret references print as ARNs or secret names, not values.
Where do ECS subnets and security groups appear if not in the task definition?
In the service revision ECS records for each deployment. describe-service-revisions returns its networkConfiguration with subnets, security groups and the public IP setting, for revisions created on or after October 25, 2024.