Learning center

Bad release or unhealthy dependency?

When errors rise during a rollout on Amazon ECS or Cloud Run, split them by revision over the same minutes and look back to before the rollout began. A dependency that was already unhealthy fails the old revision too, at least when it opens a fresh connection, and usually started before the deploy.

Two symptoms that point both ways#

A rollout puts a new revision beside the old one for a few minutes, and anything that goes wrong in those minutes looks like the release's fault. Two symptoms come up most often:

  • An ECS service's error rate rises while its deployment is in progress. The Application Load Balancer's HTTPCode_Target_5XX_Count climbs a few minutes after the new task definition's first tasks register, and the application logs show connection timed out to the database.
  • A Cloud Run service returns 500s after traffic moves to a new revision. The new revision took 10% of traffic at 14:03, and the request logs show 500 responses from it, each preceded by a timeout calling a payments API.

Either symptom has three plausible causes: the release itself, a dependency that was degrading before the release, or the rollout's own mechanics, such as tasks draining from the load balancer or new instances starting cold. Rolling back helps only with the first. The two checks below take a few minutes and separate them.

Check 1: split the errors by revision over the same window#

During a rollout, both revisions serve requests against the same database, secret and network at the same time. That overlap is a controlled comparison. If the old revision's error rate rose too, the dependency changed. If only the new revision fails, either something the new revision does or uses differently is the cause, or the dependency now refuses new connections and the old revision survives on connections it opened earlier. One fresh connection from an old instance tells the two apart.

Compare rates, not counts. A revision that serves 10% of the traffic will show a tenth of the errors even when both fail at the same rate.

On Cloud Run, every request log entry carries the revision in resource.labels.revision_name, and the run.googleapis.com/request_count metric carries it as a resource label beside the response_code_class metric label. This Metrics Explorer query in PromQL gives the 5xx share for each revision over a sliding five-minute window; the metric is sampled once a minute, so a shorter window holds too few samples for rate. Cloud Run request logs · Cloud Run metrics · PromQL in Cloud Monitoring

sum by (revision_name) (rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE", response_code_class="5xx"}[5m]))
/
sum by (revision_name) (rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))

On ECS, the load balancer's CloudWatch metrics describe the whole target group, and a rolling deployment registers the old and new tasks in the same target group. Split by task instead. With the awslogs driver, each task writes to its own log stream named PREFIX/CONTAINER/TASK_ID, so a CloudWatch Logs Insights query can count requests and errors per task, if the application logs one line per request. awslogs stream names · Logs Insights parse

fields @timestamp, @logStream, @message
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as requests,
        sum(strcontains(@message, "status=5")) as errors
  by task_id, bin(1m)

Replace status=5 with whatever your request log writes for a server error. Then map each task ID to its task definition revision. describe-tasks accepts task IDs as well as ARNs, and returns stopped tasks for at least an hour after they stop. describe-tasks

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --tasks TASK_ID_1 TASK_ID_2 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,stopped:stoppedAt}' \
  --output table --no-cli-pager

If the service has Application Load Balancer access logs enabled, they give the same split without application logging: each entry records the target's IP address and port in target:port, its target_status_code, and the load balancer's own elb_status_code. In awsvpc network mode each task has its own IP address, which describe-tasks returns in its network attachment. The load balancer publishes access logs to S3 about every five minutes, so they arrive late in an incident. Access log fields

Check 2: read the old revision's errors before the rollout started#

The word "already" in "already unhealthy" is a claim about time. Take the old revision's error rate for the 30 to 60 minutes before the rollout began, using the same query. A dependency that was degrading shows up there first: a few timeouts at 13:52, more at 13:58, then the deployment at 14:00 that everyone notices.

Use the time the new revision began serving, not the time someone merged or the pipeline started, and note two earlier moments: any migration or configuration change the pipeline applied, and the new revision's first instance start. A release can touch a dependency through either before it serves a request. On ECS, describe-services lists each deployment with its createdAt and task definition, and the service's events record when new tasks started and registered. On Cloud Run, a revision can exist for minutes before it receives traffic. ECS deployments · Cloud Run rollouts and traffic migration · Find when the rollout started serving

A worked fixture: one payments timeout, two different incidents#

The fixture below is illustrative. It uses a fictional Cloud Run service checkout with revisions checkout-00041 (old) and checkout-00042 (new), and a dependency on a payments API. The pipeline ran no migration and changed no shared configuration (no secret, network rule or service account the old revision also uses), checkout-00042 was created and started its first instance at 14:02, and the rollout moved 10% of traffic to it at 14:03 UTC. The same two symptoms produce two different tables.

Incident A. Per-minute 5xx share by revision:

Minute (UTC)checkout-00041checkout-00042Note
13:450.0%
13:520.4%First payments timeouts in the old revision's logs
13:581.9%
14:032.6%10% of traffic moves to 00042
14:052.8%3.0%
14:083.1%2.7%

The old revision's errors started 11 minutes before it had any company, and both revisions fail at about the same rate once they serve together. The release did not start this. Before you rule out a rollback, check that it is not making things worse: checkout's calls to the payments API held at about 1,200 a minute from 13:45 to 14:08, to the same two endpoints in the same proportion with the same request sizes, and the payments team's processing time per checkout call was flat until its own slowdown hit every caller, and checkout-00042 carries the same client timeout and retry count as checkout-00041, so the release added no load to a struggling API. Rolling back would leave the error rate where it is and lose the minutes the payments API needed attention. Check the dependency's own health and the provider's status next. If the call rate had risen after 14:03, or the new revision retried more, the release would be amplifying the outage, and moving traffic back could still help. Did our deploy overload a dependency? · Rule a cloud provider incident in or out

Incident B. Same service, same rollout, same timeout message:

Minute (UTC)checkout-00041checkout-00042Note
13:450.0%
13:580.0%
14:030.0%10% of traffic moves to 00042
14:050.1%8.4%
14:080.0%9.1%

The old revision keeps serving cleanly against the same payments API. Something about checkout-00042 is different: a shorter client timeout, a new endpoint in its configuration, a missing egress route, or a code path the old revision never takes. Before moving traffic back, confirm that a fresh connection from a checkout-00041 instance to the payments API succeeds; if it times out too, the old revision is living on pooled connections and a rollback would start instances that fail the same way. In this incident the fresh connection succeeds, and the service pins its secrets to version numbers, so a newly started checkout-00041 instance reads the same credentials as the running ones. Move traffic back to checkout-00041, then compare what changed between the revisions. Test a fresh connection from an old task Diff the two revisions

Read the result#

What the split showsMost likely causeFirst move
Old revision's errors started before the new revision's first instance and before any migration; both revisions fail at a similar rateThe dependency was already unhealthyInvestigate the dependency; a rollback will not help unless the release also raised this service's calls, connections or work per call on it
Old revision clean throughout; new revision fails; a fresh connection from an old instance succeedsThe release, or something only the new revision usesShift traffic back, then diff the revisions
Old revision clean throughout; new revision fails; a fresh connection from an old instance fails tooThe dependency refuses new connections, or a shared DNS or network rule changed; old instances survive on pooled onesCheck the address each resolves and the rules on the path, then the dependency; a rollback starts instances that fail the same way
Both revisions clean before; both fail once the new one servesThe release changed how the service loads the dependency, or the rollout's overlap didCheck the dependency's callers and connection count
Errors cluster at the moments tasks or instances start and stopThe rollout's mechanicsCheck draining, health checks and startup

The fourth row is the hard one. A release that adds a query, raises a retry count or opens more connections per task can make a healthy dependency unhealthy for every revision. The split then looks like a dependency problem, and the timing points at the release. Did our deploy overload a dependency? · Connection budgets during rolling deployments

Where the comparison stops working#

The comparison needs both revisions serving in the same minutes. An ECS rolling update with a low maximumPercent or a Cloud Run deploy that moves all traffic at once leaves only a short overlap, and the old revision's later minutes are missing. Then fall back to Check 2 and to the dependency's own metrics.

New instances are not the same as old ones even with identical code. They start with cold caches, fresh connection pools and a fresh copy of any secret they read at startup. A failure that appears only in instances started after a given time can come from a rotated secret or a changed network path rather than the release. A secret rotated during the rollout · Timeouts only from new tasks

A rollback that clears the errors is weaker evidence than it looks, because it also restarts instances and moves traffic. Does a rollback prove the release caused it?

The rest of this collection#

Each guide starts from one symptom on ECS or Cloud Run and gives the check that tells the release apart from what it depends on.

Finding the rollout's facts:

Amazon ECS:

Cloud Run:

Dependencies:

For startup failures that name Secrets Manager, see Fix ECS ResourceInitializationError when pulling secrets.

Compare signals at each deploy#

Proofline's deploy verification reads the commit an environment is running and compares the signals the environment watches at checks after each deploy lands. A deploy whose signals got worse reads Regressed; the verification does not decide whether to roll back, and a deploy it verified is not a safe one. Read a deploy's verification · Current product availability

Questions#

How do I know whether a deploy or a dependency caused the errors?

Split the error rate by revision over the minutes both revisions served, and read the old revision's error rate from before the rollout began. If the old revision failed before the new revision's first instance started and both fail at a similar rate, the dependency was already unhealthy. If only the new revision fails and an old instance can still open a fresh connection to the dependency, look at the release.

Why compare error rates and not error counts?

During a canary or a rolling update the revisions serve different shares of traffic. A revision with a tenth of the requests shows a tenth of the errors at the same failure rate, so counts make the larger revision look worse.

What if the old revision was already fully replaced?

Use the old revision's minutes before the rollout, the dependency's own metrics, and the provider's status for the same window. On Cloud Run you can also send test requests to a previous revision through a revision tag without moving user traffic.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading