---
title: "Bad release or unhealthy dependency? | Proofline"
description: "When errors rise during a rollout on Amazon ECS or Cloud Run, split them by revision over the same minutes and look back to before the rollout began."
url: https://proofline.sh/learn/bad-release-or-unhealthy-dependency/
---

[Learning center](/learn/)

# Bad release or unhealthy dependency?

When errors rise during a rollout on Amazon ECS or Cloud Run, split them by revision over the same minutes and look back to before the rollout began. A dependency that was already unhealthy fails the old revision too, at least when it opens a fresh connection, and usually started before the deploy.

Deployment failures The Proofline team November 20, 2026 10 min read

## Two symptoms that point both ways

A rollout puts a new revision beside the old one for a few minutes, and anything that goes wrong in those minutes looks like the release's fault. Two symptoms come up most often:

-   **An ECS service's error rate rises while its deployment is in progress.** The Application Load Balancer's `HTTPCode_Target_5XX_Count` climbs a few minutes after the new task definition's first tasks register, and the application logs show `connection timed out` to the database.
-   **A Cloud Run service returns 500s after traffic moves to a new revision.** The new revision took 10% of traffic at 14:03, and the request logs show 500 responses from it, each preceded by a timeout calling a payments API.

Either symptom has three plausible causes: the release itself, a dependency that was degrading before the release, or the rollout's own mechanics, such as tasks draining from the load balancer or new instances starting cold. Rolling back helps only with the first. The two checks below take a few minutes and separate them.

## Check 1: split the errors by revision over the same window

During a rollout, both revisions serve requests against the same database, secret and network at the same time. That overlap is a controlled comparison. If the old revision's error rate rose too, the dependency changed. If only the new revision fails, either something the new revision does or uses differently is the cause, or the dependency now refuses new connections and the old revision survives on connections it opened earlier. One fresh connection from an old instance tells the two apart.

Compare rates, not counts. A revision that serves 10% of the traffic will show a tenth of the errors even when both fail at the same rate.

**On Cloud Run**, every request log entry carries the revision in `resource.labels.revision_name`, and the `run.googleapis.com/request_count` metric carries it as a resource label beside the `response_code_class` metric label. This Metrics Explorer query in PromQL gives the 5xx share for each revision over a sliding five-minute window; the metric is sampled once a minute, so a shorter window holds too few samples for `rate`. [Cloud Run request logs](https://cloud.google.com/run/docs/logging) · [Cloud Run metrics](https://cloud.google.com/monitoring/api/metrics_gcp_p_z#gcp-run) · [PromQL in Cloud Monitoring](https://cloud.google.com/monitoring/promql)

```
sum by (revision_name) (rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE", response_code_class="5xx"}[5m]))
/
sum by (revision_name) (rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))
```

**On ECS**, the load balancer's CloudWatch metrics describe the whole target group, and a rolling deployment registers the old and new tasks in the same target group. Split by task instead. With the `awslogs` driver, each task writes to its own log stream named `PREFIX/CONTAINER/TASK_ID`, so a CloudWatch Logs Insights query can count requests and errors per task, if the application logs one line per request. [awslogs stream names](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html) · [Logs Insights `parse`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-Parse.html)

```
fields @timestamp, @logStream, @message
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as requests,
        sum(strcontains(@message, "status=5")) as errors
  by task_id, bin(1m)
```

Replace `status=5` with whatever your request log writes for a server error. Then map each task ID to its task definition revision. `describe-tasks` accepts task IDs as well as ARNs, and returns stopped tasks for at least an hour after they stop. [describe-tasks](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html)

```
aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --tasks TASK_ID_1 TASK_ID_2 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,stopped:stoppedAt}' \
  --output table --no-cli-pager
```

If the service has Application Load Balancer access logs enabled, they give the same split without application logging: each entry records the target's IP address and port in `target:port`, its `target_status_code`, and the load balancer's own `elb_status_code`. In `awsvpc` network mode each task has its own IP address, which `describe-tasks` returns in its network attachment. The load balancer publishes access logs to S3 about every five minutes, so they arrive late in an incident. [Access log fields](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html)

## Check 2: read the old revision's errors before the rollout started

The word "already" in "already unhealthy" is a claim about time. Take the old revision's error rate for the 30 to 60 minutes before the rollout began, using the same query. A dependency that was degrading shows up there first: a few timeouts at 13:52, more at 13:58, then the deployment at 14:00 that everyone notices.

Use the time the new revision began serving, not the time someone merged or the pipeline started, and note two earlier moments: any migration or configuration change the pipeline applied, and the new revision's first instance start. A release can touch a dependency through either before it serves a request. On ECS, `describe-services` lists each deployment with its `createdAt` and task definition, and the service's events record when new tasks started and registered. On Cloud Run, a revision can exist for minutes before it receives traffic. [ECS deployments](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Deployment.html) · [Cloud Run rollouts and traffic migration](https://cloud.google.com/run/docs/rollouts-rollbacks-traffic-migration) · [Find when the rollout started serving](/learn/when-did-the-rollout-start-serving/)

## A worked fixture: one payments timeout, two different incidents

The fixture below is illustrative. It uses a fictional Cloud Run service `checkout` with revisions `checkout-00041` (old) and `checkout-00042` (new), and a dependency on a payments API. The pipeline ran no migration and changed no shared configuration (no secret, network rule or service account the old revision also uses), `checkout-00042` was created and started its first instance at 14:02, and the rollout moved 10% of traffic to it at 14:03 UTC. The same two symptoms produce two different tables.

**Incident A.** Per-minute 5xx share by revision:

| Minute (UTC) | `checkout-00041` | `checkout-00042` | Note |
| --- | --- | --- | --- |
| 13:45 | 0.0% |  |  |
| 13:52 | 0.4% |  | First payments timeouts in the old revision's logs |
| 13:58 | 1.9% |  |  |
| 14:03 | 2.6% |  | 10% of traffic moves to 00042 |
| 14:05 | 2.8% | 3.0% |  |
| 14:08 | 3.1% | 2.7% |  |

The old revision's errors started 11 minutes before it had any company, and both revisions fail at about the same rate once they serve together. The release did not start this. Before you rule out a rollback, check that it is not making things worse: `checkout`'s calls to the payments API held at about 1,200 a minute from 13:45 to 14:08, to the same two endpoints in the same proportion with the same request sizes, and the payments team's processing time per `checkout` call was flat until its own slowdown hit every caller, and `checkout-00042` carries the same client timeout and retry count as `checkout-00041`, so the release added no load to a struggling API. Rolling back would leave the error rate where it is and lose the minutes the payments API needed attention. Check the dependency's own health and the provider's status next. If the call rate had risen after 14:03, or the new revision retried more, the release would be amplifying the outage, and moving traffic back could still help. [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/) · [Rule a cloud provider incident in or out](/learn/cloud-provider-incident-during-rollout/)

**Incident B.** Same service, same rollout, same timeout message:

| Minute (UTC) | `checkout-00041` | `checkout-00042` | Note |
| --- | --- | --- | --- |
| 13:45 | 0.0% |  |  |
| 13:58 | 0.0% |  |  |
| 14:03 | 0.0% |  | 10% of traffic moves to 00042 |
| 14:05 | 0.1% | 8.4% |  |
| 14:08 | 0.0% | 9.1% |  |

The old revision keeps serving cleanly against the same payments API. Something about `checkout-00042` is different: a shorter client timeout, a new endpoint in its configuration, a missing egress route, or a code path the old revision never takes. Before moving traffic back, confirm that a fresh connection from a `checkout-00041` instance to the payments API succeeds; if it times out too, the old revision is living on pooled connections and a rollback would start instances that fail the same way. In this incident the fresh connection succeeds, and the service pins its secrets to version numbers, so a newly started `checkout-00041` instance reads the same credentials as the running ones. Move traffic back to `checkout-00041`, then compare what changed between the revisions. [Test a fresh connection from an old task](/learn/dependency-timeouts-only-from-new-tasks/) [Diff the two revisions](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

## Read the result

| What the split shows | Most likely cause | First move |
| --- | --- | --- |
| Old revision's errors started before the new revision's first instance and before any migration; both revisions fail at a similar rate | The dependency was already unhealthy | Investigate the dependency; a rollback will not help unless the release also raised this service's calls, connections or work per call on it |
| Old revision clean throughout; new revision fails; a fresh connection from an old instance succeeds | The release, or something only the new revision uses | Shift traffic back, then diff the revisions |
| Old revision clean throughout; new revision fails; a fresh connection from an old instance fails too | The dependency refuses new connections, or a shared DNS or network rule changed; old instances survive on pooled ones | Check the address each resolves and the rules on the path, then the dependency; a rollback starts instances that fail the same way |
| Both revisions clean before; both fail once the new one serves | The release changed how the service loads the dependency, or the rollout's overlap did | Check the dependency's callers and connection count |
| Errors cluster at the moments tasks or instances start and stop | The rollout's mechanics | Check draining, health checks and startup |

The fourth row is the hard one. A release that adds a query, raises a retry count or opens more connections per task can make a healthy dependency unhealthy for every revision. The split then looks like a dependency problem, and the timing points at the release. [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/) · [Connection budgets during rolling deployments](/learn/connection-limits/)

## Where the comparison stops working

The comparison needs both revisions serving in the same minutes. An ECS rolling update with a low `maximumPercent` or a Cloud Run deploy that moves all traffic at once leaves only a short overlap, and the old revision's later minutes are missing. Then fall back to Check 2 and to the dependency's own metrics.

New instances are not the same as old ones even with identical code. They start with cold caches, fresh connection pools and a fresh copy of any secret they read at startup. A failure that appears only in instances started after a given time can come from a rotated secret or a changed network path rather than the release. [A secret rotated during the rollout](/learn/secret-rotation-during-rollout/) · [Timeouts only from new tasks](/learn/dependency-timeouts-only-from-new-tasks/)

A rollback that clears the errors is weaker evidence than it looks, because it also restarts instances and moves traffic. [Does a rollback prove the release caused it?](/learn/does-a-rollback-prove-the-release-caused-it/)

## The rest of this collection

Each guide starts from one symptom on ECS or Cloud Run and gives the check that tells the release apart from what it depends on.

Finding the rollout's facts:

-   [Find when an ECS deployment or Cloud Run revision started serving](/learn/when-did-the-rollout-start-serving/)
-   [Diff the ECS task definitions or Cloud Run revisions on either side of a rollout](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)
-   [Does a rollback that cleared the errors prove the release caused them?](/learn/does-a-rollback-prove-the-release-caused-it/)

Amazon ECS:

-   [Why the ECS deployment circuit breaker rolled back](/learn/ecs-circuit-breaker-rollback-cause/)
-   [New ECS tasks fail load balancer health checks: read the reason code](/learn/ecs-target-health-check-reason-codes/)
-   [An ECS deployment stuck in progress: placement or crashing tasks](/learn/ecs-deployment-stuck-in-progress/)
-   [502 and 503 errors during every ECS deployment](/learn/alb-5xx-during-ecs-deployments/)

Cloud Run:

-   [A Cloud Run revision that is not ready: the container or its startup calls](/learn/cloud-run-revision-not-ready/)
-   [Cloud Run errors after shifting traffic to a new revision](/learn/cloud-run-errors-after-traffic-shift/)
-   [Test the previous Cloud Run revision against today's dependencies](/learn/test-previous-cloud-run-revision-with-tag/)

Dependencies:

-   [Dependency timeouts only from new tasks or revisions](/learn/dependency-timeouts-only-from-new-tasks/)
-   [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/)
-   [Containers killed for memory after a deploy: leak or backlog](/learn/container-memory-kills-after-deploy/)
-   [A secret rotated during the rollout](/learn/secret-rotation-during-rollout/)
-   [Rule a cloud provider incident in or out](/learn/cloud-provider-incident-during-rollout/)
-   [Health checks that call a dependency turn its outage into a failed rollout](/learn/health-checks-that-call-dependencies/)

For startup failures that name Secrets Manager, see [Fix ECS ResourceInitializationError when pulling secrets](/learn/ecs-secrets-startup-errors/).

## Compare signals at each deploy

Proofline's deploy verification reads the commit an environment is running and compares the signals the environment watches at checks after each deploy lands. A deploy whose signals got worse reads **Regressed**; the verification does not decide whether to roll back, and a deploy it verified is not a safe one. [Read a deploy's verification](/help/read-deploy-verification/) · [Current product availability](/how-it-works/)

## Questions

### How do I know whether a deploy or a dependency caused the errors?

Split the error rate by revision over the minutes both revisions served, and read the old revision's error rate from before the rollout began. If the old revision failed before the new revision's first instance started and both fail at a similar rate, the dependency was already unhealthy. If only the new revision fails and an old instance can still open a fresh connection to the dependency, look at the release.

### Why compare error rates and not error counts?

During a canary or a rolling update the revisions serve different shares of traffic. A revision with a tenth of the requests shows a tenth of the errors at the same failure rate, so counts make the larger revision look worse.

### What if the old revision was already fully replaced?

Use the old revision's minutes before the rollout, the dependency's own metrics, and the provider's status for the same window. On Cloud Run you can also send test requests to a previous revision through a revision tag without moving user traffic.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Find when an ECS deployment or Cloud Run revision started serving](/learn/when-did-the-rollout-start-serving/)
    
    Deployment failures The Proofline team November 20, 2026 9 min read
    
-   ### [What changed between two ECS task definitions or Cloud Run revisions?](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)
    
    Deployment failures The Proofline team November 20, 2026 8 min read
