---
title: "Does a rollback that cleared the errors prove the release caused them? | Proofline"
description: "Not on its own."
url: https://proofline.sh/learn/does-a-rollback-prove-the-release-caused-it/
---

[Learning center](/learn/)

# Does a rollback that cleared the errors prove the release caused them?

Not on its own. A rollback also starts fresh tasks or instances, opens new connections, re-reads secrets and moves placement, and it can land as the dependency recovers. It is good evidence when the old revision's errors did not fall at the same moment, when the errors fell as traffic moved back, and when the new revision fails again on fresh instances with no code change.

Deployment failures The Proofline team November 20, 2026 9 min read

## Two rollbacks that look decisive

-   An ECS service ran `checkout:42` with `HTTPCode_Target_5XX_Count` climbing. At 14:20 UTC the on-call engineer updated the service back to `checkout:41`; by 14:31 every task had been replaced and the 5xx count was zero.
-   A Cloud Run service had 50% of traffic on `checkout-00042`. At 14:20 the engineer ran `gcloud run services update-traffic checkout --to-revisions checkout-00041=100`, and the 500s stopped within a minute.

Both teams wrote "rolled back, resolved" and blocked the release. Whether that was right depends on what else the rollback changed and what the old revision was doing at the same time.

During an outage, roll back whenever it is the fastest way to stop the harm. The checks below are for afterward: for the post-incident review, and for deciding whether to ship the same revision again.

## What a rollback changes besides the code

On ECS with the rolling update deployment type, a rollback is a deployment. The scheduler replaces the running tasks with new ones from the older task definition, within the deployment's `minimumHealthyPercent` and `maximumPercent`. Every replacement task gets a new network interface and IP address in `awsvpc` mode, may land in a different Availability Zone or host, opens its connection pools from empty, and receives its secrets at container start, so it reads the current value of a secret that rotated since the old tasks started. [Rolling update deployments](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html) · [Secrets injected at container start](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html)

The replacement tasks also resolve the image again. If the older task definition names a mutable tag, the rolled-back tasks can run a different image from the one `checkout:41` ran last week. Read their `imageDigest` before you call it the old code. [Compare the digests](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

A circuit breaker rollback deploys the last service revision that completed successfully; ECS does not create a new service revision for it. ECS blue/green deployments behave differently: the blue revision keeps running until the bake time expires, so a rollback inside the bake time moves traffic back to tasks that never stopped. [Service revisions](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-revision.html) · [Blue/green deployments](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-blue-green.html)

On Cloud Run, a rollback to an existing revision is a traffic change. It updates routing immediately and does not re-run health checks or verify dependencies. Whether it restarts anything depends on the old revision's state. A revision still taking part of the traffic already has running instances, and the rollback sends the rest of the traffic to them and to the instances autoscaling adds. Cloud Run removes idle instances, and a revision with no traffic and no tag keeps no minimum instances, so a rollback to a revision that sat idle for a while starts fresh ones, and each fresh instance resolves secrets passed as environment variables when it starts. [Roll back to a previous revision](https://cloud.google.com/run/docs/rollouts-rollbacks-traffic-migration) · [Idle instances and minimum instances](https://cloud.google.com/run/docs/configuring/min-instances) · [Cloud Run secrets](https://cloud.google.com/run/docs/configuring/services/secrets)

So the rollback that tells you the most is a Cloud Run traffic move back to a revision that kept serving, or an ECS blue/green rollback inside the bake time. An ECS rolling rollback changes the most besides the code.

## Check 1: did the old revision's errors fall at the same moment?

If the old revision served throughout, as in a Cloud Run gradual rollout or the overlap of an ECS rolling update, read its error rate across the rollback. Use the per-revision query from the general method on Cloud Run, or the per-task split on ECS. [Split errors by revision](/learn/bad-release-or-unhealthy-dependency/#split-by-revision)

An old revision that stayed clean while the new one failed, and stayed clean after the rollback, gives the rollback most of its weight: the shared dependency was fine for the old instances. Warm old instances can still hide a dependency that fails only fresh ones, such as a secret read at startup or a database refusing new connections, so Check 3 compares fresh instances of both revisions.

An old revision whose errors fell at the same minute as the new revision's is ambiguous. Either the shared dependency recovered on its own, or the new revision was overloading it and stopped. Tell them apart with the dependency's load from your service per request, before and after. [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/)

## Check 2: did the errors fall with the traffic, or before it?

Line up three times: when the rollback started, when traffic reached the old revision, and when errors fell. On ECS, `describe-service-deployments` gives the rollback deployment's `createdAt`, `startedAt`, `finishedAt`, `status` and `statusReason`, and for a circuit breaker rollback its `rollback.startedAt` and `rollback.reason`. [describe-service-deployments](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-service-deployments.html) · [Find when the rollout started serving](/learn/when-did-the-rollout-start-serving/)

```
aws ecs list-service-deployments \
  --profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
  --query 'serviceDeployments[].{arn:serviceDeploymentArn,created:createdAt,status:status}' \
  --output table --no-cli-pager

aws ecs describe-service-deployments \
  --profile PROFILE --region REGION \
  --service-deployment-arns SERVICE_DEPLOYMENT_ARN \
  --query 'serviceDeployments[].{created:createdAt,started:startedAt,finished:finishedAt,
    status:status,reason:statusReason,rollback:rollback,
    source:sourceServiceRevisions,target:targetServiceRevision}' \
  --output json --no-cli-pager
```

On Cloud Run, the minute traffic moved shows in the request rate by revision, and instance counts by revision show whether the rollback started new instances. [Cloud Run metrics](https://cloud.google.com/monitoring/api/metrics_gcp_p_z#gcp-run)

```
sum by (revision_name) (rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))

sum by (revision_name, state) (run_googleapis_com:container_instance_count{monitored_resource="cloud_run_revision", service_name="SERVICE"})
```

Errors that fell before the rollback started were recovering anyway. Errors that fell in step with the traffic moving, percentage by percentage on Cloud Run, follow the code, since the old instances were already running. On ECS a rolling rollback also replaces each task's connections as it moves traffic, so errors falling task by task can follow either. Errors that held until the last new task stopped and then dropped at once can follow either the code or the restart, and Check 3 decides.

## Check 3: run the new revision again on fresh instances

The cleanest control is the new revision with no code change, on fresh tasks or instances, after the incident, beside the old revision on fresh instances under the same dependencies. If the new revision fails and the old one serves cleanly, the failure belongs to the new revision. Before calling it the release's fault, check that the two read the same secret values. Matching references are not enough: a new revision on `latest` beside an old one pinned to a version number diverges as soon as someone adds a version, and two revisions both on `latest` or `AWSCURRENT` diverge when a rotation lands between their instances' starts. Pin both to the same version for the control, or, with both on the same moving selector, confirm from the secret's version history that no version changed between the first control instance's start and the last, nor in the minutes before it while `latest` reads converge. [Check the secret selectors](/learn/secret-rotation-during-rollout/) If both fail, fresh instances meet a dependency state the warm ones escaped, such as a rotated secret behind `latest`, and the release is not established as the cause. [A secret rotated during the rollout](/learn/secret-rotation-during-rollout/) Send the incident's own failing request, with the same account and payload, or an equivalent read-only one; a clean control on other requests can miss a defect on one code path. If the new revision serves that request cleanly, the original failure came from state the restart cleared, such as a stale connection pool or an old secret value, or from a dependency problem that has passed.

On Cloud Run, revision tags route test requests to `checkout-00042` and `checkout-00041` without moving user traffic, and once a revision has scaled in, those requests reach instances started after the incident. [Test a revision through a tag](/learn/test-previous-cloud-run-revision-with-tag/)

On ECS, run both task definitions as small canaries or in a staging service against the same dependencies. If the task definition uses a tag, pin the image by the digest the failing tasks ran, or the control may test different code.

A control with a handful of test requests cannot reproduce a failure that needed production load. If the suspected cause is load, such as an extra query or more connections, judge it with the dependency's per-caller metrics instead.

## A worked fixture: the same rollback, two verdicts

The fixture is illustrative. A fictional Cloud Run service `checkout` had 50% of traffic on `checkout-00042` from 14:03 UTC. At 14:20 the engineer moved all traffic to `checkout-00041`. Both cases show the 5xx share by revision.

Case A, the rollback is good evidence:

| Minute (UTC) | Traffic on 00042 | `checkout-00041` | `checkout-00042` | Note |
| --- | --- | --- | --- | --- |
| 14:10 | 50% | 0.1% | 9.2% |  |
| 14:18 | 50% | 0.0% | 9.5% |  |
| 14:20 | 0% | 0.1% |  | Rollback |
| 14:30 | 0% | 0.1% |  |  |
| 14:45 | tag only | 0.0% | 9.0% | Tagged test requests to both, fresh instances |

The old revision was clean before, during and after, and so were its fresh instances at 14:45. The new revision failed again on fresh instances under the same dependencies, and both revisions' instances read the same secret versions: they pin each secret to the same version number, and none of those versions was disabled that day. Do not ship `checkout-00042` again; diff it against `checkout-00041`.

Case B, the rollback took credit for a recovery:

| Minute (UTC) | Traffic on 00042 | `checkout-00041` | `checkout-00042` | Note |
| --- | --- | --- | --- | --- |
| 14:10 | 50% | 2.4% | 2.6% | Payments API timeouts in both |
| 14:18 | 50% | 1.1% | 1.0% | Payments latency falling since 14:15 |
| 14:20 | 0% | 0.2% |  | Rollback |
| 14:30 | 0% | 0.0% |  |  |
| 14:45 | tag only |  | 0.0% | Tagged test requests, fresh instances |

Both revisions failed at the same rate, both started recovering at 14:15, and the new revision served cleanly on fresh instances. The checkout service's calls and open connections per request to the payments API did not change across the rollout, and the payments team's per-client request rates show checkout's share flat, so nothing measured points to `checkout-00042` overloading the API. That leaves the payments API's own slowdown as the likely cause, not a proven one: `checkout-00042` can ship again once the payments team confirms what slowed it. [Rule a cloud provider incident in or out](/learn/cloud-provider-incident-during-rollout/)

## Where the evidence stays weak

A full replacement leaves no comparison. If an ECS rolling update had already replaced every old task before the rollback, there is no old revision serving in the same minutes, and Checks 1 and 2 reduce to timing alone.

Low traffic hides rates. A revision serving a few requests a minute can show 0% or 50% errors from one request either way; widen the bins before reading them.

The control runs later than the incident. The dependency, the secret and the network may all have changed in between, so a clean control shows that the revision works now, not that it was innocent at 14:10.

## Questions

### The errors stopped right after the rollback. Is that enough for the post-incident report?

It shows that something the rollback changed stopped the errors. Name the cause as the release only if the new revision fails again on fresh instances while the old revision's fresh instances serve cleanly, with both reading the same secret versions: the same pinned versions, or the same moving selector and no rotation between their instances' starts or in the minutes before them. An old revision whose errors did not fall supports that, but warm old instances can hide a dependency that fails only fresh ones. Otherwise record the restart and the dependency's recovery as alternatives.

### Does a Cloud Run rollback restart the old revision?

It changes routing only. If the old revision still had traffic, its instances keep serving. If it had no traffic and no tag, Cloud Run will have removed its idle instances after a while, and the rollback starts new ones.

### Should we wait to roll back until we have this evidence?

No. Roll back when it is the fastest way to stop user impact. Collect the evidence afterward, from metrics that cover the rollback minutes, before you decide whether to ship the same revision again.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Test the previous Cloud Run revision against today's dependencies](/learn/test-previous-cloud-run-revision-with-tag/)
    
    Deployment failures The Proofline team November 20, 2026 8 min read
    
-   ### [Did our deploy overload a dependency that was healthy before it?](/learn/did-our-deploy-overload-a-dependency/)
    
    Capacity overruns The Proofline team November 20, 2026 10 min read
