Learning center

Does a rollback that cleared the errors prove the release caused them?

Not on its own. A rollback also starts fresh tasks or instances, opens new connections, re-reads secrets and moves placement, and it can land as the dependency recovers. It is good evidence when the old revision's errors did not fall at the same moment, when the errors fell as traffic moved back, and when the new revision fails again on fresh instances with no code change.

Two rollbacks that look decisive#

  • An ECS service ran checkout:42 with HTTPCode_Target_5XX_Count climbing. At 14:20 UTC the on-call engineer updated the service back to checkout:41; by 14:31 every task had been replaced and the 5xx count was zero.
  • A Cloud Run service had 50% of traffic on checkout-00042. At 14:20 the engineer ran gcloud run services update-traffic checkout --to-revisions checkout-00041=100, and the 500s stopped within a minute.

Both teams wrote "rolled back, resolved" and blocked the release. Whether that was right depends on what else the rollback changed and what the old revision was doing at the same time.

During an outage, roll back whenever it is the fastest way to stop the harm. The checks below are for afterward: for the post-incident review, and for deciding whether to ship the same revision again.

What a rollback changes besides the code#

On ECS with the rolling update deployment type, a rollback is a deployment. The scheduler replaces the running tasks with new ones from the older task definition, within the deployment's minimumHealthyPercent and maximumPercent. Every replacement task gets a new network interface and IP address in awsvpc mode, may land in a different Availability Zone or host, opens its connection pools from empty, and receives its secrets at container start, so it reads the current value of a secret that rotated since the old tasks started. Rolling update deployments · Secrets injected at container start

The replacement tasks also resolve the image again. If the older task definition names a mutable tag, the rolled-back tasks can run a different image from the one checkout:41 ran last week. Read their imageDigest before you call it the old code. Compare the digests

A circuit breaker rollback deploys the last service revision that completed successfully; ECS does not create a new service revision for it. ECS blue/green deployments behave differently: the blue revision keeps running until the bake time expires, so a rollback inside the bake time moves traffic back to tasks that never stopped. Service revisions · Blue/green deployments

On Cloud Run, a rollback to an existing revision is a traffic change. It updates routing immediately and does not re-run health checks or verify dependencies. Whether it restarts anything depends on the old revision's state. A revision still taking part of the traffic already has running instances, and the rollback sends the rest of the traffic to them and to the instances autoscaling adds. Cloud Run removes idle instances, and a revision with no traffic and no tag keeps no minimum instances, so a rollback to a revision that sat idle for a while starts fresh ones, and each fresh instance resolves secrets passed as environment variables when it starts. Roll back to a previous revision · Idle instances and minimum instances · Cloud Run secrets

So the rollback that tells you the most is a Cloud Run traffic move back to a revision that kept serving, or an ECS blue/green rollback inside the bake time. An ECS rolling rollback changes the most besides the code.

Check 1: did the old revision's errors fall at the same moment?#

If the old revision served throughout, as in a Cloud Run gradual rollout or the overlap of an ECS rolling update, read its error rate across the rollback. Use the per-revision query from the general method on Cloud Run, or the per-task split on ECS. Split errors by revision

An old revision that stayed clean while the new one failed, and stayed clean after the rollback, gives the rollback most of its weight: the shared dependency was fine for the old instances. Warm old instances can still hide a dependency that fails only fresh ones, such as a secret read at startup or a database refusing new connections, so Check 3 compares fresh instances of both revisions.

An old revision whose errors fell at the same minute as the new revision's is ambiguous. Either the shared dependency recovered on its own, or the new revision was overloading it and stopped. Tell them apart with the dependency's load from your service per request, before and after. Did our deploy overload a dependency?

Check 2: did the errors fall with the traffic, or before it?#

Line up three times: when the rollback started, when traffic reached the old revision, and when errors fell. On ECS, describe-service-deployments gives the rollback deployment's createdAt, startedAt, finishedAt, status and statusReason, and for a circuit breaker rollback its rollback.startedAt and rollback.reason. describe-service-deployments · Find when the rollout started serving

aws ecs list-service-deployments \
  --profile PROFILE --region REGION --cluster CLUSTER --service SERVICE \
  --query 'serviceDeployments[].{arn:serviceDeploymentArn,created:createdAt,status:status}' \
  --output table --no-cli-pager

aws ecs describe-service-deployments \
  --profile PROFILE --region REGION \
  --service-deployment-arns SERVICE_DEPLOYMENT_ARN \
  --query 'serviceDeployments[].{created:createdAt,started:startedAt,finished:finishedAt,
    status:status,reason:statusReason,rollback:rollback,
    source:sourceServiceRevisions,target:targetServiceRevision}' \
  --output json --no-cli-pager

On Cloud Run, the minute traffic moved shows in the request rate by revision, and instance counts by revision show whether the rollback started new instances. Cloud Run metrics

sum by (revision_name) (rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))

sum by (revision_name, state) (run_googleapis_com:container_instance_count{monitored_resource="cloud_run_revision", service_name="SERVICE"})

Errors that fell before the rollback started were recovering anyway. Errors that fell in step with the traffic moving, percentage by percentage on Cloud Run, follow the code, since the old instances were already running. On ECS a rolling rollback also replaces each task's connections as it moves traffic, so errors falling task by task can follow either. Errors that held until the last new task stopped and then dropped at once can follow either the code or the restart, and Check 3 decides.

Check 3: run the new revision again on fresh instances#

The cleanest control is the new revision with no code change, on fresh tasks or instances, after the incident, beside the old revision on fresh instances under the same dependencies. If the new revision fails and the old one serves cleanly, the failure belongs to the new revision. Before calling it the release's fault, check that the two read the same secret values. Matching references are not enough: a new revision on latest beside an old one pinned to a version number diverges as soon as someone adds a version, and two revisions both on latest or AWSCURRENT diverge when a rotation lands between their instances' starts. Pin both to the same version for the control, or, with both on the same moving selector, confirm from the secret's version history that no version changed between the first control instance's start and the last, nor in the minutes before it while latest reads converge. Check the secret selectors If both fail, fresh instances meet a dependency state the warm ones escaped, such as a rotated secret behind latest, and the release is not established as the cause. A secret rotated during the rollout Send the incident's own failing request, with the same account and payload, or an equivalent read-only one; a clean control on other requests can miss a defect on one code path. If the new revision serves that request cleanly, the original failure came from state the restart cleared, such as a stale connection pool or an old secret value, or from a dependency problem that has passed.

On Cloud Run, revision tags route test requests to checkout-00042 and checkout-00041 without moving user traffic, and once a revision has scaled in, those requests reach instances started after the incident. Test a revision through a tag

On ECS, run both task definitions as small canaries or in a staging service against the same dependencies. If the task definition uses a tag, pin the image by the digest the failing tasks ran, or the control may test different code.

A control with a handful of test requests cannot reproduce a failure that needed production load. If the suspected cause is load, such as an extra query or more connections, judge it with the dependency's per-caller metrics instead.

A worked fixture: the same rollback, two verdicts#

The fixture is illustrative. A fictional Cloud Run service checkout had 50% of traffic on checkout-00042 from 14:03 UTC. At 14:20 the engineer moved all traffic to checkout-00041. Both cases show the 5xx share by revision.

Case A, the rollback is good evidence:

Minute (UTC)Traffic on 00042checkout-00041checkout-00042Note
14:1050%0.1%9.2%
14:1850%0.0%9.5%
14:200%0.1%Rollback
14:300%0.1%
14:45tag only0.0%9.0%Tagged test requests to both, fresh instances

The old revision was clean before, during and after, and so were its fresh instances at 14:45. The new revision failed again on fresh instances under the same dependencies, and both revisions' instances read the same secret versions: they pin each secret to the same version number, and none of those versions was disabled that day. Do not ship checkout-00042 again; diff it against checkout-00041.

Case B, the rollback took credit for a recovery:

Minute (UTC)Traffic on 00042checkout-00041checkout-00042Note
14:1050%2.4%2.6%Payments API timeouts in both
14:1850%1.1%1.0%Payments latency falling since 14:15
14:200%0.2%Rollback
14:300%0.0%
14:45tag only0.0%Tagged test requests, fresh instances

Both revisions failed at the same rate, both started recovering at 14:15, and the new revision served cleanly on fresh instances. The checkout service's calls and open connections per request to the payments API did not change across the rollout, and the payments team's per-client request rates show checkout's share flat, so nothing measured points to checkout-00042 overloading the API. That leaves the payments API's own slowdown as the likely cause, not a proven one: checkout-00042 can ship again once the payments team confirms what slowed it. Rule a cloud provider incident in or out

Where the evidence stays weak#

A full replacement leaves no comparison. If an ECS rolling update had already replaced every old task before the rollback, there is no old revision serving in the same minutes, and Checks 1 and 2 reduce to timing alone.

Low traffic hides rates. A revision serving a few requests a minute can show 0% or 50% errors from one request either way; widen the bins before reading them.

The control runs later than the incident. The dependency, the secret and the network may all have changed in between, so a clean control shows that the revision works now, not that it was innocent at 14:10.

Questions#

The errors stopped right after the rollback. Is that enough for the post-incident report?

It shows that something the rollback changed stopped the errors. Name the cause as the release only if the new revision fails again on fresh instances while the old revision's fresh instances serve cleanly, with both reading the same secret versions: the same pinned versions, or the same moving selector and no rotation between their instances' starts or in the minutes before them. An old revision whose errors did not fall supports that, but warm old instances can hide a dependency that fails only fresh ones. Otherwise record the restart and the dependency's recovery as alternatives.

Does a Cloud Run rollback restart the old revision?

It changes routing only. If the old revision still had traffic, its instances keep serving. If it had no traffic and no tag, Cloud Run will have removed its idle instances after a while, and the rollback starts new ones.

Should we wait to roll back until we have this evidence?

No. Roll back when it is the fastest way to stop user impact. Collect the evidence afterward, from metrics that cover the rollback minutes, before you decide whether to ship the same revision again.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading