Learning center

Test the previous Cloud Run revision against today's dependencies

Give the previous revision a traffic tag with gcloud run services update-traffic --update-tags. The tag gives that revision its own URL and moves no user traffic. Send the same failing request to the old and new revisions: if only the new one fails, on instances that started after the incident began, the release did. If the old one fails too, the dependency changed, either on its own or under load the current revision puts on it, so check this service's load on the dependency before ruling out a rollback.

When the old revision is no longer serving#

Splitting errors by revision works only while both revisions serve. These two situations leave you without that comparison:

  • The rollout already finished. checkout-00042 has served all traffic since 13:40 UTC. At 14:12 one call, POST /v1/quotes, starts returning 500 for about a third of requests, and its log lines read payments: upstream timeout after 2000ms. checkout-00041 serves nothing, so it has no errors to compare.
  • You are about to roll back. Cloud Run applies a rollback of traffic to the routing configuration immediately, without re-running deployment health checks or checking external dependencies first. If the old revision cannot work with today's database schema, secret or API, the rollback adds a second failure. Roll back to a previous revision

A traffic tag answers both. A tagged revision has its own URL, and requests to that URL reach it whatever its traffic percentage, including zero. Traffic tags

Tag the previous revision#

Confirm the revision still exists. Cloud Run keeps at most 1,000 revisions per service and deletes older ones automatically beyond that. Revisions

gcloud run revisions list --service SERVICE \
  --project PROJECT --region REGION

Add tags for the old revision and for the one serving now, so both requests go to a fixed revision instead of through the traffic split:

gcloud run services update-traffic SERVICE \
  --project PROJECT --region REGION \
  --update-tags prev=PREVIOUS_REVISION,cur=CURRENT_REVISION

This changes the service's traffic configuration, so it needs permission to update the service, but it does not change any traffic percentage. Use --update-tags, not --set-tags: --set-tags removes every existing tag first, including tags other teams or pipelines rely on. update-traffic flags

Each tag gets a URL that starts with the tag name and three hyphens, in the form https://prev---SERVICE-HASH.a.run.app. Read the exact URLs from the service:

gcloud run services describe SERVICE \
  --project PROJECT --region REGION \
  --flatten=status.traffic \
  --format='table(status.traffic.tag, status.traffic.revisionName, status.traffic.percent, status.traffic.url)'

Tag URLs · Traffic target fields

Send the same request to both revisions#

For a service that requires authentication, pass an identity token from your gcloud account, which needs the Cloud Run Invoker role on the service. Keep the token in a variable and out of shared terminal output. Test a private service

TOKEN="$(gcloud auth print-identity-token)"

curl -sS -o /dev/null -w 'prev %{http_code} %{time_total}s\n' \
  -H "Authorization: Bearer $TOKEN" PREV_TAG_URL/HEALTH_PATH

for TAG_URL in PREV_TAG_URL CUR_TAG_URL; do
  curl -sS -o /dev/null -w "$TAG_URL %{http_code} %{time_total}s\n" \
    -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
    --data @REQUEST_BODY.json "$TAG_URL/PATH"
done

The first call is a warm-up. A revision with no traffic runs at its minimum instances, zero by default, so the first request to it waits for an instance to start. Send the real request after it, and send it several times if the original failure was intermittent. Startup and scaling to zero

If the service's ingress setting accepts only internal or load-balancer traffic, a tag URL called from your laptop does not reach the revision. Run the test from inside the allowed network. gcloud run services proxy SERVICE --tag prev --port 8081 attaches credentials for you, but it too must be able to reach the service. Ingress settings · gcloud run services proxy

Then read each revision's log for the same minutes, to see whether the old revision failed at the same call:

gcloud logging read \
  'resource.type="cloud_run_revision"
   AND resource.labels.service_name="SERVICE"
   AND resource.labels.revision_name=("PREVIOUS_REVISION" OR "CURRENT_REVISION")
   AND timestamp>="START_TIME"' \
  --project PROJECT --order=asc --limit=200 \
  --format='value(timestamp, resource.labels.revision_name, httpRequest.status, textPayload, jsonPayload.message)'

Read the result#

Previous revisionCurrent revisionReading
Fails the same wayFailsThe dependency changed. The current revision still serves all traffic, so the old code meets any load it adds. If this service's calls, connections or work per call on the dependency rose with the release, a rollback may relieve it (did our deploy overload a dependency?); if they did not, a rollback will not help; investigate the dependency.
SucceedsFailsThe release changed how the service handles this request, provided the prev instance started recently (read its start in the system log) so it read today's secrets and opened fresh connections, both revisions reference their secrets the same way (a prev pinned to an old version and a cur on latest diverge after any rotation), and the failing cur instance also started well after the secret's latest version change, past the minutes latest can take to converge, so both read the same value. Rolling back to the tagged revision should clear it.
Fails differentlySucceedsThe old code no longer fits today's state, for example a migrated schema. A rollback would add a failure.
SucceedsSucceedsThe test missed the condition: a specific account, payload or load level. Return to the failing requests' logs.

To roll back after the second row, gcloud run services update-traffic SERVICE --to-tags prev=100 moves all traffic to the tagged revision. Migrate traffic to a tag

A worked fixture: the same quote request, two incidents#

The fixture is illustrative. It continues the symptom above: checkout-00042 serves all traffic, and POST /v1/quotes fails. At 14:20 UTC the on-call engineer tags checkout-00041 as prev and checkout-00042 as cur, then sends a quote request for a test account. A quote creates no order and charges nothing.

Incident A.

Time (UTC)TagRequestStatusDurationLog line
14:21:05prevWarm-up GET /healthz2007.9 s
14:21:20prevPOST /v1/quotes5002.1 spayments: upstream timeout after 2000ms
14:21:24curPOST /v1/quotes5002.0 spayments: upstream timeout after 2000ms

Code that served cleanly until 13:40 now times out at the same payments call. Both revisions still meet the load checkout-00042 puts on the payments API, so the engineer checks that load too: quote requests per minute, calls to the payments API per quote request, attempts per call, open connections and new connections per minute to it have been flat since 13:00, before and after the rollout, the payments team's per-client processing time for checkout is flat too, and the payments API's own dashboard shows latency rising from 14:08 for every caller. Today's payments API is the difference, and a rollback would leave the error where it is. Check the payments API's own health and whether the provider has an incident. Rule a cloud provider incident in or out

Incident B.

Time (UTC)TagRequestStatusDurationLog line
14:21:05prevWarm-up GET /healthz2008.1 s
14:21:20prevPOST /v1/quotes2000.31 s
14:21:24curPOST /v1/quotes5002.0 spayments: upstream timeout after 2000ms

The prev instance started at 14:20:58 for the warm-up, so it read today's secrets and opened its own connections. Both revisions pin the payments credential to the same version, and the failing cur instance started at 14:19:40, so both read the same value. The old code reaches the same payments API in a third of a second. The new revision differs in something about that call: its timeout, its endpoint, or the request it sends. cur's instance was serving a few requests a second at the time, with its connection pool far from full, so load does not explain the gap. Move traffic to prev, then compare the revisions. Diff the two revisions

What the test cannot show#

The tagged revision runs the old code under today's conditions, not under the conditions of the day it last served. Its new instances read secrets exposed as environment variables when they start, so a reference to latest gives them today's version. If a rotated secret broke the service, the old revision fails too, and this test reads it as a dependency change, which it is. Secrets in Cloud Run · A secret rotated during the rollout

Replaying a request runs it for real. A request that creates an order, charges a card, sends a message or writes a record does so against production, from either revision. Choose a read-only request, a test account or an idempotency key the backend honors. If the failing request has only writing forms, test the dependency call some other way.

Some failures come from Cloud Run, not from the revision's code. A 401 or 403 means the token or the invoker permission was refused, and a 404 with no request log entry means an ingress restriction or a disabled run.app URL stopped the request. Neither says anything about the dependency. Troubleshoot Cloud Run

Tags keep idle instances running in one setup. With a revision-level minimum, Cloud Run starts and keeps that minimum for every tagged revision; with a service-level minimum, tags allocate no instances. Remove the tags when you finish with gcloud run services update-traffic SERVICE --remove-tags prev,cur. Tags and minimum instances

A single test request is evidence about one request at one moment. For the general method of comparing revisions and the minutes before a rollout, see bad release or unhealthy dependency.

Questions#

Does tagging a Cloud Run revision send it user traffic?

No. A tag gives the revision a separate URL. Its traffic percentage stays as it was, and changing tags does not change any revision's percentage. Only requests sent to the tag URL reach it.

What is the URL of a tagged Cloud Run revision?

The tag name, three hyphens, then the service's run.app host, such as https://prev---checkout-abcdef.a.run.app. gcloud run services describe lists each tag's URL under the service's traffic status.

Why was the first request to the tagged revision slow?

A revision without traffic scales in to its minimum instances, which is zero unless you set one. The first request waits for an instance to start. Send a warm-up request and compare the requests that follow.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading