Learning center

Cloud Run 429, 500 and 503 errors after shifting traffic to a new revision

When errors start right after a Cloud Run traffic step, find out first whether the failed requests reached a container. Requests aborted with The request was aborted because there was no available instance are a capacity problem: the new revision starting from zero, or reaching its share of the instance limit. If they stop within a few minutes as the new revision's instance count levels off, and its instances start about as fast as a fresh old-revision instance does today, the rollout's mechanics caused them. If they persist, compare the revisions: the new revision's latency rising while the old one's stays flat against the same dependencies points at the release, and both rising points at a dependency.

Two symptoms after a traffic step#

A Cloud Run rollout moves a share of traffic to a revision that was serving nothing a moment earlier, and the first minutes after each step look like a release failure:

  • The request log shows aborted requests from the new revision. At 14:03 traffic moves to checkout-00042, and within the minute its request log entries carry HTTP 500 or HTTP 429 with The request was aborted because there was no available instance. Cloud Run logs the 429 at severity WARNING when a maximum-instances limit stops scaling, and the 500 at severity ERROR when Cloud Run could not start instances as fast as requests arrived. Troubleshoot Cloud Run
  • Clients see errors that the error-rate chart does not. Callers report 429s, yet a chart built on run.googleapis.com/request_count shows almost no errors for either revision. That metric counts requests that reached a container instance, and excludes requests that did not, for example when the maximum number of instances was reached. Cloud Run metrics

The decision is whether to shift traffic back. Errors that come from the step itself fade without a rollback, and a rollback would make the old revision scale up again, with cold starts of its own. Errors that come from the new revision needing more capacity than the old one, or from a slower dependency, do not fade.

What a traffic step does to capacity#

A revision that receives no traffic scales in to its minimum instances, zero by default. When a step sends it traffic, it starts instances; the system log records each one as Starting new instance. Reason: followed by a reason such as DEPLOYMENT_ROLLOUT and a description. A request waiting for an instance stays queued for 3.5 times the service's average startup time, or 10 seconds, whichever is greater. Container runtime contract · Instance start reasons

Instance limits split with the traffic. A service-level maximum (--max) is divided among revisions in proportion to their traffic share: with a maximum of 20 and a 50/50 split, each revision may run 10, and a revision at its share returns the no-available-instance error even though the service as a whole has not reached 20. A revision-level maximum (--max-instances) is fixed on each revision when it is created, and Cloud Run sets 100 per revision by default. Maximum instances and traffic splitting

Minimum instances split too. A service-level minimum (--min) is divided among the revisions receiving traffic, so a new revision at a small step holds only part of the warm instances. A revision-level minimum (--min-instances) is set on each new revision and copied forward unless a deploy changes it. Minimum instances · gcloud run deploy flags

Each instance takes requests up to its maximum concurrency; a service created with the gcloud CLI defaults to 80 per vCPU. Code that holds each request longer, because it calls something slow or does more work, fills that concurrency sooner and needs more instances for the same traffic. When busy instances cannot process all their requests, Cloud Run lists an HTTP 503 with The Cloud Run service probably has reached its maximum container instance limit. Consider increasing this limit. and suggests a higher maximum or a lower concurrency. Maximum concurrency · Troubleshoot Cloud Run

Check 1: count the aborted requests from the request log#

Because request_count leaves out requests that never reached a container, count those from the request log. This lists them per revision with the status code, from just before the step:

gcloud logging read \
  'resource.type="cloud_run_revision"
   AND resource.labels.location="REGION"
   AND resource.labels.service_name="SERVICE"
   AND logName="projects/PROJECT/logs/run.googleapis.com%2Frequests"
   AND "The request was aborted because there was no available instance"
   AND timestamp>="START_TIME"' \
  --project PROJECT --order=asc --limit=1000 \
  --format='value(timestamp, resource.labels.revision_name, httpRequest.status, severity)'

The command returns at most 1,000 entries, oldest first. If it returns exactly 1,000, the list stops before the end of the window, and a quiet tail is the cap, not recovery: run it again with timestamp>= set to the last timestamp it printed until it returns fewer. To see when aborts stopped, read the newest entries instead with --order=desc --limit=50.

Read 429s and 500s apart. A 429 means a revision reached an instance limit, often its share of the service maximum. A 500 with the same text means instances could not start fast enough for the arriving requests, which is what a cold step looks like. gcloud logging read · Troubleshoot Cloud Run

Requests the application answered with an error do reach request_count, so split those by revision and code in Metrics Explorer:

sum by (revision_name, response_code) (rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE", response_code_class=~"4xx|5xx"}[5m]))

Check 2: set the errors beside instances and latency, per revision#

Three more series explain the aborts. Instance count, broken down by active and idle, shows when the new revision reached a steady size or a ceiling. Startup latency shows how long each new instance took. Pending latency covers the wait for an instance, including startup, and user execution latency covers time in your container only, so a rise in the second one is the code or what it calls. Cloud Run metrics · PromQL in Cloud Monitoring

sum by (revision_name, state) (run_googleapis_com:container_instance_count{monitored_resource="cloud_run_revision", service_name="SERVICE"})
histogram_quantile(0.95, sum by (revision_name, le) (rate(run_googleapis_com:container_startup_latencies_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m])))
sum by (revision_name) (rate(run_googleapis_com:request_latency_user_execution_sum{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))
/
sum by (revision_name) (rate(run_googleapis_com:request_latency_user_execution_count{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))

The mean is what sets instance demand: at the same concurrency, a revision whose requests take twice as long on average needs about twice the instances. Read the 95th percentile with histogram_quantile(0.95, ...) on the _bucket series as well, since a slow tail can regress while the mean barely moves.

The startup and user execution latency metrics are in beta. Compare each revision's settings at the same time, since --max-instances, --min-instances and concurrency are fixed per revision and a pipeline change can alter them without a code change:

gcloud run revisions describe REVISION \
  --project PROJECT --region REGION \
  --format='yaml(metadata.annotations, spec.containerConcurrency)'

Revision-level limits appear as the autoscaling.knative.dev/minScale and autoscaling.knative.dev/maxScale annotations; gcloud run services describe SERVICE prints the service-level Scaling: Auto (Min: …, Max: …). Revision-level minimum instances · Diff the two revisions

A worked fixture: one traffic step, two incidents#

The fixture is illustrative. A fictional service checkout has a service-level maximum of 20 instances. checkout-00041 serves all traffic on 12 active instances until 14:03 UTC, when traffic splits 50/50 with checkout-00042. Each revision may now run 10.

Incident A. Per-minute values for checkout-00042, with mean user execution latency for both revisions:

Minute (UTC)Active instancesAborted, 500Aborted, 429p95 startupMean user execution, 00041 / 00042
14:0343706.1 s41 / 43 ms
14:046906.3 s40 / 42 ms
14:0560041 / 42 ms
14:0860040 / 41 ms

All aborts are 500s, they fall in the two minutes after the step, and they stop once checkout-00042 holds the six instances half the traffic needs. Both revisions spend the same time per request on average, and their 95th percentiles match too. checkout-00041 started its instances in 6.0 s at p95 when it rolled out, and a fresh checkout-00041 instance started through a tag at 14:06 took 6.1 s against today's dependencies, so startup did not regress either. The step itself caused the errors. Continue the rollout, and for the next step raise the minimum instances or move traffic in smaller increments.

Incident B. Same step, same service:

Minute (UTC)Active instancesAborted, 500Aborted, 429p95 startupMean user execution, 00041 / 00042
14:0343506.2 s41 / 96 ms
14:049606.2 s40 / 95 ms
14:0510011841 / 97 ms
14:0810024040 / 96 ms

The first minute looks like Incident A. Then checkout-00042 reaches 10 instances, its share of the maximum, and the 429s grow. Its requests take more than twice as long as the old revision's against the same dependencies, so it needs more than twice the instances. checkout-00041 runs six instances with no aborts. Shift traffic back to checkout-00041, then find what the new code does per request that the old code did not.

Read the result#

What the checks showMost likely causeFirst move
500 aborts in the first minutes after a step, stopping as the instance count levels off; equal user execution latency; startup latency close to a fresh old-revision instance's todayThe rollout's mechanics: cold instancesContinue; warm the revision or use smaller steps
429 aborts that persist while the new revision sits at its share of the maximum; its user execution latency is higherThe release needs more capacity per requestShift traffic back, then compare the code path
The new revision's startup latency is well above the old one's when that one was cold, even if the aborts stop once instances are warmThe release starts more slowly, or a startup dependency has slowed sinceStart a fresh old-revision instance through a tag; if it starts fast, treat it as a release change, which minimum instances only hide
Revision-level minimum, maximum or concurrency differ between revisionsA configuration change in the releaseDiff the revisions
User execution latency rose on both revisions, and the old one aborts tooA dependency slowed downSplit errors by revision and look before the rollout

Where the comparison misleads#

Cloud Run samples these metrics every 60 seconds and shows them up to 120 seconds later, so the first minute after a step is one or two points. Cloud Run notes that a brief traffic spike may be visible only at a fine resolution. Read the request log for the exact seconds. Cloud Run metrics · Troubleshoot Cloud Run

The old revision is not a clean control during a step. Its traffic share fell, so it rarely needs new instances, and a problem that affects only starting instances, such as a secret read at startup, can look like a new-revision problem. A secret rotated during the rollout

More instances also mean more connections to whatever the service calls. If the new revision's extra instances push a database past its connection limit, both revisions slow down and the pattern looks like a dependency fault that the release triggered. Did our deploy overload a dependency?

A 500 or 503 whose log says the instance used too much memory and was terminated is not a capacity limit. Cloud Run warns that the next request will likely use a new instance, and the errors follow memory use rather than the traffic step. Containers killed for memory after a deploy

Questions#

Why does my Cloud Run error-rate chart miss the 429 errors clients see?

The request_count metric counts requests that reached a container instance. Requests that Cloud Run aborted because no instance was available, including those refused at a maximum-instances limit, are left out. Count them from the request log, where they carry the text The request was aborted because there was no available instance.

Does a 429 "no available instance" mean the service reached its maximum instances?

Not necessarily. With a service-level maximum and a traffic split, each revision gets a share of the maximum in proportion to its traffic, and a revision at its share returns the error while the service is below its total. A revision-level maximum caps that revision alone.

Do minimum instances protect the new revision during a traffic step?

A service-level minimum is divided among the revisions receiving traffic, so a new revision at a small step holds only part of it. A revision-level minimum applies to that revision as soon as it is referenced in a traffic split or carries a tag, and it is billed while idle.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading