---
title: "Cloud Run 429, 500 and 503 errors after shifting traffic to a new revision | Proofline"
description: "When errors start right after a Cloud Run traffic step, find out first whether the failed requests reached a container."
url: https://proofline.sh/learn/cloud-run-errors-after-traffic-shift/
---

[Learning center](/learn/)

# Cloud Run 429, 500 and 503 errors after shifting traffic to a new revision

When errors start right after a Cloud Run traffic step, find out first whether the failed requests reached a container. Requests aborted with The request was aborted because there was no available instance are a capacity problem: the new revision starting from zero, or reaching its share of the instance limit. If they stop within a few minutes as the new revision's instance count levels off, and its instances start about as fast as a fresh old-revision instance does today, the rollout's mechanics caused them. If they persist, compare the revisions: the new revision's latency rising while the old one's stays flat against the same dependencies points at the release, and both rising points at a dependency.

Capacity overruns The Proofline team November 20, 2026 9 min read

## Two symptoms after a traffic step

A Cloud Run rollout moves a share of traffic to a revision that was serving nothing a moment earlier, and the first minutes after each step look like a release failure:

-   **The request log shows aborted requests from the new revision.** At 14:03 traffic moves to `checkout-00042`, and within the minute its request log entries carry HTTP 500 or HTTP 429 with `The request was aborted because there was no available instance.` Cloud Run logs the 429 at severity `WARNING` when a maximum-instances limit stops scaling, and the 500 at severity `ERROR` when Cloud Run could not start instances as fast as requests arrived. [Troubleshoot Cloud Run](https://cloud.google.com/run/docs/troubleshooting)
-   **Clients see errors that the error-rate chart does not.** Callers report 429s, yet a chart built on `run.googleapis.com/request_count` shows almost no errors for either revision. That metric counts requests that reached a container instance, and excludes requests that did not, for example when the maximum number of instances was reached. [Cloud Run metrics](https://cloud.google.com/monitoring/api/metrics_gcp_p_z#gcp-run)

The decision is whether to shift traffic back. Errors that come from the step itself fade without a rollback, and a rollback would make the old revision scale up again, with cold starts of its own. Errors that come from the new revision needing more capacity than the old one, or from a slower dependency, do not fade.

## What a traffic step does to capacity

A revision that receives no traffic scales in to its minimum instances, zero by default. When a step sends it traffic, it starts instances; the system log records each one as `Starting new instance. Reason:` followed by a reason such as `DEPLOYMENT_ROLLOUT` and a description. A request waiting for an instance stays queued for 3.5 times the service's average startup time, or 10 seconds, whichever is greater. [Container runtime contract](https://cloud.google.com/run/docs/container-contract) · [Instance start reasons](https://cloud.google.com/run/docs/logging)

Instance limits split with the traffic. A service-level maximum (`--max`) is divided among revisions in proportion to their traffic share: with a maximum of 20 and a 50/50 split, each revision may run 10, and a revision at its share returns the no-available-instance error even though the service as a whole has not reached 20. A revision-level maximum (`--max-instances`) is fixed on each revision when it is created, and Cloud Run sets 100 per revision by default. [Maximum instances and traffic splitting](https://cloud.google.com/run/docs/configuring/max-instances)

Minimum instances split too. A service-level minimum (`--min`) is divided among the revisions receiving traffic, so a new revision at a small step holds only part of the warm instances. A revision-level minimum (`--min-instances`) is set on each new revision and copied forward unless a deploy changes it. [Minimum instances](https://cloud.google.com/run/docs/configuring/min-instances) · [`gcloud run deploy` flags](https://cloud.google.com/sdk/gcloud/reference/run/deploy)

Each instance takes requests up to its maximum concurrency; a service created with the gcloud CLI defaults to 80 per vCPU. Code that holds each request longer, because it calls something slow or does more work, fills that concurrency sooner and needs more instances for the same traffic. When busy instances cannot process all their requests, Cloud Run lists an HTTP 503 with `The Cloud Run service probably has reached its maximum container instance limit. Consider increasing this limit.` and suggests a higher maximum or a lower concurrency. [Maximum concurrency](https://cloud.google.com/run/docs/about-concurrency) · [Troubleshoot Cloud Run](https://cloud.google.com/run/docs/troubleshooting)

## Check 1: count the aborted requests from the request log

Because `request_count` leaves out requests that never reached a container, count those from the request log. This lists them per revision with the status code, from just before the step:

```
gcloud logging read \
  'resource.type="cloud_run_revision"
   AND resource.labels.location="REGION"
   AND resource.labels.service_name="SERVICE"
   AND logName="projects/PROJECT/logs/run.googleapis.com%2Frequests"
   AND "The request was aborted because there was no available instance"
   AND timestamp>="START_TIME"' \
  --project PROJECT --order=asc --limit=1000 \
  --format='value(timestamp, resource.labels.revision_name, httpRequest.status, severity)'
```

The command returns at most 1,000 entries, oldest first. If it returns exactly 1,000, the list stops before the end of the window, and a quiet tail is the cap, not recovery: run it again with `timestamp>=` set to the last timestamp it printed until it returns fewer. To see when aborts stopped, read the newest entries instead with `--order=desc --limit=50`.

Read 429s and 500s apart. A 429 means a revision reached an instance limit, often its share of the service maximum. A 500 with the same text means instances could not start fast enough for the arriving requests, which is what a cold step looks like. [`gcloud logging read`](https://cloud.google.com/sdk/gcloud/reference/logging/read) · [Troubleshoot Cloud Run](https://cloud.google.com/run/docs/troubleshooting)

Requests the application answered with an error do reach `request_count`, so split those by revision and code in Metrics Explorer:

```
sum by (revision_name, response_code) (rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE", response_code_class=~"4xx|5xx"}[5m]))
```

## Check 2: set the errors beside instances and latency, per revision

Three more series explain the aborts. Instance count, broken down by `active` and `idle`, shows when the new revision reached a steady size or a ceiling. Startup latency shows how long each new instance took. Pending latency covers the wait for an instance, including startup, and user execution latency covers time in your container only, so a rise in the second one is the code or what it calls. [Cloud Run metrics](https://cloud.google.com/monitoring/api/metrics_gcp_p_z#gcp-run) · [PromQL in Cloud Monitoring](https://cloud.google.com/monitoring/promql)

```
sum by (revision_name, state) (run_googleapis_com:container_instance_count{monitored_resource="cloud_run_revision", service_name="SERVICE"})
```

```
histogram_quantile(0.95, sum by (revision_name, le) (rate(run_googleapis_com:container_startup_latencies_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m])))
```

```
sum by (revision_name) (rate(run_googleapis_com:request_latency_user_execution_sum{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))
/
sum by (revision_name) (rate(run_googleapis_com:request_latency_user_execution_count{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))
```

The mean is what sets instance demand: at the same concurrency, a revision whose requests take twice as long on average needs about twice the instances. Read the 95th percentile with `histogram_quantile(0.95, ...)` on the `_bucket` series as well, since a slow tail can regress while the mean barely moves.

The startup and user execution latency metrics are in beta. Compare each revision's settings at the same time, since `--max-instances`, `--min-instances` and concurrency are fixed per revision and a pipeline change can alter them without a code change:

```
gcloud run revisions describe REVISION \
  --project PROJECT --region REGION \
  --format='yaml(metadata.annotations, spec.containerConcurrency)'
```

Revision-level limits appear as the `autoscaling.knative.dev/minScale` and `autoscaling.knative.dev/maxScale` annotations; `gcloud run services describe SERVICE` prints the service-level `Scaling: Auto (Min: …, Max: …)`. [Revision-level minimum instances](https://cloud.google.com/run/docs/configuring/min-instances) · [Diff the two revisions](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

## A worked fixture: one traffic step, two incidents

The fixture is illustrative. A fictional service `checkout` has a service-level maximum of 20 instances. `checkout-00041` serves all traffic on 12 active instances until 14:03 UTC, when traffic splits 50/50 with `checkout-00042`. Each revision may now run 10.

**Incident A.** Per-minute values for `checkout-00042`, with mean user execution latency for both revisions:

| Minute (UTC) | Active instances | Aborted, 500 | Aborted, 429 | p95 startup | Mean user execution, 00041 / 00042 |
| --- | --- | --- | --- | --- | --- |
| 14:03 | 4 | 37 | 0 | 6.1 s | 41 / 43 ms |
| 14:04 | 6 | 9 | 0 | 6.3 s | 40 / 42 ms |
| 14:05 | 6 | 0 | 0 |  | 41 / 42 ms |
| 14:08 | 6 | 0 | 0 |  | 40 / 41 ms |

All aborts are 500s, they fall in the two minutes after the step, and they stop once `checkout-00042` holds the six instances half the traffic needs. Both revisions spend the same time per request on average, and their 95th percentiles match too. `checkout-00041` started its instances in 6.0 s at p95 when it rolled out, and a fresh `checkout-00041` instance started through a tag at 14:06 took 6.1 s against today's dependencies, so startup did not regress either. The step itself caused the errors. Continue the rollout, and for the next step raise the minimum instances or move traffic in smaller increments.

**Incident B.** Same step, same service:

| Minute (UTC) | Active instances | Aborted, 500 | Aborted, 429 | p95 startup | Mean user execution, 00041 / 00042 |
| --- | --- | --- | --- | --- | --- |
| 14:03 | 4 | 35 | 0 | 6.2 s | 41 / 96 ms |
| 14:04 | 9 | 6 | 0 | 6.2 s | 40 / 95 ms |
| 14:05 | 10 | 0 | 118 |  | 41 / 97 ms |
| 14:08 | 10 | 0 | 240 |  | 40 / 96 ms |

The first minute looks like Incident A. Then `checkout-00042` reaches 10 instances, its share of the maximum, and the 429s grow. Its requests take more than twice as long as the old revision's against the same dependencies, so it needs more than twice the instances. `checkout-00041` runs six instances with no aborts. Shift traffic back to `checkout-00041`, then find what the new code does per request that the old code did not.

## Read the result

| What the checks show | Most likely cause | First move |
| --- | --- | --- |
| 500 aborts in the first minutes after a step, stopping as the instance count levels off; equal user execution latency; startup latency close to a fresh old-revision instance's today | The rollout's mechanics: cold instances | Continue; warm the revision or use smaller steps |
| 429 aborts that persist while the new revision sits at its share of the maximum; its user execution latency is higher | The release needs more capacity per request | Shift traffic back, then compare the code path |
| The new revision's startup latency is well above the old one's when that one was cold, even if the aborts stop once instances are warm | The release starts more slowly, or a startup dependency has slowed since | Start a fresh old-revision instance through a tag; if it starts fast, treat it as a release change, which minimum instances only hide |
| Revision-level minimum, maximum or concurrency differ between revisions | A configuration change in the release | Diff the revisions |
| User execution latency rose on both revisions, and the old one aborts too | A dependency slowed down | [Split errors by revision and look before the rollout](/learn/bad-release-or-unhealthy-dependency/#split-by-revision) |

## Where the comparison misleads

Cloud Run samples these metrics every 60 seconds and shows them up to 120 seconds later, so the first minute after a step is one or two points. Cloud Run notes that a brief traffic spike may be visible only at a fine resolution. Read the request log for the exact seconds. [Cloud Run metrics](https://cloud.google.com/monitoring/api/metrics_gcp_p_z#gcp-run) · [Troubleshoot Cloud Run](https://cloud.google.com/run/docs/troubleshooting)

The old revision is not a clean control during a step. Its traffic share fell, so it rarely needs new instances, and a problem that affects only starting instances, such as a secret read at startup, can look like a new-revision problem. [A secret rotated during the rollout](/learn/secret-rotation-during-rollout/)

More instances also mean more connections to whatever the service calls. If the new revision's extra instances push a database past its connection limit, both revisions slow down and the pattern looks like a dependency fault that the release triggered. [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/)

A 500 or 503 whose log says the instance used too much memory and was terminated is not a capacity limit. Cloud Run warns that the next request will likely use a new instance, and the errors follow memory use rather than the traffic step. [Containers killed for memory after a deploy](/learn/container-memory-kills-after-deploy/)

## Questions

### Why does my Cloud Run error-rate chart miss the 429 errors clients see?

The `request_count` metric counts requests that reached a container instance. Requests that Cloud Run aborted because no instance was available, including those refused at a maximum-instances limit, are left out. Count them from the request log, where they carry the text `The request was aborted because there was no available instance`.

### Does a 429 "no available instance" mean the service reached its maximum instances?

Not necessarily. With a service-level maximum and a traffic split, each revision gets a share of the maximum in proportion to its traffic, and a revision at its share returns the error while the service is below its total. A revision-level maximum caps that revision alone.

### Do minimum instances protect the new revision during a traffic step?

A service-level minimum is divided among the revisions receiving traffic, so a new revision at a small step holds only part of it. A revision-level minimum applies to that revision as soon as it is referenced in a traffic split or carries a tag, and it is billed while idle.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Did our deploy overload a dependency that was healthy before it?](/learn/did-our-deploy-overload-a-dependency/)
    
    Capacity overruns The Proofline team November 20, 2026 10 min read
    
-   ### [Containers killed for memory after a deploy: leak or backlog](/learn/container-memory-kills-after-deploy/)
    
    Capacity overruns The Proofline team November 20, 2026 8 min read
