---
title: "A secret rotated during the rollout breaks new instances of both revisions | Proofline"
description: "A secret that changes during a rollout reaches every new instance through a moving reference such as latest or AWSCURRENT, whatever its revision."
url: https://proofline.sh/learn/secret-rotation-during-rollout/
---

[Learning center](/learn/)

# A secret rotated during the rollout breaks new instances of both revisions

A secret that changes during a rollout reaches every new instance through a moving reference such as latest or AWSCURRENT, whatever its revision. Instances started earlier keep the old value. Sort the failures by when each instance started instead of by revision. The secret is the cause, and rolling back will not fix it, when three things hold: the failures begin at the moment of a Secret Manager version change or a Secrets Manager rotation event in CloudTrail, both revisions reference the secret the same moving way, the failure is the credential itself, such as a refused login, and fresh instances of both revisions fail while older instances of both succeed. A revision pinned to a version number does not move with a rotation.

Missing secrets The Proofline team November 20, 2026 8 min read

## Two symptoms that look like the release

A rollout starts many instances in a few minutes, so anything that affects only new instances shows up first in the new revision:

-   **A Cloud Run service fails on its new revision, then on an old instance.** Traffic moves 10% to `checkout-00042` at 14:03 UTC, and every request it serves fails to authenticate to the database. At 14:06 `checkout-00041` scales out by one instance, and that instance fails the same way while its older instances keep serving.
-   **An ECS rollback fails like the deployment it replaced.** Tasks from task definition `checkout:42` fail their database login, the deployment circuit breaker rolls the service back to `checkout:41`, and the tasks the rollback starts fail too. If the secret could not be read at all, the tasks stop with `ResourceInitializationError` instead. [Fix ECS ResourceInitializationError](/learn/ecs-secrets-startup-errors/) · [Why the circuit breaker rolled back](/learn/ecs-circuit-breaker-rollback-cause/)

In both cases both revisions read the secret through `latest` or `AWSCURRENT`, the release is innocent, and the revision split from [the general method](/learn/bad-release-or-unhealthy-dependency/#split-by-revision) points at it anyway, because the old revision's errors come late and from few instances.

## When each platform reads the secret

On Cloud Run, a secret exposed as an environment variable is resolved when the instance starts. Cloud Run retrieves it before starting the instance, and if retrieval fails the instance does not start. A secret mounted as a volume is fetched from Secret Manager when the file is read, so running instances that reference `latest` see a new version, and an unreadable secret fails the read, not the startup. [Secrets in Cloud Run](https://cloud.google.com/run/docs/configuring/services/secrets)

The version reference decides which value a new instance gets. `latest` is an alias for the most recently created version, so adding a version changes what every later instance start reads. Google recommends pinning environment-variable secrets to a version number for this reason. Reads through `latest` are eventually consistent, typically converging within minutes, so instances started just after a change may get either version. [`latest` alias](https://cloud.google.com/secret-manager/docs/reference/rest/v1/projects.secrets.versions/access) · [Pin versions](https://cloud.google.com/secret-manager/docs/best-practices) · [Consistency](https://cloud.google.com/secret-manager/docs/access-secret-version)

On ECS, secrets in a container definition's `secrets` are injected when the container starts. A running container keeps that value after a rotation; a new task reads whatever its `valueFrom` selects at its own start, `AWSCURRENT` when no version is given. [ECS secret injection](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html)

Both platforms therefore tie the value to the instance's start time. The question this guide settles is whether the failures follow that time or follow the revision.

## Check 1: sort the failures by instance start time

**On Cloud Run**, each container log entry carries `labels.instanceId` beside the revision. List the failing entries with both, using text from the failure:

```
gcloud logging read \
  'resource.type="cloud_run_revision"
   AND resource.labels.location="REGION"
   AND resource.labels.service_name="SERVICE"
   AND "ERROR_TEXT"
   AND timestamp>="START_TIME"' \
  --project PROJECT --order=asc --limit=1000 \
  --format='value(timestamp, resource.labels.revision_name, labels.instanceId)'
```

Run it again with the line your application writes at startup, such as its listening message, in place of `ERROR_TEXT`. Its first entry for each `instanceId` gives the instance's start time. The system log also records each start as `Starting new instance` with a reason such as `AUTOSCALING` or `DEPLOYMENT_ROLLOUT`. [Cloud Run log fields](https://cloud.google.com/run/docs/logging)

**On ECS**, list the service's task IDs with `aws ecs list-tasks`, once with `--desired-status RUNNING` and once with `STOPPED`, and order the tasks by creation time. Stopped tasks remain visible for at least an hour. Then count errors per task with the per-task Logs Insights query from the general method linked above. [describe-tasks](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html)

```
aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --tasks TASK_ID_1 TASK_ID_2 TASK_ID_3 \
  --query 'sort_by(tasks, &(startedAt || createdAt))[].{task:taskArn,definition:taskDefinitionArn,created:createdAt,started:startedAt,stopped:stoppedAt,reason:stoppedReason}' \
  --output table --no-cli-pager
```

A secret problem draws a line in time: instances or tasks started after it fail, from every revision, and those started before it work. An ECS task reads its secrets between its `createdAt` and `startedAt`, so a task whose span straddles the change settles nothing. A release problem draws the line between revisions.

## Check 2: find when the secret changed

**In Secret Manager**, list the versions with their creation and destruction times. A version's record has no time for being disabled or enabled. [Secret version fields](https://cloud.google.com/secret-manager/docs/reference/rest/v1/projects.secrets.versions) · [`gcloud secrets versions list`](https://cloud.google.com/sdk/gcloud/reference/secrets/versions/list)

```
gcloud secrets versions list SECRET \
  --project PROJECT \
  --format='table(name, state, createTime, destroyTime)'
```

The Admin Activity audit log, which Google always writes, records enabling, disabling and destroying a version, with its time and caller. Adding a version (`AddSecretVersion`) and reading one (`AccessSecretVersion`) go to the Data Access log, which is off by default. Without it the query below finds no addition, and that is not evidence that none happened: the version's `createTime` in the listing above still dates it, though not who added it. You also usually cannot see which version a given instance read. [Secret Manager audit logging](https://cloud.google.com/secret-manager/docs/audit-logging) · [Audit log types](https://cloud.google.com/logging/docs/audit/configure-data-access)

```
gcloud logging read \
  'protoPayload.serviceName="secretmanager.googleapis.com"
   AND protoPayload.resourceName:"secrets/SECRET"
   AND protoPayload.methodName=(
     "google.cloud.secretmanager.v1.SecretManagerService.AddSecretVersion" OR
     "google.cloud.secretmanager.v1.SecretManagerService.EnableSecretVersion" OR
     "google.cloud.secretmanager.v1.SecretManagerService.DisableSecretVersion" OR
     "google.cloud.secretmanager.v1.SecretManagerService.DestroySecretVersion")
   AND timestamp>="START_TIME"' \
  --project PROJECT --order=asc \
  --format='table(timestamp, protoPayload.methodName, protoPayload.resourceName, protoPayload.authenticationInfo.principalEmail)'
```

**In AWS Secrets Manager**, CloudTrail records each API call and the rotation service's own events. `PutSecretValue` without staging labels moves `AWSCURRENT` to the new version at once. A rotation function moves `AWSCURRENT` in its final step with `UpdateSecretVersionStage`, and that moment is when new tasks start reading the new value. `RotationStarted`, `RotationSucceeded` and `RotationFailed` frame a scheduled or requested rotation. [CloudTrail entries for Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/cloudtrail_log_entries.html) · [`PutSecretValue`](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_PutSecretValue.html) · [Rotation function steps](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotate-secrets_lambda-functions.html)

```
aws cloudtrail lookup-events \
  --profile PROFILE --region REGION \
  --lookup-attributes AttributeKey=EventSource,AttributeValue=secretsmanager.amazonaws.com \
  --start-time START_TIME --end-time END_TIME \
  --query 'Events[?contains(`["PutSecretValue","UpdateSecretVersionStage","RotateSecret","RotationStarted","RotationSucceeded","RotationFailed","RotationAbandoned"]`, EventName)].{time:EventTime,event:EventName,user:Username,resources:Resources[].ResourceName}' \
  --output table --no-cli-pager
```

Event history covers management events from the last 90 days in one Region, so run it in the secret's Region. `aws secretsmanager list-secret-version-ids --include-deprecated` adds each version's `CreatedDate` and staging labels. [lookup-events](https://docs.aws.amazon.com/cli/latest/reference/cloudtrail/lookup-events.html) · [Version listing](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/list-secret-version-ids.html)

## A worked fixture: one database error, two causes

The fixture is illustrative. A fictional Cloud Run service `checkout` reads `DB_PASSWORD` from the secret `checkout-db-password` at version `latest`. Traffic moves 10% to `checkout-00042` at 14:03 UTC. Requests per instance between 14:03 and 14:10, and how many of them failed to authenticate to the database:

| Instance | Revision | First log entry (UTC) | Requests | Failed, case A | Failed, case B |
| --- | --- | --- | --- | --- | --- |
| `…a1` | `checkout-00041` | 09:12 | 2,310 | 0 | 0 |
| `…a2` | `checkout-00041` | 11:47 | 2,295 | 0 | 0 |
| `…b1` | `checkout-00042` | 14:03 | 268 | 268 | 268 |
| `…b2` | `checkout-00042` | 14:04 | 251 | 251 | 251 |
| `…a3` | `checkout-00041` | 14:06 | 412 | 412 | 0 |

**Case A, the secret.** The audit log shows `AddSecretVersion` creating version 8 at 13:58:12, called by the rotation job's service account. Every instance that started after 13:58 fails, including `…a3` from the old revision, and every instance that started earlier works. Version 8 holds a password the database does not accept yet. Moving traffic back to `checkout-00041` would send it to instances that must scale out, and each new one would fail. Fix the credential instead: complete the database side of the rotation, or add a version with the working value. Disabling version 8 does not send `latest` back to version 7, because `latest` names the most recently created version, and a disabled version cannot be accessed. The next instance start would then fail to read the secret at all.

**Case B, the release.** The newest version of each secret the service reads was created the previous day, long enough before 14:03 for `latest` to have converged, and `…a3` serves cleanly against the same database. Only the new revision's instances fail, from their first request. `checkout-00042` reads a different secret or version, or uses the value differently. Shift traffic back, then compare the revisions' secret references. [Diff the two revisions](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

## Where the pattern blurs

Some secrets are not read at startup. A Cloud Run volume is read when the file is opened, and an application that fetches a secret through an SDK has its own cache. Failures then follow the read or the cache expiry, and they can hit old instances at any moment.

A rotation that also changes the credential on the database side reaches running instances later. Their open connections may keep working while new connections made with the old value fail as pools reconnect, so the old revision's errors arrive in a slow trickle behind the new instances'.

Without many new instances there is little to sort. A service that scales rarely may show only the rollout's instances failing, which looks exactly like a release. The secret timeline from Check 2 then carries the argument alone, and testing the old revision through a tag reads today's secret version only on an instance that started after the change had converged; check the instance's start in the system log. [Test the previous revision with a tag](/learn/test-previous-cloud-run-revision-with-tag/)

## Questions

### Does a Cloud Run service pick up a new secret version without a redeploy?

A secret exposed as an environment variable is resolved when each instance starts, so only instances started after the change get the new version, from any revision that references `latest`. A secret mounted as a volume at `latest` is fetched from Secret Manager when the file is read, so running instances see the new version.

### Why did my ECS rollback fail with the same error as the deployment?

A circuit-breaker rollback starts a new deployment of the last completed one, and any task it launches reads its secrets when its containers start. If the secret changed before the rollback, those tasks read the new value too. Check that both task definitions' `valueFrom` select the same version stage, then compare the tasks' `createdAt` to `startedAt` spans with the secret's `PutSecretValue` or `UpdateSecretVersionStage` events in CloudTrail.

### Should Cloud Run services reference secret versions by number?

Google's Secret Manager guidance recommends pinning a version number instead of `latest`, and Cloud Run's documentation recommends it for secrets passed as environment variables. A new version then reaches the service only through a deploy, which your release process can check and roll back.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Fix ECS ResourceInitializationError when pulling secrets](/learn/ecs-secrets-startup-errors/)
    
    Missing secrets The Proofline team November 15, 2026 8 min read
    
-   ### [Test the previous Cloud Run revision against today's dependencies](/learn/test-previous-cloud-run-revision-with-tag/)
    
    Deployment failures The Proofline team November 20, 2026 8 min read
