Learning center

A secret rotated during the rollout breaks new instances of both revisions

A secret that changes during a rollout reaches every new instance through a moving reference such as latest or AWSCURRENT, whatever its revision. Instances started earlier keep the old value. Sort the failures by when each instance started instead of by revision. The secret is the cause, and rolling back will not fix it, when three things hold: the failures begin at the moment of a Secret Manager version change or a Secrets Manager rotation event in CloudTrail, both revisions reference the secret the same moving way, the failure is the credential itself, such as a refused login, and fresh instances of both revisions fail while older instances of both succeed. A revision pinned to a version number does not move with a rotation.

Two symptoms that look like the release#

A rollout starts many instances in a few minutes, so anything that affects only new instances shows up first in the new revision:

  • A Cloud Run service fails on its new revision, then on an old instance. Traffic moves 10% to checkout-00042 at 14:03 UTC, and every request it serves fails to authenticate to the database. At 14:06 checkout-00041 scales out by one instance, and that instance fails the same way while its older instances keep serving.
  • An ECS rollback fails like the deployment it replaced. Tasks from task definition checkout:42 fail their database login, the deployment circuit breaker rolls the service back to checkout:41, and the tasks the rollback starts fail too. If the secret could not be read at all, the tasks stop with ResourceInitializationError instead. Fix ECS ResourceInitializationError · Why the circuit breaker rolled back

In both cases both revisions read the secret through latest or AWSCURRENT, the release is innocent, and the revision split from the general method points at it anyway, because the old revision's errors come late and from few instances.

When each platform reads the secret#

On Cloud Run, a secret exposed as an environment variable is resolved when the instance starts. Cloud Run retrieves it before starting the instance, and if retrieval fails the instance does not start. A secret mounted as a volume is fetched from Secret Manager when the file is read, so running instances that reference latest see a new version, and an unreadable secret fails the read, not the startup. Secrets in Cloud Run

The version reference decides which value a new instance gets. latest is an alias for the most recently created version, so adding a version changes what every later instance start reads. Google recommends pinning environment-variable secrets to a version number for this reason. Reads through latest are eventually consistent, typically converging within minutes, so instances started just after a change may get either version. latest alias · Pin versions · Consistency

On ECS, secrets in a container definition's secrets are injected when the container starts. A running container keeps that value after a rotation; a new task reads whatever its valueFrom selects at its own start, AWSCURRENT when no version is given. ECS secret injection

Both platforms therefore tie the value to the instance's start time. The question this guide settles is whether the failures follow that time or follow the revision.

Check 1: sort the failures by instance start time#

On Cloud Run, each container log entry carries labels.instanceId beside the revision. List the failing entries with both, using text from the failure:

gcloud logging read \
  'resource.type="cloud_run_revision"
   AND resource.labels.location="REGION"
   AND resource.labels.service_name="SERVICE"
   AND "ERROR_TEXT"
   AND timestamp>="START_TIME"' \
  --project PROJECT --order=asc --limit=1000 \
  --format='value(timestamp, resource.labels.revision_name, labels.instanceId)'

Run it again with the line your application writes at startup, such as its listening message, in place of ERROR_TEXT. Its first entry for each instanceId gives the instance's start time. The system log also records each start as Starting new instance with a reason such as AUTOSCALING or DEPLOYMENT_ROLLOUT. Cloud Run log fields

On ECS, list the service's task IDs with aws ecs list-tasks, once with --desired-status RUNNING and once with STOPPED, and order the tasks by creation time. Stopped tasks remain visible for at least an hour. Then count errors per task with the per-task Logs Insights query from the general method linked above. describe-tasks

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --tasks TASK_ID_1 TASK_ID_2 TASK_ID_3 \
  --query 'sort_by(tasks, &(startedAt || createdAt))[].{task:taskArn,definition:taskDefinitionArn,created:createdAt,started:startedAt,stopped:stoppedAt,reason:stoppedReason}' \
  --output table --no-cli-pager

A secret problem draws a line in time: instances or tasks started after it fail, from every revision, and those started before it work. An ECS task reads its secrets between its createdAt and startedAt, so a task whose span straddles the change settles nothing. A release problem draws the line between revisions.

Check 2: find when the secret changed#

In Secret Manager, list the versions with their creation and destruction times. A version's record has no time for being disabled or enabled. Secret version fields · gcloud secrets versions list

gcloud secrets versions list SECRET \
  --project PROJECT \
  --format='table(name, state, createTime, destroyTime)'

The Admin Activity audit log, which Google always writes, records enabling, disabling and destroying a version, with its time and caller. Adding a version (AddSecretVersion) and reading one (AccessSecretVersion) go to the Data Access log, which is off by default. Without it the query below finds no addition, and that is not evidence that none happened: the version's createTime in the listing above still dates it, though not who added it. You also usually cannot see which version a given instance read. Secret Manager audit logging · Audit log types

gcloud logging read \
  'protoPayload.serviceName="secretmanager.googleapis.com"
   AND protoPayload.resourceName:"secrets/SECRET"
   AND protoPayload.methodName=(
     "google.cloud.secretmanager.v1.SecretManagerService.AddSecretVersion" OR
     "google.cloud.secretmanager.v1.SecretManagerService.EnableSecretVersion" OR
     "google.cloud.secretmanager.v1.SecretManagerService.DisableSecretVersion" OR
     "google.cloud.secretmanager.v1.SecretManagerService.DestroySecretVersion")
   AND timestamp>="START_TIME"' \
  --project PROJECT --order=asc \
  --format='table(timestamp, protoPayload.methodName, protoPayload.resourceName, protoPayload.authenticationInfo.principalEmail)'

In AWS Secrets Manager, CloudTrail records each API call and the rotation service's own events. PutSecretValue without staging labels moves AWSCURRENT to the new version at once. A rotation function moves AWSCURRENT in its final step with UpdateSecretVersionStage, and that moment is when new tasks start reading the new value. RotationStarted, RotationSucceeded and RotationFailed frame a scheduled or requested rotation. CloudTrail entries for Secrets Manager · PutSecretValue · Rotation function steps

aws cloudtrail lookup-events \
  --profile PROFILE --region REGION \
  --lookup-attributes AttributeKey=EventSource,AttributeValue=secretsmanager.amazonaws.com \
  --start-time START_TIME --end-time END_TIME \
  --query 'Events[?contains(`["PutSecretValue","UpdateSecretVersionStage","RotateSecret","RotationStarted","RotationSucceeded","RotationFailed","RotationAbandoned"]`, EventName)].{time:EventTime,event:EventName,user:Username,resources:Resources[].ResourceName}' \
  --output table --no-cli-pager

Event history covers management events from the last 90 days in one Region, so run it in the secret's Region. aws secretsmanager list-secret-version-ids --include-deprecated adds each version's CreatedDate and staging labels. lookup-events · Version listing

A worked fixture: one database error, two causes#

The fixture is illustrative. A fictional Cloud Run service checkout reads DB_PASSWORD from the secret checkout-db-password at version latest. Traffic moves 10% to checkout-00042 at 14:03 UTC. Requests per instance between 14:03 and 14:10, and how many of them failed to authenticate to the database:

InstanceRevisionFirst log entry (UTC)RequestsFailed, case AFailed, case B
…a1checkout-0004109:122,31000
…a2checkout-0004111:472,29500
…b1checkout-0004214:03268268268
…b2checkout-0004214:04251251251
…a3checkout-0004114:064124120

Case A, the secret. The audit log shows AddSecretVersion creating version 8 at 13:58:12, called by the rotation job's service account. Every instance that started after 13:58 fails, including …a3 from the old revision, and every instance that started earlier works. Version 8 holds a password the database does not accept yet. Moving traffic back to checkout-00041 would send it to instances that must scale out, and each new one would fail. Fix the credential instead: complete the database side of the rotation, or add a version with the working value. Disabling version 8 does not send latest back to version 7, because latest names the most recently created version, and a disabled version cannot be accessed. The next instance start would then fail to read the secret at all.

Case B, the release. The newest version of each secret the service reads was created the previous day, long enough before 14:03 for latest to have converged, and …a3 serves cleanly against the same database. Only the new revision's instances fail, from their first request. checkout-00042 reads a different secret or version, or uses the value differently. Shift traffic back, then compare the revisions' secret references. Diff the two revisions

Where the pattern blurs#

Some secrets are not read at startup. A Cloud Run volume is read when the file is opened, and an application that fetches a secret through an SDK has its own cache. Failures then follow the read or the cache expiry, and they can hit old instances at any moment.

A rotation that also changes the credential on the database side reaches running instances later. Their open connections may keep working while new connections made with the old value fail as pools reconnect, so the old revision's errors arrive in a slow trickle behind the new instances'.

Without many new instances there is little to sort. A service that scales rarely may show only the rollout's instances failing, which looks exactly like a release. The secret timeline from Check 2 then carries the argument alone, and testing the old revision through a tag reads today's secret version only on an instance that started after the change had converged; check the instance's start in the system log. Test the previous revision with a tag

Questions#

Does a Cloud Run service pick up a new secret version without a redeploy?

A secret exposed as an environment variable is resolved when each instance starts, so only instances started after the change get the new version, from any revision that references latest. A secret mounted as a volume at latest is fetched from Secret Manager when the file is read, so running instances see the new version.

Why did my ECS rollback fail with the same error as the deployment?

A circuit-breaker rollback starts a new deployment of the last completed one, and any task it launches reads its secrets when its containers start. If the secret changed before the rollback, those tasks read the new value too. Check that both task definitions' valueFrom select the same version stage, then compare the tasks' createdAt to startedAt spans with the secret's PutSecretValue or UpdateSecretVersionStage events in CloudTrail.

Should Cloud Run services reference secret versions by number?

Google's Secret Manager guidance recommends pinning a version number instead of latest, and Cloud Run's documentation recommends it for secrets passed as environment variables. A new version then reaches the service only through a deploy, which your release process can check and roll back.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading