Learning center

Find when an ECS deployment or Cloud Run revision started serving

Line up the first symptom with the moment the new code answered real requests, not with the merge or the pipeline's last step. On ECS that is the first request a new task served after its target passed a health check. On Cloud Run it is the new revision's first entry in the request log. A release can act before that moment, through a migration its pipeline ran or work its new instances do as they start, so check those before clearing it.

Two symptoms read against the wrong clock#

A rollout has several timestamps, and the one people remember is usually the pipeline's. Two situations come up often:

  • The pipeline says the deploy finished at 14:00, and errors started at 14:03. The Application Load Balancer's HTTPCode_Target_5XX_Count for an ECS service rose three minutes after the deploy job turned green, so the release gets the blame. The new tasks may not have served a request yet.
  • Nothing deployed for 45 minutes, yet a Cloud Run service returns 500s from 14:07. The last pipeline run created revision checkout-00042 at 13:21, so the release is ruled out. The revision may have received its first request at 14:06.

Both conclusions rest on the same mistake. The pipeline records when someone asked for a change. The error rate responds to when the new code began handling traffic, and on both platforms that can be minutes or hours later.

Four clocks in one rollout#

Separate four moments before comparing anything with the symptom:

  1. The pipeline or merge time: when your CI system called the provider.
  2. The provider accepted the change: the ECS deployment's createdAt, or the Cloud Run revision's creationTimestamp and the audit log entry for the service update.
  3. The new code became able to serve: an ECS task reached RUNNING and its target passed the load balancer's first health check, or a Cloud Run revision's Ready condition became True.
  4. The new code served real traffic: the first request a new task or revision answered.

Use the fourth moment as the start of "after the release", and the span from it until the old code stopped serving as the overlap window. An error that starts before the fourth moment did not come from the new code answering requests. It came from the old code, from something outside the service, or from what the rollout did before serving: a migration or configuration change the pipeline applied, or new instances that opened connections, ran startup jobs or consumed queues before taking a request. Check the pipeline's steps and the first new instance's start time before ruling the release out. Compare both revisions in the same minutes

On ECS: deployment, tasks and first requests#

Start with the service's deployments and recent events. Each deployment records the task definition it rolls out, when it was created and its rolloutState. The event list holds at most the 100 most recent events, and the scheduler omits a repeated message until its cause is resolved or six hours pass. ECS Deployment fields · Service event messages

aws ecs describe-services \
  --profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
  --query 'services[0].{deployments:deployments[].{id:id,status:status,definition:taskDefinition,created:createdAt,updated:updatedAt,rollout:rolloutState,running:runningCount},events:events[].{at:createdAt,message:message}}' \
  --output json --no-cli-pager

The deployment's createdAt is clock 2. The events usually record the scheduler starting tasks, registering them with the target group, draining the old ones, and finally (service SERVICE) has reached a steady state. For a rollout that has already finished, list-service-deployments and describe-service-deployments return each deployment's startedAt, finishedAt and status for the last 90 days. Service deployment history · ServiceDeployment fields

Then read the tasks. A task started by a service carries the deployment ID in startedBy. Its startedAt is when it moved from PENDING to RUNNING, and stoppingAt and stoppedAt mark the old tasks leaving. ECS Task fields

aws ecs list-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER \
  --service-name SERVICE --desired-status RUNNING \
  --query 'taskArns' --output json --no-cli-pager

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ARN_1 TASK_ARN_2 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,deployment:startedBy,created:createdAt,pulled:pullStoppedAt,started:startedAt,health:healthStatus,stopping:stoppingAt,stopped:stoppedAt}' \
  --output table --no-cli-pager

Run list-tasks again with --desired-status STOPPED to find the old tasks, and describe them the same way. A task's startedAt is not yet clock 4. ECS registers the task's target while the task is still activating, and the load balancer routes requests to a new target only after it passes one health check, which by default runs every 30 seconds. Task lifecycle · Target group health checks

The direct answer to clock 4 is in the application's request log. With the awslogs driver each task writes to its own stream, so a CloudWatch Logs Insights query over the rollout window gives each task's first request. Replace status= with whatever marks a request line in your logs, and leave out the load balancer's health checks, which arrive before any user request and carry the user agent ELB-HealthChecker/2.0; if your request log does not record the user agent, filter out the health check path instead. awslogs stream names · Logs Insights stats functions

fields @timestamp, @logStream, @message
| filter strcontains(@message, "status=") and not strcontains(@message, "ELB-HealthChecker")
| parse @logStream "*/*/*" as prefix, container, task_id
| stats min(@timestamp) as first_request, count(*) as requests by task_id
| sort first_request asc

Start the query's time range before the deployment's createdAt and confirm the log group retains that far back; the earliest first_request among tasks of the new task definition is then the moment the release began serving. With a shorter range it is only the earliest request observed. If the application does not log requests, the load balancer's access logs record the target IP address for each request, and each awsvpc task has its own address. They arrive in S3 about every five minutes. Access log fields

On Cloud Run: revision, readiness and traffic#

A Cloud Run revision can exist for a long time before it receives traffic. A deploy with --no-traffic creates it with none, and once someone has split traffic or pinned it to a revision, later deployments keep that pattern, so a routine pipeline deploy can leave the new revision at 0%. Rollouts and traffic migration

Read the revision's creation time and conditions. Ready becoming True is clock 3: by default Cloud Run starts an instance and waits for its startup probe before it routes traffic to a revision. Revision status · Condition fields · Deployment health check

gcloud run revisions describe REVISION \
  --project PROJECT --region REGION \
  --format='yaml(metadata.creationTimestamp,status.conditions)'

gcloud run services describe SERVICE \
  --project PROJECT --region REGION \
  --format='yaml(status.latestCreatedRevisionName,status.latestReadyRevisionName,status.traffic)'

status.traffic shows the split now, not when it changed. The history of traffic moves is in the Admin Activity audit log, where each service update appears as google.cloud.run.v1.Services.ReplaceService or google.cloud.run.v2.Services.UpdateService. Cloud Run audit logging · Logging query language

gcloud logging read '
  logName="projects/PROJECT/logs/cloudaudit.googleapis.com%2Factivity"
  AND protoPayload.serviceName="run.googleapis.com"
  AND protoPayload.methodName=("google.cloud.run.v1.Services.ReplaceService" OR "google.cloud.run.v2.Services.UpdateService")
  AND protoPayload.resourceName:"services/SERVICE"
  AND timestamp>="START_TIME"' \
  --project PROJECT --order asc \
  --format='table(timestamp,protoPayload.methodName,protoPayload.authenticationInfo.principalEmail)'

An audit entry says when someone changed the service. The request log says when the revision answered. Every request log entry carries the revision in resource.labels.revision_name, so the first one that a container answered is clock 4, provided START_TIME is at or before the revision's creationTimestamp and the request log is retained that far back; otherwise it is only the earliest request observed. Requests Cloud Run aborted because no instance was available also appear there, with The request was aborted in the entry, and the query leaves them out. The new revision's instances start earlier, when Cloud Run logs Starting new instance in the revision's system log, and any work they do at startup reaches dependencies from then on. Cloud Run request logs · gcloud logging read

gcloud logging read '
  logName="projects/PROJECT/logs/run.googleapis.com%2Frequests"
  AND resource.type="cloud_run_revision"
  AND resource.labels.service_name="SERVICE"
  AND resource.labels.revision_name="REVISION"
  AND NOT "The request was aborted"
  AND timestamp>="START_TIME"' \
  --project PROJECT --order asc --limit 5 \
  --format='table(timestamp,httpRequest.status,httpRequest.requestUrl)'

A worked fixture: one pipeline clock, two wrong conclusions#

The fixture is illustrative. The service is called checkout on both platforms, and all times are UTC on one day.

ECS, where the pipeline time blames the release. Task definition checkout:42 replaces checkout:41.

TimeSourceWhat it shows
13:52CIMerge; pipeline starts
14:00:02CIDeploy job succeeds after update-service returns
14:00:04describe-servicesDeployment for checkout:42 created, IN_PROGRESS
14:03CloudWatchHTTPCode_Target_5XX_Count rises from 0 to 40 a minute
14:06:41describe-tasksFirst checkout:42 task startedAt
14:07:15Logs InsightsFirst request served by a checkout:42 task
14:15Service eventsSteady state

The errors began four minutes before any checkout:42 task answered a request, so every error until 14:07 came from checkout:41. The pipeline ran no migration, and CloudTrail shows no change from its role between 13:52 and 14:03 to a secret, security group or IAM policy the old tasks also use. The first checkout:42 task finished pulling its image at 14:06:20 (pullStoppedAt), so none of its code ran before then. Containers do run before startedAt, while ECS registers the task with the load balancer, so read the pull time, not the start, as the earliest moment new code could touch a dependency. No new code ran by 14:03, so the release had not yet touched anything the old tasks use. It did not start these errors. Look at the old tasks' dependency errors from before 14:03, and once checkout:42 serves, check whether it adds calls or connections to the failing dependency. Did our deploy overload a dependency? · Rule a cloud provider incident in or out

Cloud Run, where the pipeline time clears the release. An engineer pinned traffic to checkout-00041 the previous week.

TimeSourceWhat it shows
13:21:40Revisioncheckout-00042 creationTimestamp
13:22:30RevisionReady condition becomes True
13:23CIDeploy job succeeds; the pin keeps 100% of traffic on checkout-00041
14:06:05Audit logReplaceService call by the on-call engineer
14:06:09Request logFirst request served by checkout-00042
14:07Request log500s begin, all from checkout-00042

The 45-minute gap belonged to the pipeline. The new revision served its first request one minute before the errors, and only it fails. Before moving traffic back to checkout-00041, check through a tag that a fresh instance of it works with today's secrets, then compare the two revisions. Test the previous revision with a tag · Diff the two revisions · Errors after a traffic shift

Where the timestamps mislead#

describe-tasks returns stopped tasks for at least an hour after they stop, and the service keeps only its latest 100 events. Capture both early, or fall back to logs and the deployment history. describe-tasks

The first request is a lower bound on exposure, not a measure of it. A rolling update brings new tasks in one or a few at a time, and a Cloud Run split sends a fraction of traffic, so compare rates per task or revision once you have the start. Cloud Run traffic changes are not instantaneous either: during the transition, in-flight requests can land on either revision.

Requests through a revision tag URL also appear in the request log under that revision. If someone tested checkout-00042 through its tag before moving traffic, the first entries are theirs; check httpRequest.requestUrl for the tag prefix. Revision tags · Test a previous revision with a tag

Compare signals from the moment a deploy lands#

Proofline's deploy verification reads the commit an environment is running and compares the signals the environment watches at checks after each deploy lands; a deploy whose signals got worse reads Regressed. It does not decide a rollback. Read a deploy's verification

Questions#

Which timestamp should I treat as the deploy time?

For errors in requests, the first request served by the new tasks or revision; for anything the new code does at startup, the first new Cloud Run instance's start, or on ECS the first new task's pullStoppedAt, since containers run before startedAt. Merge and pipeline times record when someone asked for a change; the provider can take minutes to place, start and health-check new code, and Cloud Run can hold a new revision at 0% traffic indefinitely.

Why does an ECS deployment's createdAt come before the new code serves anything?

createdAt is when ECS created the deployment. The scheduler still has to place and start tasks, register them with the target group and wait for a passing health check before the load balancer routes requests to them.

How do I find when traffic moved to an older Cloud Run revision?

Query the Admin Activity audit log for service updates in the window, then read the first request log entry for that revision. status.traffic shows only the current split.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading