Two symptoms#
A dependency outage that lands during a deploy does not look like a dependency outage when the health checks call it. It looks like this:
- An ECS deployment of
checkout:42fails its load balancer health checks, the circuit breaker rolls back tocheckout:41, and the rollback's new tasks fail the same checks. The service events show(service checkout) (task TASK_ID) (port 8080) is unhealthy in (target-group checkout-tg) due to (reason ...)for tasks of both task definitions. - A Cloud Run revision
checkout-00042never becomes ready, and in the same minutes instances of the serving revisioncheckout-00041are shut down with log entries of the formLIVENESS HTTP probe failed 1 time consecutively for container CONTAINER_NAME on port 8080. The instance has been shut down.
In both cases the platform is doing what it was told: replacing instances whose health check fails. The health check is reporting on the database. Service unhealthy events · Cloud Run liveness failures
What each check does when it fails#
The checks differ in what they do to an instance that fails and in what happens when every instance fails at once.
| Check | On a failing instance | When every instance fails |
|---|---|---|
| ALB target group health check | The load balancer stops routing to the target; ECS stops the task and starts a replacement | The load balancer fails open and routes to all targets; ECS keeps replacing tasks |
ECS container healthCheck | An essential container marked UNHEALTHY makes the task unhealthy; ECS stops and replaces it | No fail-open; every task is replaced |
| Cloud Run startup probe | After failureThreshold × periodSeconds without success, the container is shut down | No instance of that revision can start, whether for a new deploy or for scale-out |
| Cloud Run liveness probe | The container is shut down with SIGKILL; requests it was serving end with 503; a new instance starts | Every instance is restarted; Cloud Run limits restarts after repeated probe failures |
| Cloud Run readiness probe | Cloud Run stops routing new requests to the instance without shutting it down | The documentation does not say; do not assume it fails open |
ECS starts the replacement first when maximumPercent leaves room. If the replacement is also unhealthy, the scheduler stops one of the two to return to the desired count, so a check that fails on every task keeps the service churning without end. ALB health checks and fail-open · Unhealthy task replacement · ECS container health checks · Cloud Run probes
The ECS container health check defaults to a 30-second interval, a 5-second timeout and 3 retries, with no start period unless you set startPeriod. A Cloud Run HTTP probe counts any 2xx or 3xx response as success. A configured probe's timeout defaults to 1 second, so a health endpoint that waits on a slow database fails before the database answers. ECS HealthCheck fields · Cloud Run probe settings
The check: do old instances fail the same health check?#
Two observations separate a health check that reports a dependency from a release that broke its own health check.
The first is whether instances of the old revision fail the same check in the same minutes, and whether they started failing before the rollout. On ECS, the service events name the failing task; map each task ID to its task definition. Stopped tasks stay visible to describe-tasks for at least an hour. describe-services · describe-tasks
aws ecs describe-services \
--profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
--query 'services[0].events[?contains(message, `is unhealthy in`)].[createdAt,message]' \
--output text --no-cli-pager
aws ecs describe-tasks \
--profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 \
--query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,stopped:stoppedAt,stopCode:stopCode,reason:stoppedReason}' \
--output table --no-cli-pager
On Cloud Run, probe failures are logged against the revision that ran the probe. This Cloud Logging query matches the documented liveness message for both revisions. Open one startup probe failure in Logs Explorer first and confirm its text also matches before you trust a count of zero. Cloud Run logs · Logging query language
gcloud logging read \
'resource.type="cloud_run_revision" AND resource.labels.service_name="SERVICE" AND "probe failed"' \
--project PROJECT --freshness 2h \
--format 'table(timestamp,resource.labels.revision_name)'
The second observation is what the health endpoint was doing when it failed. A dependency outage leaves the dependency's error in the health requests' log lines: a connection timeout to the database host, a refused connection, a 503 from the downstream API. The load balancer's health checks carry the User-Agent header ELB-HealthChecker/2.0, so on ECS you can count failed health requests per task if the application logs one line per request. Replace status=503 with what your request log writes for a failed check. Health check request attributes · Logs Insights parse
fields @timestamp, @logStream, @message
| filter @message like /ELB-HealthChecker/
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as checks, sum(strcontains(@message, "status=503")) as failed by task_id, bin(1m)
For the target group's own reason codes per task, see reading target health reason codes.
A worked fixture: one failed deploy, two causes#
This fixture is illustrative. A fictional Cloud Run service checkout serves revision checkout-00041. Both its startup probe and its liveness probe call GET /health, which runs SELECT 1 against the orders database. Revision checkout-00042 was deployed at 14:00 UTC. The table counts probe failure log entries per revision per minute.
In the dependency case, the database stopped accepting connections at 13:57:
| Minute (UTC) | checkout-00041 | checkout-00042 | Health endpoint log |
|---|---|---|---|
| 13:55 | 0 | ||
| 13:58 | 4 | /health timed out connecting to the orders database | |
| 14:00 | 6 | Deploy of 00042 starts | |
| 14:02 | 7 | 3 | Same timeout in both revisions |
| 14:05 | 5 | 3 | 00042 never becomes ready; deploy fails |
The old revision failed first, from the same database timeout. Its instances were being restarted before anyone deployed, and every restart cost the requests in flight. The release did not start this. checkout-00042 adds one starting instance, with the same single-connection health check and no retries, beside the old revision's instances, so rolling back would change little for the database. Rolling back changes nothing. The incident is the database, and the probes made it larger by shutting down instances that could still have served requests that do not need the database.
In the release case, the database is healthy throughout:
| Minute (UTC) | checkout-00041 | checkout-00042 | Health endpoint log |
|---|---|---|---|
| 13:55 | 0 | ||
| 14:00 | 0 | Deploy of 00042 starts | |
| 14:02 | 0 | 3 | 00042's /health returns 500: it also checks a new search index it lacks permission to read |
| 14:05 | 0 | 3 | 00042 never becomes ready; deploy fails |
The old revision passes the same probe against the same database, so the new revision's code or configuration is the cause. Cloud Run sends traffic to the latest healthy revision, so checkout-00042 took none and users saw no change. Fix the release; nothing needs rolling back. Cloud Run resource model · Cloud Run revision not ready
What each check should test#
A check that replaces an instance should fail only for a fault that replacing the instance can repair. Restarting a process does not repair a database.
- A liveness probe or ECS container health check should test the process: it answers, it is not deadlocked, its own state is usable. Cloud Run describes liveness probes as meant for unrecoverable instance failures such as a deadlock.
- A startup probe should test that the process has loaded its configuration and can serve. If it also requires the database, a database outage stops every new instance from starting, including the instances autoscaling adds to the old revision.
- An ALB target health check should test that the task can serve requests routed to it. A dependency call there makes every target fail together, and the result is the fail-open routing above plus continuous task replacement.
Dependency health still needs watching. Put it in alarms on the dependency and in the application's error metrics, where it pages someone without replacing instances. If a deep check is useful for diagnosis, serve it on a separate path that no health check calls.
Where the comparison misleads#
On ECS, the target group's health check settings are shared by old and new tasks. A deploy that also changes the health check path, port or matcher makes old tasks fail too, which reads like a dependency outage. Check the target group's settings and when they last changed before blaming the dependency. Cloud Run probes are part of each revision, so a probe change affects only the new revision.
A release can make a healthy dependency unhealthy, for example by opening more connections per instance. Then old and new instances fail together, and the timing points at the release. Did our deploy overload a dependency? · Connection budgets during rolling deployments
A dependency that failed in the same minute the rollout began leaves no "before" to compare. Read the dependency's own metrics and the provider's status for that window, as the collection's method describes.
Questions#
Should a health check call the database?
Not a check that replaces instances. Liveness probes, ECS container health checks and load balancer target checks should test the process itself, because replacing an instance does not repair the database. Watch the database with alarms instead.
Why did the ECS rollback also fail health checks?
If the health endpoint calls a dependency that is failing, the rolled-back tasks call it too and fail the same check. Look at whether tasks of the old task definition were failing before the deployment started.
What happens on Cloud Run when a liveness probe fails?
After the failure threshold is reached, Cloud Run shuts the container down with SIGKILL, ends the requests it was still serving with a 503, and starts a new instance. After repeated probe failures it limits instance restarts.