---
title: "Health checks that call a dependency turn its outage into a failed rollout | Proofline"
description: "A health check that queries the database or a downstream API fails whenever that dependency does, on old and new instances alike."
url: https://proofline.sh/learn/health-checks-that-call-dependencies/
---

[Learning center](/learn/)

# Health checks that call a dependency turn its outage into a failed rollout

A health check that queries the database or a downstream API fails whenever that dependency does, on old and new instances alike. During a rollout this looks like a bad release. Check whether old instances fail the same check in the same minutes, and whether the health endpoint's own logs show the dependency's error.

Broken dependencies The Proofline team November 20, 2026 8 min read

## Two symptoms

A dependency outage that lands during a deploy does not look like a dependency outage when the health checks call it. It looks like this:

-   An ECS deployment of `checkout:42` fails its load balancer health checks, the circuit breaker rolls back to `checkout:41`, and the rollback's new tasks fail the same checks. The service events show `(service checkout) (task TASK_ID) (port 8080) is unhealthy in (target-group checkout-tg) due to (reason ...)` for tasks of both task definitions.
-   A Cloud Run revision `checkout-00042` never becomes ready, and in the same minutes instances of the serving revision `checkout-00041` are shut down with log entries of the form `LIVENESS HTTP probe failed 1 time consecutively for container CONTAINER_NAME on port 8080. The instance has been shut down.`

In both cases the platform is doing what it was told: replacing instances whose health check fails. The health check is reporting on the database. [Service unhealthy events](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-unhealthy-event-messages.html) · [Cloud Run liveness failures](https://cloud.google.com/run/docs/troubleshooting)

## What each check does when it fails

The checks differ in what they do to an instance that fails and in what happens when every instance fails at once.

| Check | On a failing instance | When every instance fails |
| --- | --- | --- |
| ALB target group health check | The load balancer stops routing to the target; ECS stops the task and starts a replacement | The load balancer fails open and routes to all targets; ECS keeps replacing tasks |
| ECS container `healthCheck` | An essential container marked `UNHEALTHY` makes the task unhealthy; ECS stops and replaces it | No fail-open; every task is replaced |
| Cloud Run startup probe | After `failureThreshold` × `periodSeconds` without success, the container is shut down | No instance of that revision can start, whether for a new deploy or for scale-out |
| Cloud Run liveness probe | The container is shut down with `SIGKILL`; requests it was serving end with 503; a new instance starts | Every instance is restarted; Cloud Run limits restarts after repeated probe failures |
| Cloud Run readiness probe | Cloud Run stops routing new requests to the instance without shutting it down | The documentation does not say; do not assume it fails open |

ECS starts the replacement first when `maximumPercent` leaves room. If the replacement is also unhealthy, the scheduler stops one of the two to return to the desired count, so a check that fails on every task keeps the service churning without end. [ALB health checks and fail-open](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html) · [Unhealthy task replacement](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html) · [ECS container health checks](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_HealthCheck.html) · [Cloud Run probes](https://cloud.google.com/run/docs/configuring/healthchecks)

The ECS container health check defaults to a 30-second interval, a 5-second timeout and 3 retries, with no start period unless you set `startPeriod`. A Cloud Run HTTP probe counts any 2xx or 3xx response as success. A configured probe's timeout defaults to 1 second, so a health endpoint that waits on a slow database fails before the database answers. [ECS HealthCheck fields](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_HealthCheck.html) · [Cloud Run probe settings](https://cloud.google.com/run/docs/configuring/healthchecks)

## The check: do old instances fail the same health check?

Two observations separate a health check that reports a dependency from a release that broke its own health check.

The first is whether instances of the old revision fail the same check in the same minutes, and whether they started failing before the rollout. On ECS, the service events name the failing task; map each task ID to its task definition. Stopped tasks stay visible to `describe-tasks` for at least an hour. [describe-services](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-services.html) · [describe-tasks](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html)

```
aws ecs describe-services \
  --profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
  --query 'services[0].events[?contains(message, `is unhealthy in`)].[createdAt,message]' \
  --output text --no-cli-pager

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,stopped:stoppedAt,stopCode:stopCode,reason:stoppedReason}' \
  --output table --no-cli-pager
```

On Cloud Run, probe failures are logged against the revision that ran the probe. This Cloud Logging query matches the documented liveness message for both revisions. Open one startup probe failure in Logs Explorer first and confirm its text also matches before you trust a count of zero. [Cloud Run logs](https://cloud.google.com/run/docs/logging) · [Logging query language](https://cloud.google.com/logging/docs/view/logging-query-language)

```
gcloud logging read \
  'resource.type="cloud_run_revision" AND resource.labels.service_name="SERVICE" AND "probe failed"' \
  --project PROJECT --freshness 2h \
  --format 'table(timestamp,resource.labels.revision_name)'
```

The second observation is what the health endpoint was doing when it failed. A dependency outage leaves the dependency's error in the health requests' log lines: a connection timeout to the database host, a refused connection, a 503 from the downstream API. The load balancer's health checks carry the `User-Agent` header `ELB-HealthChecker/2.0`, so on ECS you can count failed health requests per task if the application logs one line per request. Replace `status=503` with what your request log writes for a failed check. [Health check request attributes](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#target-not-inservice) · [Logs Insights `parse`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-Parse.html)

```
fields @timestamp, @logStream, @message
| filter @message like /ELB-HealthChecker/
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as checks, sum(strcontains(@message, "status=503")) as failed by task_id, bin(1m)
```

For the target group's own reason codes per task, see [reading target health reason codes](/learn/ecs-target-health-check-reason-codes/).

## A worked fixture: one failed deploy, two causes

This fixture is illustrative. A fictional Cloud Run service `checkout` serves revision `checkout-00041`. Both its startup probe and its liveness probe call `GET /health`, which runs `SELECT 1` against the orders database. Revision `checkout-00042` was deployed at 14:00 UTC. The table counts probe failure log entries per revision per minute.

In the dependency case, the database stopped accepting connections at 13:57:

| Minute (UTC) | `checkout-00041` | `checkout-00042` | Health endpoint log |
| --- | --- | --- | --- |
| 13:55 | 0 |  |  |
| 13:58 | 4 |  | `/health` timed out connecting to the orders database |
| 14:00 | 6 |  | Deploy of 00042 starts |
| 14:02 | 7 | 3 | Same timeout in both revisions |
| 14:05 | 5 | 3 | 00042 never becomes ready; deploy fails |

The old revision failed first, from the same database timeout. Its instances were being restarted before anyone deployed, and every restart cost the requests in flight. The release did not start this. `checkout-00042` adds one starting instance, with the same single-connection health check and no retries, beside the old revision's instances, so rolling back would change little for the database. Rolling back changes nothing. The incident is the database, and the probes made it larger by shutting down instances that could still have served requests that do not need the database.

In the release case, the database is healthy throughout:

| Minute (UTC) | `checkout-00041` | `checkout-00042` | Health endpoint log |
| --- | --- | --- | --- |
| 13:55 | 0 |  |  |
| 14:00 | 0 |  | Deploy of 00042 starts |
| 14:02 | 0 | 3 | 00042's `/health` returns 500: it also checks a new search index it lacks permission to read |
| 14:05 | 0 | 3 | 00042 never becomes ready; deploy fails |

The old revision passes the same probe against the same database, so the new revision's code or configuration is the cause. Cloud Run sends traffic to the latest healthy revision, so `checkout-00042` took none and users saw no change. Fix the release; nothing needs rolling back. [Cloud Run resource model](https://cloud.google.com/run/docs/resource-model) · [Cloud Run revision not ready](/learn/cloud-run-revision-not-ready/)

## What each check should test

A check that replaces an instance should fail only for a fault that replacing the instance can repair. Restarting a process does not repair a database.

-   A liveness probe or ECS container health check should test the process: it answers, it is not deadlocked, its own state is usable. Cloud Run describes liveness probes as meant for unrecoverable instance failures such as a deadlock.
-   A startup probe should test that the process has loaded its configuration and can serve. If it also requires the database, a database outage stops every new instance from starting, including the instances autoscaling adds to the old revision.
-   An ALB target health check should test that the task can serve requests routed to it. A dependency call there makes every target fail together, and the result is the fail-open routing above plus continuous task replacement.

Dependency health still needs watching. Put it in alarms on the dependency and in the application's error metrics, where it pages someone without replacing instances. If a deep check is useful for diagnosis, serve it on a separate path that no health check calls.

## Where the comparison misleads

On ECS, the target group's health check settings are shared by old and new tasks. A deploy that also changes the health check path, port or matcher makes old tasks fail too, which reads like a dependency outage. Check the target group's settings and when they last changed before blaming the dependency. Cloud Run probes are part of each revision, so a probe change affects only the new revision.

A release can make a healthy dependency unhealthy, for example by opening more connections per instance. Then old and new instances fail together, and the timing points at the release. [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/) · [Connection budgets during rolling deployments](/learn/connection-limits/)

A dependency that failed in the same minute the rollout began leaves no "before" to compare. Read the dependency's own metrics and the provider's status for that window, as the [collection's method](/learn/bad-release-or-unhealthy-dependency/#before-the-rollout) describes.

## Questions

### Should a health check call the database?

Not a check that replaces instances. Liveness probes, ECS container health checks and load balancer target checks should test the process itself, because replacing an instance does not repair the database. Watch the database with alarms instead.

### Why did the ECS rollback also fail health checks?

If the health endpoint calls a dependency that is failing, the rolled-back tasks call it too and fail the same check. Look at whether tasks of the old task definition were failing before the deployment started.

### What happens on Cloud Run when a liveness probe fails?

After the failure threshold is reached, Cloud Run shuts the container down with `SIGKILL`, ends the requests it was still serving with a 503, and starts a new instance. After repeated probe failures it limits instance restarts.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [New ECS tasks fail load balancer health checks: read the reason code](/learn/ecs-target-health-check-reason-codes/)
    
    Deployment failures The Proofline team November 20, 2026 8 min read
    
-   ### [A Cloud Run revision that is not ready: the container or its startup calls](/learn/cloud-run-revision-not-ready/)
    
    Deployment failures The Proofline team November 20, 2026 9 min read
