---
title: "Did our deploy overload a dependency that was healthy before it? | Proofline"
description: "When both revisions fail against a dependency that was healthy before the rollout, compare your service's load on it per request before and after."
url: https://proofline.sh/learn/did-our-deploy-overload-a-dependency/
---

[Learning center](/learn/)

# Did our deploy overload a dependency that was healthy before it?

When both revisions fail against a dependency that was healthy before the rollout, compare your service's load on it per request before and after. If calls or attempts per request stepped up with the rollout, the release made the dependency unhealthy, even though the old revision fails too. Database time per request is not enough on its own, because it rises whenever the database slows down.

Capacity overruns The Proofline team November 20, 2026 10 min read

## Two symptoms where every revision fails

-   An ECS service rolls from `checkout:41` to `checkout:42`. Ten minutes in, Amazon RDS reports `DatabaseConnections` at twice its usual level and `CPUUtilization` above 90%, and requests from old and new tasks alike time out on the database.
-   A Cloud Run service moves 50% of traffic to `checkout-00042`. Both `checkout-00041` and `checkout-00042` start returning 500s after timeouts calling an internal inventory API, and the inventory team reports that its service is up but saturated.

The usual split by revision reads these as a dependency problem, because the old revision fails at the same rate as the new one. The timing says otherwise: the dependency was clean until the new revision took traffic. The release can be the cause while every revision fails, because they all share the dependency it overloaded. [Read the split by revision](/learn/bad-release-or-unhealthy-dependency/#read-the-result)

## How a release raises load without more traffic

A release changes how much work each incoming request sends to a dependency. The common ways:

-   A new query or API call on a hot path, such as an extra lookup per item in a list.
-   A cache key change, or a cache bypassed by a new code path, which sends reads that used to hit the cache to the database.
-   More retries, or a shorter timeout with the same retry count. Each request then makes more attempts, and the extra attempts arrive exactly when the dependency is slow. A value such as the AWS SDK's `max_attempts` counts the initial request, so 3 means up to two retries. [AWS SDK retry settings](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html)
-   A larger connection pool, or more tasks or instances, each with its own pool. [Connection budgets during rolling deployments](/learn/connection-limits/)

None of these needs more user traffic. All of them change the ratio between the dependency's work and your service's requests, and that ratio is the check.

## The check: dependency load from your service, per request

Measure two things over the same minutes, from 30 to 60 minutes before the rollout until now:

1.  The dependency's load attributed to your service: queries or calls, connections, and database time, filtered to your database user, `application_name`, client addresses or API credential.
2.  Your service's own request count.

Divide the first by the second. Use counts of work your service issued, calls and attempts, for the decision. Held connections rise both when the release opens more and when slower calls keep a pool's connections checked out longer, so count them as release load only when the pool size or task count changed. Database time per request rises whenever the database slows down, whoever slowed it, so a rise in time with flat calls and connections is ambiguous: it fits another caller's load as well as yours. If the count ratio steps up when the new revision begins serving, and climbs further as it takes more traffic, the release raised the load. If the ratio stays flat while the dependency's total load rises, the extra load came from another caller, a batch job or a maintenance task, or from heavier calls: the same statement can scan far more rows after a release changes its parameters. Before excluding the release, compare per-statement work per call over matching intervals before and after the rollout: the change in `total_exec_time`, `rows` and `shared_blks_read` between two snapshots, divided by the change in `calls`. `mean_exec_time` averages a statement's whole history and moves little when recent calls get heavier. Also look for a caller whose calls rose. If the ratio stays flat and your request count rose, the traffic grew, not the work per request.

The attribution only works if callers are distinguishable. A dedicated database user or `application_name` per service makes this easy; a shared user across services makes the dependency's own metrics useless for this check.

## Amazon RDS for PostgreSQL behind an ECS service

Start with the instance's connections and CPU at one-minute resolution. `DatabaseConnections` counts client network connections, so it can read lower than the number of sessions the engine holds. [RDS CloudWatch metrics](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-metrics.html)

```
aws cloudwatch get-metric-statistics \
  --profile PROFILE --region REGION \
  --namespace AWS/RDS --metric-name DatabaseConnections \
  --dimensions Name=DBInstanceIdentifier,Value=DB_INSTANCE \
  --start-time START_UTC --end-time END_UTC --period 60 --statistics Maximum \
  --output table --no-cli-pager
```

Repeat with `--metric-name CPUUtilization --statistics Average`. Then take your service's request count over the same minutes from the load balancer: `RequestCount` with the `LoadBalancer` and `TargetGroup` dimensions and the `Sum` statistic. [ALB metrics](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html)

```
aws cloudwatch get-metric-statistics \
  --profile PROFILE --region REGION \
  --namespace AWS/ApplicationELB --metric-name RequestCount \
  --dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID Name=TargetGroup,Value=targetgroup/TG_NAME/TG_ID \
  --start-time START_UTC --end-time END_UTC --period 60 --statistics Sum \
  --output table --no-cli-pager
```

The Performance Insights API splits database load by caller. `db.load.avg` grouped by `db.user` or `db.host` shows which user or client host the active sessions came from, minute by minute. The identifier is the instance's `DbiResourceId`, not its name. [get-resource-metrics](https://docs.aws.amazon.com/cli/latest/reference/pi/get-resource-metrics.html) · [Database Insights](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.html)

```
aws pi get-resource-metrics \
  --profile PROFILE --region REGION \
  --service-type RDS --identifier DBI_RESOURCE_ID \
  --start-time START_UTC --end-time END_UTC --period-in-seconds 60 \
  --metric-queries 'Metric=db.load.avg,GroupBy={Group=db.user}' \
  --output json --no-cli-pager
```

Database load measures time, and it rises when the database slows down even if the calls stay the same. To count calls, read `pg_stat_statements` from a read-only session. Its `calls` column is cumulative, so take two snapshots a few minutes apart and subtract. RDS for PostgreSQL 11 and later load the library by default; each database still needs `CREATE EXTENSION pg_stat_statements` before the view exists. [pg\_stat\_statements](https://www.postgresql.org/docs/current/pgstatstatements.html) · [RDS SQL statistics](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PerfInsights.UsingDashboard.AnalyzeDBLoad.AdditionalMetrics.PostgreSQL.html)

```
SELECT now() AS taken_at, r.rolname, sum(s.calls) AS calls
FROM pg_stat_statements s JOIN pg_roles r ON r.oid = s.userid
GROUP BY r.rolname ORDER BY calls DESC;
```

To find a statement that is new or much hotter since the rollout, snapshot every statement your service's user runs, once before and once after, and compare the two. Keep every row: a limit applied to cumulative counters drops an expensive statement that runs rarely, and makes a statement that only crossed the cutoff look new.

```
psql "SERVICE_DB_URL" -At -F $'\t' -c "
  SELECT s.queryid, sum(s.calls), sum(s.total_exec_time), sum(s.rows), sum(s.shared_blks_read)
  FROM pg_stat_statements s JOIN pg_roles r ON r.oid = s.userid
  WHERE r.rolname = 'SERVICE_DB_USER'
  GROUP BY s.queryid" > after.tsv
```

Run it into `before.tsv`, wait a few minutes, and run it again into `after.tsv`. With each snapshot, read `SELECT dealloc, stats_reset FROM pg_stat_statements_info` (PostgreSQL 14 and later); if either value changed between the two, counters were reset or statements evicted in between, so discard the pair and take two new snapshots. Then subtract per `queryid` and rank by the time each statement spent in the interval:

```
awk -F'\t' -v OFS='\t' '
  NR == FNR { c[$1] = $2; t[$1] = $3; r[$1] = $4; b[$1] = $5; next }
  { seen = ($1 in c); dc = $2 - c[$1]; dt = $3 - t[$1]
    dr = $4 - r[$1]; db = $5 - b[$1]
    if (dc < 0) { print $1, "reset" > "/dev/stderr"; next }
    print $1, dc, dt, (dc ? dt / dc : 0), (dc ? dr / dc : 0), (dc ? db / dc : 0), (seen ? "" : "new") }
' before.tsv after.tsv | sort -t$'\t' -k3,3gr | head -20
```

The columns are `queryid`, calls in the interval, milliseconds spent in the interval, then milliseconds, rows and blocks read per call, and `new` for a statement the first snapshot lacks. Rows and blocks per call rise when the statement itself does more work; time per call alone also rises when the database slows for any reason. The counters are cumulative, so an interval before the rollout exists only if someone snapshotted it then; during an incident, divide each statement's calls in the interval by your service's requests in the same minutes. A statement whose calls grew far faster than the service's requests, or whose rows or blocks per call jumped, is the one to read in the diff. Look up its text with `SELECT query FROM pg_stat_statements WHERE queryid = QUERYID`. For connections now, group `pg_stat_activity` by `usename`, `application_name`, `client_addr` and `state` where `backend_type = 'client backend'`; in `awsvpc` mode each task has its own client address. [pg\_stat\_activity](https://www.postgresql.org/docs/current/monitoring-stats.html)

If the release changed caching, check the cache too: ElastiCache publishes `CacheHits`, `CacheMisses` and `CacheHitRate`. [ElastiCache metrics](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/CacheMetrics.Redis.html)

## Cloud SQL for PostgreSQL behind a Cloud Run service

Cloud SQL publishes connections by application and, with query insights enabled, query latency distributions per database user and client address. The count of the distribution is the number of queries, so this PromQL query divides your database user's query rate by your service's request rate. [Cloud SQL metrics](https://cloud.google.com/monitoring/api/metrics_gcp_c) · [Query insights](https://cloud.google.com/sql/docs/postgres/using-query-insights) · [Distributions in PromQL](https://cloud.google.com/monitoring/promql/promql-mapping)

```
sum(rate(cloudsql_googleapis_com:database_postgresql_insights_aggregate_latencies_count{monitored_resource="cloudsql_instance_database", database="DATABASE", user="SERVICE_DB_USER"}[5m]))
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))
```

For connections, chart `cloudsql.googleapis.com/database/postgresql/num_backends_by_application` grouped by its `application` label. `database/cpu/utilization` gives CPU as a fraction of the reserved CPU. Memorystore for Redis reports `redis.googleapis.com/stats/cache_hit_ratio`. [Memorystore metrics](https://cloud.google.com/monitoring/api/metrics_gcp_p_z)

## Downstream HTTP APIs

An API owned by another team needs their per-client request rate: from their load balancer or gateway logs grouped by client address or API key, or from their own metrics labeled by caller. Compare it with your service's request count exactly as above. If the dependency is a Google Cloud API, `serviceruntime.googleapis.com/api/request_count` on the `consumed_api` resource carries `method` and `credential_id` labels, so you can chart your service's calls by method. [Consumed API resource](https://cloud.google.com/monitoring/api/resources)

Your own logs answer the same question from the caller's side if they record one line per outgoing attempt with the incoming request's ID. Count attempts per request ID, before and after.

## A worked fixture: one timeout, two causes

The fixture is illustrative: a fictional ECS service `checkout` with its own database user `checkout`, rolling from `checkout:41` to `checkout:42`. New tasks began serving at 14:03 UTC and all traffic reached them by 14:12. Calls come from two `pg_stat_statements` snapshots per interval.

Case A, the release raised the load:

| Minute (UTC) | Requests/min | `checkout` calls/min | Calls per request | `DatabaseConnections` | CPU |
| --- | --- | --- | --- | --- | --- |
| 13:50 | 6,000 | 18,600 | 3.1 | 118 | 34% |
| 14:00 | 6,100 | 18,900 | 3.1 | 121 | 35% |
| 14:06 | 6,050 | 27,800 | 4.6 | 190 | 61% |
| 14:12 | 6,000 | 42,000 | 7.0 | 262 | 93% |

Requests stayed flat while calls per request more than doubled, rising with the new tasks' share of traffic. A new statement, run once per cart item, appears in the snapshot at 14:04. Roll back, even though old tasks fail too; the load they share comes from the new ones.

Case B, another caller:

| Minute (UTC) | Requests/min | `checkout` calls/min | Calls per request | `reporting` calls/min | CPU |
| --- | --- | --- | --- | --- | --- |
| 13:50 | 6,000 | 18,600 | 3.1 | 900 | 34% |
| 14:00 | 6,100 | 18,900 | 3.1 | 14,500 | 78% |
| 14:06 | 6,050 | 18,800 | 3.1 | 15,200 | 91% |
| 14:12 | 6,000 | 18,600 | 3.1 | 15,000 | 92% |

`checkout`'s calls per request never moved, and neither did its connections or its statements' time and rows per call between snapshots, before and after 14:03, so its work per request never moved. `checkout:42` keeps `checkout:41`'s client timeout and retry count, and the service's own logs show one attempt per call throughout, so it is not retrying harder against the slow database. The `reporting` user's calls rose from 13:55, before the rollout. A rollback would leave CPU where it is.

## Where the ratio misleads

Retries feed back on themselves. A release that only lowered a timeout keeps calls per request flat until the dependency slows for any reason, and then multiplies attempts. The ratio jumps after the slowdown starts, which makes the release look innocent if you read only the first minutes. Look at attempts per request in your own logs, and at the diff for timeout and retry settings. [Diff the two revisions](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

Pools open connections without queries. A larger pool or more tasks raises `DatabaseConnections` with the same calls per request, and the database can fail on connection slots alone. Count connections per task across both revisions. [Connection budgets during rolling deployments](/learn/connection-limits/)

Cumulative counters can drop. `pg_stat_statements_reset` discards the counts, and once more distinct statements arrive than `pg_stat_statements.max` allows, the least-executed ones are discarded; a new statement that churns the view can shrink totals between snapshots. Query insights reports a bounded number of top queries, so a low-volume caller can be missing. [pg\_stat\_statements settings](https://www.postgresql.org/docs/current/pgstatstatements.html)

New instances start cold. Freshly started tasks or instances often run more queries until their caches warm, which raises calls per request briefly for any deploy, including a rollback. A ratio that returns to normal within minutes is warmup, not the release.

## Questions

### If the old revision fails too, can the release still be the cause?

Yes. Both revisions share the dependency, so a release that overloads it degrades every caller. The split by revision cannot show this; the dependency's load per request from your service, before and after the rollout, can.

### Which metric shows that our service, and not another caller, raised database load?

A count attributed to the caller: `pg_stat_statements` calls by role, Cloud SQL query insights query counts labeled by `user`, or connections grouped by user or application. Performance Insights `db.load.avg` grouped by `db.user` or `db.host` on RDS shows whose sessions spent the time, which rises for every caller when the database slows, so use it to corroborate a count, not on its own. Instance-wide `DatabaseConnections` and CPU cannot tell callers apart.

### Should we raise the database's connection limit or size instead of rolling back?

Not as a first move during the incident. If calls or attempts per request rose with the release, or its pool size or task count raised connections, a larger database absorbs the change but keeps it. Roll back, then decide whether the new load is intended and size for it.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [How rolling deployments change your PostgreSQL connection budget](/learn/connection-limits/)
    
    Capacity overruns The Proofline team November 5, 2026 8 min read
    
-   ### [What changed between two ECS task definitions or Cloud Run revisions?](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)
    
    Deployment failures The Proofline team November 20, 2026 8 min read
