Two symptoms where every revision fails#
- An ECS service rolls from
checkout:41tocheckout:42. Ten minutes in, Amazon RDS reportsDatabaseConnectionsat twice its usual level andCPUUtilizationabove 90%, and requests from old and new tasks alike time out on the database. - A Cloud Run service moves 50% of traffic to
checkout-00042. Bothcheckout-00041andcheckout-00042start returning 500s after timeouts calling an internal inventory API, and the inventory team reports that its service is up but saturated.
The usual split by revision reads these as a dependency problem, because the old revision fails at the same rate as the new one. The timing says otherwise: the dependency was clean until the new revision took traffic. The release can be the cause while every revision fails, because they all share the dependency it overloaded. Read the split by revision
How a release raises load without more traffic#
A release changes how much work each incoming request sends to a dependency. The common ways:
- A new query or API call on a hot path, such as an extra lookup per item in a list.
- A cache key change, or a cache bypassed by a new code path, which sends reads that used to hit the cache to the database.
- More retries, or a shorter timeout with the same retry count. Each request then makes more attempts, and the extra attempts arrive exactly when the dependency is slow. A value such as the AWS SDK's
max_attemptscounts the initial request, so 3 means up to two retries. AWS SDK retry settings - A larger connection pool, or more tasks or instances, each with its own pool. Connection budgets during rolling deployments
None of these needs more user traffic. All of them change the ratio between the dependency's work and your service's requests, and that ratio is the check.
The check: dependency load from your service, per request#
Measure two things over the same minutes, from 30 to 60 minutes before the rollout until now:
- The dependency's load attributed to your service: queries or calls, connections, and database time, filtered to your database user,
application_name, client addresses or API credential. - Your service's own request count.
Divide the first by the second. Use counts of work your service issued, calls and attempts, for the decision. Held connections rise both when the release opens more and when slower calls keep a pool's connections checked out longer, so count them as release load only when the pool size or task count changed. Database time per request rises whenever the database slows down, whoever slowed it, so a rise in time with flat calls and connections is ambiguous: it fits another caller's load as well as yours. If the count ratio steps up when the new revision begins serving, and climbs further as it takes more traffic, the release raised the load. If the ratio stays flat while the dependency's total load rises, the extra load came from another caller, a batch job or a maintenance task, or from heavier calls: the same statement can scan far more rows after a release changes its parameters. Before excluding the release, compare per-statement work per call over matching intervals before and after the rollout: the change in total_exec_time, rows and shared_blks_read between two snapshots, divided by the change in calls. mean_exec_time averages a statement's whole history and moves little when recent calls get heavier. Also look for a caller whose calls rose. If the ratio stays flat and your request count rose, the traffic grew, not the work per request.
The attribution only works if callers are distinguishable. A dedicated database user or application_name per service makes this easy; a shared user across services makes the dependency's own metrics useless for this check.
Amazon RDS for PostgreSQL behind an ECS service#
Start with the instance's connections and CPU at one-minute resolution. DatabaseConnections counts client network connections, so it can read lower than the number of sessions the engine holds. RDS CloudWatch metrics
aws cloudwatch get-metric-statistics \
--profile PROFILE --region REGION \
--namespace AWS/RDS --metric-name DatabaseConnections \
--dimensions Name=DBInstanceIdentifier,Value=DB_INSTANCE \
--start-time START_UTC --end-time END_UTC --period 60 --statistics Maximum \
--output table --no-cli-pager
Repeat with --metric-name CPUUtilization --statistics Average. Then take your service's request count over the same minutes from the load balancer: RequestCount with the LoadBalancer and TargetGroup dimensions and the Sum statistic. ALB metrics
aws cloudwatch get-metric-statistics \
--profile PROFILE --region REGION \
--namespace AWS/ApplicationELB --metric-name RequestCount \
--dimensions Name=LoadBalancer,Value=app/LB_NAME/LB_ID Name=TargetGroup,Value=targetgroup/TG_NAME/TG_ID \
--start-time START_UTC --end-time END_UTC --period 60 --statistics Sum \
--output table --no-cli-pager
The Performance Insights API splits database load by caller. db.load.avg grouped by db.user or db.host shows which user or client host the active sessions came from, minute by minute. The identifier is the instance's DbiResourceId, not its name. get-resource-metrics · Database Insights
aws pi get-resource-metrics \
--profile PROFILE --region REGION \
--service-type RDS --identifier DBI_RESOURCE_ID \
--start-time START_UTC --end-time END_UTC --period-in-seconds 60 \
--metric-queries 'Metric=db.load.avg,GroupBy={Group=db.user}' \
--output json --no-cli-pager
Database load measures time, and it rises when the database slows down even if the calls stay the same. To count calls, read pg_stat_statements from a read-only session. Its calls column is cumulative, so take two snapshots a few minutes apart and subtract. RDS for PostgreSQL 11 and later load the library by default; each database still needs CREATE EXTENSION pg_stat_statements before the view exists. pg_stat_statements · RDS SQL statistics
SELECT now() AS taken_at, r.rolname, sum(s.calls) AS calls
FROM pg_stat_statements s JOIN pg_roles r ON r.oid = s.userid
GROUP BY r.rolname ORDER BY calls DESC;
To find a statement that is new or much hotter since the rollout, snapshot every statement your service's user runs, once before and once after, and compare the two. Keep every row: a limit applied to cumulative counters drops an expensive statement that runs rarely, and makes a statement that only crossed the cutoff look new.
psql "SERVICE_DB_URL" -At -F $'\t' -c "
SELECT s.queryid, sum(s.calls), sum(s.total_exec_time), sum(s.rows), sum(s.shared_blks_read)
FROM pg_stat_statements s JOIN pg_roles r ON r.oid = s.userid
WHERE r.rolname = 'SERVICE_DB_USER'
GROUP BY s.queryid" > after.tsv
Run it into before.tsv, wait a few minutes, and run it again into after.tsv. With each snapshot, read SELECT dealloc, stats_reset FROM pg_stat_statements_info (PostgreSQL 14 and later); if either value changed between the two, counters were reset or statements evicted in between, so discard the pair and take two new snapshots. Then subtract per queryid and rank by the time each statement spent in the interval:
awk -F'\t' -v OFS='\t' '
NR == FNR { c[$1] = $2; t[$1] = $3; r[$1] = $4; b[$1] = $5; next }
{ seen = ($1 in c); dc = $2 - c[$1]; dt = $3 - t[$1]
dr = $4 - r[$1]; db = $5 - b[$1]
if (dc < 0) { print $1, "reset" > "/dev/stderr"; next }
print $1, dc, dt, (dc ? dt / dc : 0), (dc ? dr / dc : 0), (dc ? db / dc : 0), (seen ? "" : "new") }
' before.tsv after.tsv | sort -t$'\t' -k3,3gr | head -20
The columns are queryid, calls in the interval, milliseconds spent in the interval, then milliseconds, rows and blocks read per call, and new for a statement the first snapshot lacks. Rows and blocks per call rise when the statement itself does more work; time per call alone also rises when the database slows for any reason. The counters are cumulative, so an interval before the rollout exists only if someone snapshotted it then; during an incident, divide each statement's calls in the interval by your service's requests in the same minutes. A statement whose calls grew far faster than the service's requests, or whose rows or blocks per call jumped, is the one to read in the diff. Look up its text with SELECT query FROM pg_stat_statements WHERE queryid = QUERYID. For connections now, group pg_stat_activity by usename, application_name, client_addr and state where backend_type = 'client backend'; in awsvpc mode each task has its own client address. pg_stat_activity
If the release changed caching, check the cache too: ElastiCache publishes CacheHits, CacheMisses and CacheHitRate. ElastiCache metrics
Cloud SQL for PostgreSQL behind a Cloud Run service#
Cloud SQL publishes connections by application and, with query insights enabled, query latency distributions per database user and client address. The count of the distribution is the number of queries, so this PromQL query divides your database user's query rate by your service's request rate. Cloud SQL metrics · Query insights · Distributions in PromQL
sum(rate(cloudsql_googleapis_com:database_postgresql_insights_aggregate_latencies_count{monitored_resource="cloudsql_instance_database", database="DATABASE", user="SERVICE_DB_USER"}[5m]))
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m]))
For connections, chart cloudsql.googleapis.com/database/postgresql/num_backends_by_application grouped by its application label. database/cpu/utilization gives CPU as a fraction of the reserved CPU. Memorystore for Redis reports redis.googleapis.com/stats/cache_hit_ratio. Memorystore metrics
Downstream HTTP APIs#
An API owned by another team needs their per-client request rate: from their load balancer or gateway logs grouped by client address or API key, or from their own metrics labeled by caller. Compare it with your service's request count exactly as above. If the dependency is a Google Cloud API, serviceruntime.googleapis.com/api/request_count on the consumed_api resource carries method and credential_id labels, so you can chart your service's calls by method. Consumed API resource
Your own logs answer the same question from the caller's side if they record one line per outgoing attempt with the incoming request's ID. Count attempts per request ID, before and after.
A worked fixture: one timeout, two causes#
The fixture is illustrative: a fictional ECS service checkout with its own database user checkout, rolling from checkout:41 to checkout:42. New tasks began serving at 14:03 UTC and all traffic reached them by 14:12. Calls come from two pg_stat_statements snapshots per interval.
Case A, the release raised the load:
| Minute (UTC) | Requests/min | checkout calls/min | Calls per request | DatabaseConnections | CPU |
|---|---|---|---|---|---|
| 13:50 | 6,000 | 18,600 | 3.1 | 118 | 34% |
| 14:00 | 6,100 | 18,900 | 3.1 | 121 | 35% |
| 14:06 | 6,050 | 27,800 | 4.6 | 190 | 61% |
| 14:12 | 6,000 | 42,000 | 7.0 | 262 | 93% |
Requests stayed flat while calls per request more than doubled, rising with the new tasks' share of traffic. A new statement, run once per cart item, appears in the snapshot at 14:04. Roll back, even though old tasks fail too; the load they share comes from the new ones.
Case B, another caller:
| Minute (UTC) | Requests/min | checkout calls/min | Calls per request | reporting calls/min | CPU |
|---|---|---|---|---|---|
| 13:50 | 6,000 | 18,600 | 3.1 | 900 | 34% |
| 14:00 | 6,100 | 18,900 | 3.1 | 14,500 | 78% |
| 14:06 | 6,050 | 18,800 | 3.1 | 15,200 | 91% |
| 14:12 | 6,000 | 18,600 | 3.1 | 15,000 | 92% |
checkout's calls per request never moved, and neither did its connections or its statements' time and rows per call between snapshots, before and after 14:03, so its work per request never moved. checkout:42 keeps checkout:41's client timeout and retry count, and the service's own logs show one attempt per call throughout, so it is not retrying harder against the slow database. The reporting user's calls rose from 13:55, before the rollout. A rollback would leave CPU where it is.
Where the ratio misleads#
Retries feed back on themselves. A release that only lowered a timeout keeps calls per request flat until the dependency slows for any reason, and then multiplies attempts. The ratio jumps after the slowdown starts, which makes the release look innocent if you read only the first minutes. Look at attempts per request in your own logs, and at the diff for timeout and retry settings. Diff the two revisions
Pools open connections without queries. A larger pool or more tasks raises DatabaseConnections with the same calls per request, and the database can fail on connection slots alone. Count connections per task across both revisions. Connection budgets during rolling deployments
Cumulative counters can drop. pg_stat_statements_reset discards the counts, and once more distinct statements arrive than pg_stat_statements.max allows, the least-executed ones are discarded; a new statement that churns the view can shrink totals between snapshots. Query insights reports a bounded number of top queries, so a low-volume caller can be missing. pg_stat_statements settings
New instances start cold. Freshly started tasks or instances often run more queries until their caches warm, which raises calls per request briefly for any deploy, including a rollback. A ratio that returns to normal within minutes is warmup, not the release.
Questions#
If the old revision fails too, can the release still be the cause?
Yes. Both revisions share the dependency, so a release that overloads it degrades every caller. The split by revision cannot show this; the dependency's load per request from your service, before and after the rollout, can.
Which metric shows that our service, and not another caller, raised database load?
A count attributed to the caller: pg_stat_statements calls by role, Cloud SQL query insights query counts labeled by user, or connections grouped by user or application. Performance Insights db.load.avg grouped by db.user or db.host on RDS shows whose sessions spent the time, which rises for every caller when the database slows, so use it to corroborate a count, not on its own. Instance-wide DatabaseConnections and CPU cannot tell callers apart.
Should we raise the database's connection limit or size instead of rolling back?
Not as a first move during the incident. If calls or attempts per request rose with the release, or its pool size or task count raised connections, a larger database absorbs the change but keeps it. Roll back, then decide whether the new load is intended and size for it.