Two symptoms#
- On ECS, tasks of
checkout:42stop about 20 minutes after they start.describe-tasksshows the container'sexitCodeas 137 and areasonin theOutOfMemoryErrorcategory, which AWS documents as a container killed due to memory usage: its processes used more memory than the task definition allocated, or hit host or operating system limits. OutOfMemoryError · Exit code 137 - On Cloud Run, requests to
checkout-00042fail with a 500 or 503, and Cloud Logging shows the error that beginsWhile handling this request, the container instance was found to be using too much memory and was terminated.Every request still in progress on that instance ends with a 500. Cloud Run memory errors · Forced termination
Exit code 137 means the container received SIGKILL, which is not always a memory kill. ECS also sends SIGKILL to a container that does not exit within the stop timeout, 30 seconds by default, after a stop request. Read the container's reason before treating 137 as memory. StopTask
aws ecs list-tasks \
--profile PROFILE --region REGION --cluster CLUSTER --service-name SERVICE \
--desired-status STOPPED --query 'taskArns' --output text --no-cli-pager
aws ecs describe-tasks \
--profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 \
--query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,stopped:stoppedAt,stopCode:stopCode,containers:containers[].{name:name,exit:exitCode,reason:reason}}' \
--output json --no-cli-pager
Stopped tasks stay visible for at least an hour. describe-tasks
Two ways a deploy ends in memory kills#
A memory regression is the release using more memory: a larger cache, a new library, a buffer that is never released. Its memory grows with time or with requests served, at whatever latency the dependency has. The old revision, serving the same kind of traffic, stays where it was.
A backlog is the dependency's doing. Each request in flight holds memory for its request body, its partial response, its connection and whatever it has loaded. The number in flight is roughly the arrival rate times the time each request takes. If calls to a payments API slow from 150 milliseconds to 5 seconds at the same request rate, each instance holds about 33 times as many requests at once, and memory follows. A backlog grows on the old revision too, because it calls the same API.
On Cloud Run the revision's maximum concurrent requests setting caps how many requests one instance holds, so a backlog kills an instance only if that many requests do not fit in its memory. Raising the cap in the new revision is itself a release change. On ECS nothing caps in-flight requests unless the application server does. Cloud Run concurrency
The check: memory against in-flight requests and latency, per revision#
Put three series on one time axis for each revision: memory, in-flight requests and the dependency's latency as the service sees it.
On Cloud Run, all three come from built-in metrics, labelled by revision. container/memory/utilizations is the share of the memory limit in use across instances, container/max_request_concurrencies is the most requests each instance served at once in a minute, and request_latencies is the time from the request reaching the container to leaving it. In Cloud Monitoring, these distributions can be queried like Prometheus histograms. Cloud Run metrics · PromQL metric names
histogram_quantile(0.99, sum by (revision_name, le) (rate(run_googleapis_com:container_memory_utilizations_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m])))
histogram_quantile(0.99, sum by (revision_name, le) (rate(run_googleapis_com:container_max_request_concurrencies_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE", state="active"}[5m])))
histogram_quantile(0.95, sum by (revision_name, le) (rate(run_googleapis_com:request_latencies_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m])))
Request latency includes everything the request did. For the dependency itself, use the latency your application logs or traces for that call.
On ECS, Container Insights, if it is enabled on the cluster, writes a performance event per task to the log group /aws/ecs/containerinsights/CLUSTER/performance. Task events carry TaskDefinitionRevision and MemoryUtilized in MiB, so this Logs Insights query splits memory by revision. Container Insights performance events · Container Insights metrics
fields @timestamp, TaskId, TaskDefinitionRevision, MemoryUtilized
| filter Type = "Task" and TaskDefinitionFamily = "FAMILY"
| stats max(MemoryUtilized) as memory_mib by TaskDefinitionRevision, bin(1m)
ECS has no built-in per-task count of requests in flight. If the application logs one line per request with its duration and the dependency call's duration, this query gives both per task; requests per second times average duration in seconds approximates the number in flight. Replace the parse pattern with your log format and map task IDs to revisions with describe-tasks. Logs Insights functions · Logs Insights pct
fields @timestamp, @logStream, @message
| parse @message "duration_ms=* payments_ms=*" as duration_ms, payments_ms
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as requests, avg(toNumber(duration_ms)) as avg_ms, pct(toNumber(payments_ms), 95) as payments_p95_ms by task_id, bin(1m)
A worked fixture: same kill, two causes#
This fixture is illustrative. A fictional Cloud Run service checkout moved 50% of traffic to checkout-00042 at 14:00 UTC, with checkout-00041 serving the rest. Each row gives p99 memory utilization, p99 maximum concurrent requests per instance, and the p95 latency of calls to the payments API from the application's logs.
In the release case:
| Minute (UTC) | Memory 00041 | Memory 00042 | Concurrency 00041 | Concurrency 00042 | Payments p95 |
|---|---|---|---|---|---|
| 14:00 | 52% | 48% | 6 | 6 | 180 ms |
| 14:10 | 53% | 71% | 6 | 7 | 190 ms |
| 14:20 | 52% | 94% | 7 | 6 | 185 ms |
| 14:23 | 52% | 6 | 180 ms |
At 14:23 checkout-00042 instances are terminated for memory. Its memory climbed while its concurrency and the payments latency stayed flat, and the old revision held steady on the same traffic. That is a regression in the release. Move traffic back to checkout-00041 and compare the revisions. Diff the two revisions
In the backlog case:
| Minute (UTC) | Memory 00041 | Memory 00042 | Concurrency 00041 | Concurrency 00042 | Payments p95 |
|---|---|---|---|---|---|
| 14:00 | 52% | 49% | 6 | 6 | 180 ms |
| 14:06 | 55% | 54% | 9 | 10 | 900 ms |
| 14:10 | 78% | 80% | 31 | 33 | 2,400 ms |
| 14:14 | 97% | 96% | 58 | 60 | 4,900 ms |
Both revisions climb together, in step with concurrency, after the payments API slowed at 14:06. Instances of both are terminated at 14:15. Before ruling out a rollback, the engineer checks whether checkout-00042 added to the payments API's load: its payments calls per request and open connections match checkout-00041's, the payments team's per-client metrics show checkout's share of their requests and of their processing time flat since before 14:00, and the slowdown reached every caller. The release did not create this backlog, so rolling back moves traffic onto checkout-00041, which fails the same way. Work on the payments API and on how long the service waits for it: a shorter client timeout or a lower concurrency cap bounds the memory each instance can be asked to hold. Rule a cloud provider incident in or out
Where the check misleads#
A release can create its own backlog. New code that calls the dependency twice per request, drops a client timeout or runs a slower query makes latency and in-flight requests rise on the new revision only. The memory curve then looks like a backlog, and while the dependency keeps up, the old revision stays flat and its calls to the same dependency stay fast. If the added calls saturate a dependency both revisions share, both slow down together and the release looks like the backlog case. Compare the dependency's latency as each revision sees it, and this service's calls to the dependency per request before and after the rollout, before concluding that a rollback will not help. Did our deploy overload a dependency?
A slow leak may take longer than the rollout's overlap to show. If the old revision was fully replaced within minutes, compare the new revision's memory with the old revision's memory at the same age from before the deploy, and with earlier windows when the dependency was slow. The method in the collection's guide applies to memory as well as errors.
The metrics are coarse. Cloud Run samples these metrics every 60 seconds and may not show a sample for up to 120 seconds, and a spike that kills an instance can fall between samples. Container Insights must be enabled before the incident, and AWS notes that MemoryUtilized can be inaccurate for applications that use the Java ZGC garbage collector. Cloud Run metric sampling · Container Insights metrics
Not every memory failure kills the container. An out-of-memory error inside the runtime, such as java.lang.OutOfMemoryError, does not terminate a Cloud Run instance unless the container exceeds its limit, and on Cloud Run files written to the local file system count toward memory. Cloud Run troubleshooting · Cloud Run memory limits
Compare signals at each deploy#
Proofline's deploy verification reads the commit an environment is running and compares the signals that environment watches at checks after each deploy lands; a deploy whose signals got worse reads Regressed. It does not decide whether to roll back, and a verified deploy is not a safe one. Read a deploy's verification
Questions#
Does exit code 137 always mean an ECS container ran out of memory?
No. It means the container received SIGKILL. ECS also sends SIGKILL when a container does not stop within the stop timeout after a stop request. Check the container's reason for the OutOfMemoryError category before treating it as a memory kill.
How do I tell a memory leak from a backlog caused by a slow dependency?
Plot memory, in-flight requests and the dependency's latency for each revision. A leak grows memory with flat concurrency and flat latency, on the new revision only. A backlog grows memory together with concurrency and latency, on the old revision too.
Will adding memory fix memory kills after a deploy?
It gives the instance room, which helps a release that legitimately needs more. For a backlog it helps only where in-flight requests are capped: memory that holds a Cloud Run instance's full concurrency, or an ECS service's own limit, stops the kills while the dependency stays slow. Without a cap, more memory helps only while arrivals stay below what the slow dependency completes, so the backlog levels off; if the backlog keeps growing, it only raises the point at which instances die. Bound the wait with client timeouts and concurrency limits.