Learning center

Containers killed for memory after a deploy: leak or backlog

A container killed for memory soon after a deploy has two usual causes. The release needs more memory, or a dependency got slow and each request waiting on it holds memory, so in-flight requests and memory rise together. Plot memory against in-flight requests and dependency latency for each revision, and check whether the old revision grows too under the same latency.

Two symptoms#

  • On ECS, tasks of checkout:42 stop about 20 minutes after they start. describe-tasks shows the container's exitCode as 137 and a reason in the OutOfMemoryError category, which AWS documents as a container killed due to memory usage: its processes used more memory than the task definition allocated, or hit host or operating system limits. OutOfMemoryError · Exit code 137
  • On Cloud Run, requests to checkout-00042 fail with a 500 or 503, and Cloud Logging shows the error that begins While handling this request, the container instance was found to be using too much memory and was terminated. Every request still in progress on that instance ends with a 500. Cloud Run memory errors · Forced termination

Exit code 137 means the container received SIGKILL, which is not always a memory kill. ECS also sends SIGKILL to a container that does not exit within the stop timeout, 30 seconds by default, after a stop request. Read the container's reason before treating 137 as memory. StopTask

aws ecs list-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --service-name SERVICE \
  --desired-status STOPPED --query 'taskArns' --output text --no-cli-pager

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,stopped:stoppedAt,stopCode:stopCode,containers:containers[].{name:name,exit:exitCode,reason:reason}}' \
  --output json --no-cli-pager

Stopped tasks stay visible for at least an hour. describe-tasks

Two ways a deploy ends in memory kills#

A memory regression is the release using more memory: a larger cache, a new library, a buffer that is never released. Its memory grows with time or with requests served, at whatever latency the dependency has. The old revision, serving the same kind of traffic, stays where it was.

A backlog is the dependency's doing. Each request in flight holds memory for its request body, its partial response, its connection and whatever it has loaded. The number in flight is roughly the arrival rate times the time each request takes. If calls to a payments API slow from 150 milliseconds to 5 seconds at the same request rate, each instance holds about 33 times as many requests at once, and memory follows. A backlog grows on the old revision too, because it calls the same API.

On Cloud Run the revision's maximum concurrent requests setting caps how many requests one instance holds, so a backlog kills an instance only if that many requests do not fit in its memory. Raising the cap in the new revision is itself a release change. On ECS nothing caps in-flight requests unless the application server does. Cloud Run concurrency

The check: memory against in-flight requests and latency, per revision#

Put three series on one time axis for each revision: memory, in-flight requests and the dependency's latency as the service sees it.

On Cloud Run, all three come from built-in metrics, labelled by revision. container/memory/utilizations is the share of the memory limit in use across instances, container/max_request_concurrencies is the most requests each instance served at once in a minute, and request_latencies is the time from the request reaching the container to leaving it. In Cloud Monitoring, these distributions can be queried like Prometheus histograms. Cloud Run metrics · PromQL metric names

histogram_quantile(0.99, sum by (revision_name, le) (rate(run_googleapis_com:container_memory_utilizations_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m])))

histogram_quantile(0.99, sum by (revision_name, le) (rate(run_googleapis_com:container_max_request_concurrencies_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE", state="active"}[5m])))

histogram_quantile(0.95, sum by (revision_name, le) (rate(run_googleapis_com:request_latencies_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m])))

Request latency includes everything the request did. For the dependency itself, use the latency your application logs or traces for that call.

On ECS, Container Insights, if it is enabled on the cluster, writes a performance event per task to the log group /aws/ecs/containerinsights/CLUSTER/performance. Task events carry TaskDefinitionRevision and MemoryUtilized in MiB, so this Logs Insights query splits memory by revision. Container Insights performance events · Container Insights metrics

fields @timestamp, TaskId, TaskDefinitionRevision, MemoryUtilized
| filter Type = "Task" and TaskDefinitionFamily = "FAMILY"
| stats max(MemoryUtilized) as memory_mib by TaskDefinitionRevision, bin(1m)

ECS has no built-in per-task count of requests in flight. If the application logs one line per request with its duration and the dependency call's duration, this query gives both per task; requests per second times average duration in seconds approximates the number in flight. Replace the parse pattern with your log format and map task IDs to revisions with describe-tasks. Logs Insights functions · Logs Insights pct

fields @timestamp, @logStream, @message
| parse @message "duration_ms=* payments_ms=*" as duration_ms, payments_ms
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as requests, avg(toNumber(duration_ms)) as avg_ms, pct(toNumber(payments_ms), 95) as payments_p95_ms by task_id, bin(1m)

A worked fixture: same kill, two causes#

This fixture is illustrative. A fictional Cloud Run service checkout moved 50% of traffic to checkout-00042 at 14:00 UTC, with checkout-00041 serving the rest. Each row gives p99 memory utilization, p99 maximum concurrent requests per instance, and the p95 latency of calls to the payments API from the application's logs.

In the release case:

Minute (UTC)Memory 00041Memory 00042Concurrency 00041Concurrency 00042Payments p95
14:0052%48%66180 ms
14:1053%71%67190 ms
14:2052%94%76185 ms
14:2352%6180 ms

At 14:23 checkout-00042 instances are terminated for memory. Its memory climbed while its concurrency and the payments latency stayed flat, and the old revision held steady on the same traffic. That is a regression in the release. Move traffic back to checkout-00041 and compare the revisions. Diff the two revisions

In the backlog case:

Minute (UTC)Memory 00041Memory 00042Concurrency 00041Concurrency 00042Payments p95
14:0052%49%66180 ms
14:0655%54%910900 ms
14:1078%80%31332,400 ms
14:1497%96%58604,900 ms

Both revisions climb together, in step with concurrency, after the payments API slowed at 14:06. Instances of both are terminated at 14:15. Before ruling out a rollback, the engineer checks whether checkout-00042 added to the payments API's load: its payments calls per request and open connections match checkout-00041's, the payments team's per-client metrics show checkout's share of their requests and of their processing time flat since before 14:00, and the slowdown reached every caller. The release did not create this backlog, so rolling back moves traffic onto checkout-00041, which fails the same way. Work on the payments API and on how long the service waits for it: a shorter client timeout or a lower concurrency cap bounds the memory each instance can be asked to hold. Rule a cloud provider incident in or out

Where the check misleads#

A release can create its own backlog. New code that calls the dependency twice per request, drops a client timeout or runs a slower query makes latency and in-flight requests rise on the new revision only. The memory curve then looks like a backlog, and while the dependency keeps up, the old revision stays flat and its calls to the same dependency stay fast. If the added calls saturate a dependency both revisions share, both slow down together and the release looks like the backlog case. Compare the dependency's latency as each revision sees it, and this service's calls to the dependency per request before and after the rollout, before concluding that a rollback will not help. Did our deploy overload a dependency?

A slow leak may take longer than the rollout's overlap to show. If the old revision was fully replaced within minutes, compare the new revision's memory with the old revision's memory at the same age from before the deploy, and with earlier windows when the dependency was slow. The method in the collection's guide applies to memory as well as errors.

The metrics are coarse. Cloud Run samples these metrics every 60 seconds and may not show a sample for up to 120 seconds, and a spike that kills an instance can fall between samples. Container Insights must be enabled before the incident, and AWS notes that MemoryUtilized can be inaccurate for applications that use the Java ZGC garbage collector. Cloud Run metric sampling · Container Insights metrics

Not every memory failure kills the container. An out-of-memory error inside the runtime, such as java.lang.OutOfMemoryError, does not terminate a Cloud Run instance unless the container exceeds its limit, and on Cloud Run files written to the local file system count toward memory. Cloud Run troubleshooting · Cloud Run memory limits

Compare signals at each deploy#

Proofline's deploy verification reads the commit an environment is running and compares the signals that environment watches at checks after each deploy lands; a deploy whose signals got worse reads Regressed. It does not decide whether to roll back, and a verified deploy is not a safe one. Read a deploy's verification

Questions#

Does exit code 137 always mean an ECS container ran out of memory?

No. It means the container received SIGKILL. ECS also sends SIGKILL when a container does not stop within the stop timeout after a stop request. Check the container's reason for the OutOfMemoryError category before treating it as a memory kill.

How do I tell a memory leak from a backlog caused by a slow dependency?

Plot memory, in-flight requests and the dependency's latency for each revision. A leak grows memory with flat concurrency and flat latency, on the new revision only. A backlog grows memory together with concurrency and latency, on the old revision too.

Will adding memory fix memory kills after a deploy?

It gives the instance room, which helps a release that legitimately needs more. For a backlog it helps only where in-flight requests are capped: memory that holds a Cloud Run instance's full concurrency, or an ECS service's own limit, stops the kills while the dependency stays slow. Without a cap, more memory helps only while arrivals stay below what the slow dependency completes, so the backlog levels off; if the backlog keeps growing, it only raises the point at which instances die. Bound the wait with client timeouts and concurrency limits.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading