---
title: "Containers killed for memory after a deploy: leak or backlog | Proofline"
description: "A container killed for memory soon after a deploy has two usual causes."
url: https://proofline.sh/learn/container-memory-kills-after-deploy/
---

[Learning center](/learn/)

# Containers killed for memory after a deploy: leak or backlog

A container killed for memory soon after a deploy has two usual causes. The release needs more memory, or a dependency got slow and each request waiting on it holds memory, so in-flight requests and memory rise together. Plot memory against in-flight requests and dependency latency for each revision, and check whether the old revision grows too under the same latency.

Capacity overruns The Proofline team November 20, 2026 8 min read

## Two symptoms

-   On ECS, tasks of `checkout:42` stop about 20 minutes after they start. `describe-tasks` shows the container's `exitCode` as 137 and a `reason` in the `OutOfMemoryError` category, which AWS documents as a container killed due to memory usage: its processes used more memory than the task definition allocated, or hit host or operating system limits. [OutOfMemoryError](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/out-of-memory.html) · [Exit code 137](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-diags.html)
-   On Cloud Run, requests to `checkout-00042` fail with a 500 or 503, and Cloud Logging shows the error that begins `While handling this request, the container instance was found to be using too much memory and was terminated.` Every request still in progress on that instance ends with a 500. [Cloud Run memory errors](https://cloud.google.com/run/docs/troubleshooting) · [Forced termination](https://cloud.google.com/run/docs/container-contract)

Exit code 137 means the container received `SIGKILL`, which is not always a memory kill. ECS also sends `SIGKILL` to a container that does not exit within the stop timeout, 30 seconds by default, after a stop request. Read the container's `reason` before treating 137 as memory. [StopTask](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_StopTask.html)

```
aws ecs list-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --service-name SERVICE \
  --desired-status STOPPED --query 'taskArns' --output text --no-cli-pager

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks TASK_ID_1 TASK_ID_2 \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,started:startedAt,stopped:stoppedAt,stopCode:stopCode,containers:containers[].{name:name,exit:exitCode,reason:reason}}' \
  --output json --no-cli-pager
```

Stopped tasks stay visible for at least an hour. [describe-tasks](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-tasks.html)

## Two ways a deploy ends in memory kills

A memory regression is the release using more memory: a larger cache, a new library, a buffer that is never released. Its memory grows with time or with requests served, at whatever latency the dependency has. The old revision, serving the same kind of traffic, stays where it was.

A backlog is the dependency's doing. Each request in flight holds memory for its request body, its partial response, its connection and whatever it has loaded. The number in flight is roughly the arrival rate times the time each request takes. If calls to a payments API slow from 150 milliseconds to 5 seconds at the same request rate, each instance holds about 33 times as many requests at once, and memory follows. A backlog grows on the old revision too, because it calls the same API.

On Cloud Run the revision's maximum concurrent requests setting caps how many requests one instance holds, so a backlog kills an instance only if that many requests do not fit in its memory. Raising the cap in the new revision is itself a release change. On ECS nothing caps in-flight requests unless the application server does. [Cloud Run concurrency](https://cloud.google.com/run/docs/about-concurrency)

## The check: memory against in-flight requests and latency, per revision

Put three series on one time axis for each revision: memory, in-flight requests and the dependency's latency as the service sees it.

On Cloud Run, all three come from built-in metrics, labelled by revision. `container/memory/utilizations` is the share of the memory limit in use across instances, `container/max_request_concurrencies` is the most requests each instance served at once in a minute, and `request_latencies` is the time from the request reaching the container to leaving it. In Cloud Monitoring, these distributions can be queried like Prometheus histograms. [Cloud Run metrics](https://cloud.google.com/monitoring/api/metrics_gcp_p_z#gcp-run) · [PromQL metric names](https://cloud.google.com/monitoring/promql/promql-mapping)

```
histogram_quantile(0.99, sum by (revision_name, le) (rate(run_googleapis_com:container_memory_utilizations_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m])))

histogram_quantile(0.99, sum by (revision_name, le) (rate(run_googleapis_com:container_max_request_concurrencies_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE", state="active"}[5m])))

histogram_quantile(0.95, sum by (revision_name, le) (rate(run_googleapis_com:request_latencies_bucket{monitored_resource="cloud_run_revision", service_name="SERVICE"}[5m])))
```

Request latency includes everything the request did. For the dependency itself, use the latency your application logs or traces for that call.

On ECS, Container Insights, if it is enabled on the cluster, writes a performance event per task to the log group `/aws/ecs/containerinsights/CLUSTER/performance`. Task events carry `TaskDefinitionRevision` and `MemoryUtilized` in MiB, so this Logs Insights query splits memory by revision. [Container Insights performance events](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-reference-performance-logs-ECS.html) · [Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html)

```
fields @timestamp, TaskId, TaskDefinitionRevision, MemoryUtilized
| filter Type = "Task" and TaskDefinitionFamily = "FAMILY"
| stats max(MemoryUtilized) as memory_mib by TaskDefinitionRevision, bin(1m)
```

ECS has no built-in per-task count of requests in flight. If the application logs one line per request with its duration and the dependency call's duration, this query gives both per task; requests per second times average duration in seconds approximates the number in flight. Replace the `parse` pattern with your log format and map task IDs to revisions with `describe-tasks`. [Logs Insights functions](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-operations-functions.html) · [Logs Insights `pct`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html)

```
fields @timestamp, @logStream, @message
| parse @message "duration_ms=* payments_ms=*" as duration_ms, payments_ms
| parse @logStream "*/*/*" as prefix, container, task_id
| stats count(*) as requests, avg(toNumber(duration_ms)) as avg_ms, pct(toNumber(payments_ms), 95) as payments_p95_ms by task_id, bin(1m)
```

## A worked fixture: same kill, two causes

This fixture is illustrative. A fictional Cloud Run service `checkout` moved 50% of traffic to `checkout-00042` at 14:00 UTC, with `checkout-00041` serving the rest. Each row gives p99 memory utilization, p99 maximum concurrent requests per instance, and the p95 latency of calls to the payments API from the application's logs.

In the release case:

| Minute (UTC) | Memory 00041 | Memory 00042 | Concurrency 00041 | Concurrency 00042 | Payments p95 |
| --- | --- | --- | --- | --- | --- |
| 14:00 | 52% | 48% | 6 | 6 | 180 ms |
| 14:10 | 53% | 71% | 6 | 7 | 190 ms |
| 14:20 | 52% | 94% | 7 | 6 | 185 ms |
| 14:23 | 52% |  | 6 |  | 180 ms |

At 14:23 `checkout-00042` instances are terminated for memory. Its memory climbed while its concurrency and the payments latency stayed flat, and the old revision held steady on the same traffic. That is a regression in the release. Move traffic back to `checkout-00041` and compare the revisions. [Diff the two revisions](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

In the backlog case:

| Minute (UTC) | Memory 00041 | Memory 00042 | Concurrency 00041 | Concurrency 00042 | Payments p95 |
| --- | --- | --- | --- | --- | --- |
| 14:00 | 52% | 49% | 6 | 6 | 180 ms |
| 14:06 | 55% | 54% | 9 | 10 | 900 ms |
| 14:10 | 78% | 80% | 31 | 33 | 2,400 ms |
| 14:14 | 97% | 96% | 58 | 60 | 4,900 ms |

Both revisions climb together, in step with concurrency, after the payments API slowed at 14:06. Instances of both are terminated at 14:15. Before ruling out a rollback, the engineer checks whether `checkout-00042` added to the payments API's load: its payments calls per request and open connections match `checkout-00041`'s, the payments team's per-client metrics show checkout's share of their requests and of their processing time flat since before 14:00, and the slowdown reached every caller. The release did not create this backlog, so rolling back moves traffic onto `checkout-00041`, which fails the same way. Work on the payments API and on how long the service waits for it: a shorter client timeout or a lower concurrency cap bounds the memory each instance can be asked to hold. [Rule a cloud provider incident in or out](/learn/cloud-provider-incident-during-rollout/)

## Where the check misleads

A release can create its own backlog. New code that calls the dependency twice per request, drops a client timeout or runs a slower query makes latency and in-flight requests rise on the new revision only. The memory curve then looks like a backlog, and while the dependency keeps up, the old revision stays flat and its calls to the same dependency stay fast. If the added calls saturate a dependency both revisions share, both slow down together and the release looks like the backlog case. Compare the dependency's latency as each revision sees it, and this service's calls to the dependency per request before and after the rollout, before concluding that a rollback will not help. [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/)

A slow leak may take longer than the rollout's overlap to show. If the old revision was fully replaced within minutes, compare the new revision's memory with the old revision's memory at the same age from before the deploy, and with earlier windows when the dependency was slow. The method in the [collection's guide](/learn/bad-release-or-unhealthy-dependency/#before-the-rollout) applies to memory as well as errors.

The metrics are coarse. Cloud Run samples these metrics every 60 seconds and may not show a sample for up to 120 seconds, and a spike that kills an instance can fall between samples. Container Insights must be enabled before the incident, and AWS notes that `MemoryUtilized` can be inaccurate for applications that use the Java ZGC garbage collector. [Cloud Run metric sampling](https://cloud.google.com/monitoring/api/metrics_gcp_p_z#gcp-run) · [Container Insights metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-metrics-ECS.html)

Not every memory failure kills the container. An out-of-memory error inside the runtime, such as `java.lang.OutOfMemoryError`, does not terminate a Cloud Run instance unless the container exceeds its limit, and on Cloud Run files written to the local file system count toward memory. [Cloud Run troubleshooting](https://cloud.google.com/run/docs/troubleshooting) · [Cloud Run memory limits](https://cloud.google.com/run/docs/configuring/services/memory-limits)

## Compare signals at each deploy

Proofline's deploy verification reads the commit an environment is running and compares the signals that environment watches at checks after each deploy lands; a deploy whose signals got worse reads **Regressed**. It does not decide whether to roll back, and a verified deploy is not a safe one. [Read a deploy's verification](/help/read-deploy-verification/)

## Questions

### Does exit code 137 always mean an ECS container ran out of memory?

No. It means the container received `SIGKILL`. ECS also sends `SIGKILL` when a container does not stop within the stop timeout after a stop request. Check the container's `reason` for the `OutOfMemoryError` category before treating it as a memory kill.

### How do I tell a memory leak from a backlog caused by a slow dependency?

Plot memory, in-flight requests and the dependency's latency for each revision. A leak grows memory with flat concurrency and flat latency, on the new revision only. A backlog grows memory together with concurrency and latency, on the old revision too.

### Will adding memory fix memory kills after a deploy?

It gives the instance room, which helps a release that legitimately needs more. For a backlog it helps only where in-flight requests are capped: memory that holds a Cloud Run instance's full concurrency, or an ECS service's own limit, stops the kills while the dependency stays slow. Without a cap, more memory helps only while arrivals stay below what the slow dependency completes, so the backlog levels off; if the backlog keeps growing, it only raises the point at which instances die. Bound the wait with client timeouts and concurrency limits.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Did our deploy overload a dependency that was healthy before it?](/learn/did-our-deploy-overload-a-dependency/)
    
    Capacity overruns The Proofline team November 20, 2026 10 min read
    
-   ### [What changed between two ECS task definitions or Cloud Run revisions?](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)
    
    Deployment failures The Proofline team November 20, 2026 8 min read
