---
title: "Dependency timeouts only from new ECS tasks or Cloud Run revisions | Proofline"
description: "When only the new tasks or the new revision time out calling a dependency, compare the network path each one uses before blaming the dependency."
url: https://proofline.sh/learn/dependency-timeouts-only-from-new-tasks/
---

[Learning center](/learn/)

# Dependency timeouts only from new ECS tasks or Cloud Run revisions

When only the new tasks or the new revision time out calling a dependency, compare the network path each one uses before blaming the dependency. Then open a new connection from an old instance: if it fails too, new connections fail for everyone and the old instances were coasting on connections they already held. Either the dependency stopped accepting connections or a shared part of the path changed in place, so check the address and the dependency's listener before blaming either.

Network outages The Proofline team November 20, 2026 9 min read

## Two symptoms

Timeouts that only new instances report look like a release problem, and often are one. Two common forms:

-   The tasks of task definition `checkout:42` log connect timeouts to the orders database on port 5432 from 14:02 UTC. Tasks of `checkout:41`, in the same subnets and the same minutes, keep serving requests that use the same database.
-   Requests on the Cloud Run revision `checkout-00042` wait 10 seconds and fail calling a payments API, while `checkout-00041` completes the same calls in about 200 milliseconds. Splitting the request logs by `resource.labels.revision_name` shows every timeout on `checkout-00042`. [Cloud Run request log fields](https://cloud.google.com/run/docs/logging)

Two different causes produce both. The deploy may have changed the path new instances take to the dependency. Or the dependency may have stopped accepting new connections while old instances kept working over connections they opened earlier. The first is the release, even though the dependency is fine. The second is the dependency, even though only new instances notice.

## Why new instances can take a different path

On ECS with `awsvpc` networking, each task gets its own network interface in the subnets and security groups of the deployment that started it. The service's network configuration is recorded per deployment, so a deploy that also changes subnets or security groups changes them for new tasks only. A database whose security group allows the old task security group by ID then drops the new tasks' traffic, which the client sees as a connect timeout. [Deployment network configuration](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_Deployment.html) · [Service revisions](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ServiceRevision.html)

On Cloud Run, every configuration change creates a new immutable revision, and VPC egress is part of the revision's template: Direct VPC egress or a Serverless VPC Access connector, the network and subnet, network tags, and the egress setting. `private-ranges-only` sends only internal addresses through the VPC. `all-traffic` sends everything through it, so a revision that switches to `all-traffic` reaches a public API only if that network has Cloud NAT. Firewall rules that match network tags apply to a revision only if it carries the tags. [Cloud Run revisions](https://cloud.google.com/run/docs/managing/revisions) · [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) · [Serverless VPC Access connectors](https://cloud.google.com/run/docs/configuring/vpc-connectors) · [Cloud NAT for all-traffic egress](https://cloud.google.com/run/docs/configuring/networking-best-practices)

The release can also change the path from inside the container: a new hostname or port in configuration, a proxy setting, a shorter client timeout. Those show up in a diff of the task definitions or revisions. [Diff the two revisions](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)

## Compare the paths

On ECS, list the deployments with their network configuration, then read the dependency's inbound rules and the route tables of each subnet. [describe-services](https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-services.html) · [describe-security-groups](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-security-groups.html) · [describe-route-tables](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-route-tables.html)

```
aws ecs describe-services \
  --profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
  --query 'services[0].deployments[].{status:status,definition:taskDefinition,subnets:networkConfiguration.awsvpcConfiguration.subnets,securityGroups:networkConfiguration.awsvpcConfiguration.securityGroups,publicIp:networkConfiguration.awsvpcConfiguration.assignPublicIp}' \
  --output json --no-cli-pager

aws ec2 describe-security-groups \
  --profile PROFILE --region REGION --group-ids DEPENDENCY_SECURITY_GROUP_ID \
  --query 'SecurityGroups[0].IpPermissions[].{from:FromPort,to:ToPort,groups:UserIdGroupPairs[].GroupId,cidrs:IpRanges[].CidrIp}' \
  --output json --no-cli-pager

aws ec2 describe-route-tables \
  --profile PROFILE --region REGION \
  --filters Name=association.subnet-id,Values=OLD_SUBNET_ID,NEW_SUBNET_ID \
  --query 'RouteTables[].{table:RouteTableId,subnets:Associations[].SubnetId,routes:Routes[].{to:DestinationCidrBlock,nat:NatGatewayId,gateway:GatewayId,state:State}}' \
  --output json --no-cli-pager
```

A subnet with no explicit association uses the VPC's main route table, which this filter does not return. Check the security groups' outbound rules and the subnets' network ACLs too if the inbound side looks right.

On Cloud Run, print the annotations of both revisions and compare them. The VPC settings appear as `run.googleapis.com/vpc-access-egress` together with `run.googleapis.com/network-interfaces` for Direct VPC egress or `run.googleapis.com/vpc-access-connector` for a connector. [gcloud run revisions describe](https://cloud.google.com/sdk/gcloud/reference/run/revisions/describe)

```
gcloud run revisions describe OLD_REVISION --project PROJECT --region REGION --format 'yaml(metadata.annotations)'
gcloud run revisions describe NEW_REVISION --project PROJECT --region REGION --format 'yaml(metadata.annotations)'
```

## Open the same connection from an old and a new instance

A configuration comparison shows what was intended. A connection test shows what happens now, and it answers the second question: can anyone open a new connection to the dependency?

On ECS, ECS Exec runs a command inside a running container. It works only on tasks started with `enableExecuteCommand` turned on, and it cannot be turned on for a task that is already running. It also needs the Session Manager plugin for the AWS CLI, a task role with the Session Manager permissions, an `ssmmessages` VPC endpoint when the task has no internet path, Fargate platform version 1.4.0 or later on Linux, and a writable root file system. Commands run as root inside the production container. Confirm a task is ready before you rely on it. [ECS Exec considerations](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) · [ECS Exec prerequisites](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec-run.html)

```
aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks OLD_TASK_ID NEW_TASK_ID \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,exec:enableExecuteCommand,agent:containers[0].managedAgents[?name==`ExecuteCommandAgent`].lastStatus}' \
  --output table --no-cli-pager

aws ecs execute-command \
  --profile PROFILE --region REGION --cluster CLUSTER --task OLD_TASK_ID --container CONTAINER \
  --interactive --command "nc -vz -w 5 DEPENDENCY_HOST 5432"
```

Run the same command against `NEW_TASK_ID`. Use whatever connection tool the image contains; `nc` is only an example, and it checks TCP alone. A database that has run out of connection slots still accepts the TCP connection and then refuses the session, so for a database also open a real client session, such as `psql` with the service's credentials, from each task. PostgreSQL refuses it with `FATAL: sorry, too many clients already`. `getent hosts DEPENDENCY_HOST` shows whether both tasks resolve the name to the same address. [execute-command](https://docs.aws.amazon.com/cli/latest/reference/ecs/execute-command.html) · [PostgreSQL `max_connections`](https://www.postgresql.org/docs/current/runtime-config-connection.html)

| Old task, new connection | New task, new connection | Points to |
| --- | --- | --- |
| Connects | Times out | The new deployment's path: compare security groups, subnets and routes |
| Fails | Fails | New connections fail from both, and old tasks were using connections they already held. The dependency may refuse them, or a shared part of the path changed in place: a DNS record now pointing elsewhere, or an edited security group or network ACL rule. Compare the address both resolve with the remote address of an old task's open connections (`ss -tn` or `netstat -tn` in the old task), then read the dependency's connection count and listener |
| Connects | Connects | Not the network path; compare client settings and code in the release |

Without ECS Exec, three other sources help. VPC Flow Logs, if they were enabled, record each interface's flows with an `action` of `ACCEPT` or `REJECT`; a `REJECT` on the dependency's interface from a new task's address points at its security group or network ACL. If the flow log format includes `ecs-task-definition-arn`, you can split flows by task definition directly. Reachability Analyzer models the path from a task's network interface to the dependency's address and port and names the blocking component; it reads configuration only, sends no packets and is billed per analysis. On Cloud Run, Connectivity Tests can analyze a revision's path when it uses a Serverless VPC Access connector. [Flow log records](https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html) · [Reachability Analyzer](https://docs.aws.amazon.com/vpc/latest/reachability/how-reachability-analyzer-works.html) · [Connectivity Tests sources](https://cloud.google.com/network-intelligence-center/docs/connectivity-tests/concepts/overview)

```
fields @timestamp, interfaceId, srcAddr, dstAddr, dstPort, action
| filter dstPort = 5432 and srcAddr in ["OLD_TASK_IP", "NEW_TASK_IP"]
| stats count(*) as flows by srcAddr, action, bin(5m)
```

## A worked fixture: two timeouts that look the same

This fixture is illustrative. A fictional ECS service `checkout` moved from `checkout:41` to `checkout:42` at 14:00 UTC. Both incidents begin with failed connections to the orders database only from `checkout:42` tasks.

In the first incident, the deploy also moved the tasks to a new security group:

| Check | `checkout:41` task | `checkout:42` task |
| --- | --- | --- |
| Deployment security groups | `sg-0aa1` (checkout-tasks) | `sg-0bb2` (checkout-tasks-v2) |
| Database security group allows 5432 from | `sg-0aa1` | not allowed |
| `nc -vz -w 5` to the database, 14:09 | connects | times out |
| Flow logs on the database interface | `ACCEPT` | `REJECT` |

The database is healthy and the deploy changed the path. Either add the inbound rule for the new security group or roll the service back to the previous network configuration; the database needs nothing.

In the second incident, nothing about the network changed:

| Check | `checkout:41` task | `checkout:42` task |
| --- | --- | --- |
| Deployment security groups | `sg-0aa1` | `sg-0aa1` |
| `nc -vz -w 5` to the database, 14:09 | connects | connects |
| New `psql` session, 14:09 | `FATAL: sorry, too many clients already` | `FATAL: sorry, too many clients already` |
| Requests served | succeed over pooled connections opened before 13:50 | fail; the pool is empty at start |

The old tasks' open connections go to the same address the name resolves to now, and no security group or network ACL rule changed after 13:50. The database's connection count sits at its limit from 13:52. The old tasks look healthy only because they opened their connections before the database stopped accepting new ones. The slots filled at 13:52, before the deploy, so the release did not cause the shortage. At the limit every new session is refused, so held connections cannot show whether `checkout:42` adds demand; its attempts can. Both task definitions keep the same pool size and reconnect backoff, and the database log shows `too many clients` refusals at the same rate per task before and after 14:00, and the service ran 4 tasks throughout, replacing them one for one, so total refusals held flat too. A rollback would replace the old tasks with fresh ones that need new connections too, and make the incident larger. Look at the database's connection count and the proxy or listener in front of it. [Connection budgets during rolling deployments](/learn/connection-limits/)

The Cloud Run version of the first incident is a revision whose annotations show `run.googleapis.com/vpc-access-egress: all-traffic` where the old revision had `private-ranges-only`, deployed to a network without Cloud NAT: the payments API, a public address, times out from `checkout-00042` only.

## Limits of the comparison

The comparison assumes old instances are still running. ECS replaces them during the rollout, and a stopped task's network interface is detached and deleted as it deprovisions, so run the tests while the overlap lasts. On Cloud Run you can still send requests to a previous revision through a revision tag. [Task lifecycle](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-lifecycle-explanation.html) · [Test the previous revision with a tag](/learn/test-previous-cloud-run-revision-with-tag/)

Flow logs show accepted and rejected packets, not why a connection hung. They do not record traffic to the Amazon DNS server, and they are not delivered in real time. With Direct VPC egress, VPC Flow Logs do not name the Cloud Run revision, and firewall rules logging is not supported. Reachability Analyzer reads IPv4 configuration only and does not consider the health of load balancer targets. [Flow log limitations](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-limitations.html) · [Direct VPC egress limitations](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc)

A successful `nc` proves only that a TCP connection opens. TLS, authentication and slow queries can still fail afterwards; a credential the new tasks read at startup is one example. [A secret rotated during the rollout](/learn/secret-rotation-during-rollout/)

New tasks also add load. If the dependency started timing out because the rollout's overlap opened more connections than it could take, every instance eventually suffers and the timing points at the release. [Did our deploy overload a dependency?](/learn/did-our-deploy-overload-a-dependency/) · [The general method](/learn/bad-release-or-unhealthy-dependency/#limits)

## Questions

### Why do only new ECS tasks time out connecting to the database?

New tasks start with the subnets and security groups of the deployment that launched them, and with empty connection pools. Either the deploy changed their network path, or new connections fail for every task and old tasks are still using connections they opened earlier. Open a new connection from an old task to tell the two apart. If that fails too, check whether the database is refusing connections or a shared DNS record or network rule changed in place.

### Can I use ECS Exec on a task that is already running?

Only if the task was started with execute command turned on. ECS Exec cannot be turned on for existing tasks; it applies to tasks started after you enable it on the service or task.

### Are Cloud Run VPC egress settings per revision?

Yes. They are set in the revision template, so each revision carries its own connector or Direct VPC network, subnet, network tags and egress setting. Compare the annotations of the old and new revisions.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [What changed between two ECS task definitions or Cloud Run revisions?](/learn/diff-ecs-task-definitions-and-cloud-run-revisions/)
    
    Deployment failures The Proofline team November 20, 2026 8 min read
    
-   ### [How rolling deployments change your PostgreSQL connection budget](/learn/connection-limits/)
    
    Capacity overruns The Proofline team November 5, 2026 8 min read
