Learning center

Dependency timeouts only from new ECS tasks or Cloud Run revisions

When only the new tasks or the new revision time out calling a dependency, compare the network path each one uses before blaming the dependency. Then open a new connection from an old instance: if it fails too, new connections fail for everyone and the old instances were coasting on connections they already held. Either the dependency stopped accepting connections or a shared part of the path changed in place, so check the address and the dependency's listener before blaming either.

Two symptoms#

Timeouts that only new instances report look like a release problem, and often are one. Two common forms:

  • The tasks of task definition checkout:42 log connect timeouts to the orders database on port 5432 from 14:02 UTC. Tasks of checkout:41, in the same subnets and the same minutes, keep serving requests that use the same database.
  • Requests on the Cloud Run revision checkout-00042 wait 10 seconds and fail calling a payments API, while checkout-00041 completes the same calls in about 200 milliseconds. Splitting the request logs by resource.labels.revision_name shows every timeout on checkout-00042. Cloud Run request log fields

Two different causes produce both. The deploy may have changed the path new instances take to the dependency. Or the dependency may have stopped accepting new connections while old instances kept working over connections they opened earlier. The first is the release, even though the dependency is fine. The second is the dependency, even though only new instances notice.

Why new instances can take a different path#

On ECS with awsvpc networking, each task gets its own network interface in the subnets and security groups of the deployment that started it. The service's network configuration is recorded per deployment, so a deploy that also changes subnets or security groups changes them for new tasks only. A database whose security group allows the old task security group by ID then drops the new tasks' traffic, which the client sees as a connect timeout. Deployment network configuration · Service revisions

On Cloud Run, every configuration change creates a new immutable revision, and VPC egress is part of the revision's template: Direct VPC egress or a Serverless VPC Access connector, the network and subnet, network tags, and the egress setting. private-ranges-only sends only internal addresses through the VPC. all-traffic sends everything through it, so a revision that switches to all-traffic reaches a public API only if that network has Cloud NAT. Firewall rules that match network tags apply to a revision only if it carries the tags. Cloud Run revisions · Direct VPC egress · Serverless VPC Access connectors · Cloud NAT for all-traffic egress

The release can also change the path from inside the container: a new hostname or port in configuration, a proxy setting, a shorter client timeout. Those show up in a diff of the task definitions or revisions. Diff the two revisions

Compare the paths#

On ECS, list the deployments with their network configuration, then read the dependency's inbound rules and the route tables of each subnet. describe-services · describe-security-groups · describe-route-tables

aws ecs describe-services \
  --profile PROFILE --region REGION --cluster CLUSTER --services SERVICE \
  --query 'services[0].deployments[].{status:status,definition:taskDefinition,subnets:networkConfiguration.awsvpcConfiguration.subnets,securityGroups:networkConfiguration.awsvpcConfiguration.securityGroups,publicIp:networkConfiguration.awsvpcConfiguration.assignPublicIp}' \
  --output json --no-cli-pager

aws ec2 describe-security-groups \
  --profile PROFILE --region REGION --group-ids DEPENDENCY_SECURITY_GROUP_ID \
  --query 'SecurityGroups[0].IpPermissions[].{from:FromPort,to:ToPort,groups:UserIdGroupPairs[].GroupId,cidrs:IpRanges[].CidrIp}' \
  --output json --no-cli-pager

aws ec2 describe-route-tables \
  --profile PROFILE --region REGION \
  --filters Name=association.subnet-id,Values=OLD_SUBNET_ID,NEW_SUBNET_ID \
  --query 'RouteTables[].{table:RouteTableId,subnets:Associations[].SubnetId,routes:Routes[].{to:DestinationCidrBlock,nat:NatGatewayId,gateway:GatewayId,state:State}}' \
  --output json --no-cli-pager

A subnet with no explicit association uses the VPC's main route table, which this filter does not return. Check the security groups' outbound rules and the subnets' network ACLs too if the inbound side looks right.

On Cloud Run, print the annotations of both revisions and compare them. The VPC settings appear as run.googleapis.com/vpc-access-egress together with run.googleapis.com/network-interfaces for Direct VPC egress or run.googleapis.com/vpc-access-connector for a connector. gcloud run revisions describe

gcloud run revisions describe OLD_REVISION --project PROJECT --region REGION --format 'yaml(metadata.annotations)'
gcloud run revisions describe NEW_REVISION --project PROJECT --region REGION --format 'yaml(metadata.annotations)'

Open the same connection from an old and a new instance#

A configuration comparison shows what was intended. A connection test shows what happens now, and it answers the second question: can anyone open a new connection to the dependency?

On ECS, ECS Exec runs a command inside a running container. It works only on tasks started with enableExecuteCommand turned on, and it cannot be turned on for a task that is already running. It also needs the Session Manager plugin for the AWS CLI, a task role with the Session Manager permissions, an ssmmessages VPC endpoint when the task has no internet path, Fargate platform version 1.4.0 or later on Linux, and a writable root file system. Commands run as root inside the production container. Confirm a task is ready before you rely on it. ECS Exec considerations · ECS Exec prerequisites

aws ecs describe-tasks \
  --profile PROFILE --region REGION --cluster CLUSTER --tasks OLD_TASK_ID NEW_TASK_ID \
  --query 'tasks[].{task:taskArn,definition:taskDefinitionArn,exec:enableExecuteCommand,agent:containers[0].managedAgents[?name==`ExecuteCommandAgent`].lastStatus}' \
  --output table --no-cli-pager

aws ecs execute-command \
  --profile PROFILE --region REGION --cluster CLUSTER --task OLD_TASK_ID --container CONTAINER \
  --interactive --command "nc -vz -w 5 DEPENDENCY_HOST 5432"

Run the same command against NEW_TASK_ID. Use whatever connection tool the image contains; nc is only an example, and it checks TCP alone. A database that has run out of connection slots still accepts the TCP connection and then refuses the session, so for a database also open a real client session, such as psql with the service's credentials, from each task. PostgreSQL refuses it with FATAL: sorry, too many clients already. getent hosts DEPENDENCY_HOST shows whether both tasks resolve the name to the same address. execute-command · PostgreSQL max_connections

Old task, new connectionNew task, new connectionPoints to
ConnectsTimes outThe new deployment's path: compare security groups, subnets and routes
FailsFailsNew connections fail from both, and old tasks were using connections they already held. The dependency may refuse them, or a shared part of the path changed in place: a DNS record now pointing elsewhere, or an edited security group or network ACL rule. Compare the address both resolve with the remote address of an old task's open connections (ss -tn or netstat -tn in the old task), then read the dependency's connection count and listener
ConnectsConnectsNot the network path; compare client settings and code in the release

Without ECS Exec, three other sources help. VPC Flow Logs, if they were enabled, record each interface's flows with an action of ACCEPT or REJECT; a REJECT on the dependency's interface from a new task's address points at its security group or network ACL. If the flow log format includes ecs-task-definition-arn, you can split flows by task definition directly. Reachability Analyzer models the path from a task's network interface to the dependency's address and port and names the blocking component; it reads configuration only, sends no packets and is billed per analysis. On Cloud Run, Connectivity Tests can analyze a revision's path when it uses a Serverless VPC Access connector. Flow log records · Reachability Analyzer · Connectivity Tests sources

fields @timestamp, interfaceId, srcAddr, dstAddr, dstPort, action
| filter dstPort = 5432 and srcAddr in ["OLD_TASK_IP", "NEW_TASK_IP"]
| stats count(*) as flows by srcAddr, action, bin(5m)

A worked fixture: two timeouts that look the same#

This fixture is illustrative. A fictional ECS service checkout moved from checkout:41 to checkout:42 at 14:00 UTC. Both incidents begin with failed connections to the orders database only from checkout:42 tasks.

In the first incident, the deploy also moved the tasks to a new security group:

Checkcheckout:41 taskcheckout:42 task
Deployment security groupssg-0aa1 (checkout-tasks)sg-0bb2 (checkout-tasks-v2)
Database security group allows 5432 fromsg-0aa1not allowed
nc -vz -w 5 to the database, 14:09connectstimes out
Flow logs on the database interfaceACCEPTREJECT

The database is healthy and the deploy changed the path. Either add the inbound rule for the new security group or roll the service back to the previous network configuration; the database needs nothing.

In the second incident, nothing about the network changed:

Checkcheckout:41 taskcheckout:42 task
Deployment security groupssg-0aa1sg-0aa1
nc -vz -w 5 to the database, 14:09connectsconnects
New psql session, 14:09FATAL: sorry, too many clients alreadyFATAL: sorry, too many clients already
Requests servedsucceed over pooled connections opened before 13:50fail; the pool is empty at start

The old tasks' open connections go to the same address the name resolves to now, and no security group or network ACL rule changed after 13:50. The database's connection count sits at its limit from 13:52. The old tasks look healthy only because they opened their connections before the database stopped accepting new ones. The slots filled at 13:52, before the deploy, so the release did not cause the shortage. At the limit every new session is refused, so held connections cannot show whether checkout:42 adds demand; its attempts can. Both task definitions keep the same pool size and reconnect backoff, and the database log shows too many clients refusals at the same rate per task before and after 14:00, and the service ran 4 tasks throughout, replacing them one for one, so total refusals held flat too. A rollback would replace the old tasks with fresh ones that need new connections too, and make the incident larger. Look at the database's connection count and the proxy or listener in front of it. Connection budgets during rolling deployments

The Cloud Run version of the first incident is a revision whose annotations show run.googleapis.com/vpc-access-egress: all-traffic where the old revision had private-ranges-only, deployed to a network without Cloud NAT: the payments API, a public address, times out from checkout-00042 only.

Limits of the comparison#

The comparison assumes old instances are still running. ECS replaces them during the rollout, and a stopped task's network interface is detached and deleted as it deprovisions, so run the tests while the overlap lasts. On Cloud Run you can still send requests to a previous revision through a revision tag. Task lifecycle · Test the previous revision with a tag

Flow logs show accepted and rejected packets, not why a connection hung. They do not record traffic to the Amazon DNS server, and they are not delivered in real time. With Direct VPC egress, VPC Flow Logs do not name the Cloud Run revision, and firewall rules logging is not supported. Reachability Analyzer reads IPv4 configuration only and does not consider the health of load balancer targets. Flow log limitations · Direct VPC egress limitations

A successful nc proves only that a TCP connection opens. TLS, authentication and slow queries can still fail afterwards; a credential the new tasks read at startup is one example. A secret rotated during the rollout

New tasks also add load. If the dependency started timing out because the rollout's overlap opened more connections than it could take, every instance eventually suffers and the timing points at the release. Did our deploy overload a dependency? · The general method

Questions#

Why do only new ECS tasks time out connecting to the database?

New tasks start with the subnets and security groups of the deployment that launched them, and with empty connection pools. Either the deploy changed their network path, or new connections fail for every task and old tasks are still using connections they opened earlier. Open a new connection from an old task to tell the two apart. If that fails too, check whether the database is refusing connections or a shared DNS record or network rule changed in place.

Can I use ECS Exec on a task that is already running?

Only if the task was started with execute command turned on. ECS Exec cannot be turned on for existing tasks; it applies to tasks started after you enable it on the service or task.

Are Cloud Run VPC egress settings per revision?

Yes. They are set in the revision template, so each revision carries its own connector or Direct VPC network, subnet, network tags and egress setting. Compare the annotations of the old and new revisions.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading