Two symptoms that could be the provider#
- An ECS service in
us-east-1is halfway through a rollout fromcheckout:41tocheckout:42. Tasks from both task definitions logconnection timed outto an Amazon RDS instance, and the public AWS Health Dashboard shows no open issue. - A Cloud Run service in
us-central1moved traffic tocheckout-00042at 14:03 UTC. Request logs from both revisions show timeouts to Cloud SQL, the old revision's first ones at 13:52, and status.cloud.google.com lists no incident.
Both point away from the release: the old revision fails too, and in the second case it failed first. The open question is whether the dependency degraded on its own or the provider's platform under it did. That decides where the next half hour goes. A rollback helps with neither, unless the release also added load the struggling dependency now has to carry, which a comparison of this service's calls, connections and work per call before and after the rollout rules in or out. Split errors by revision first
Why a clean public status page proves little#
Each provider publishes two kinds of event, and the public page carries only one of them.
On AWS, public events describe a service issue in a Region whether or not you use it, and account-specific events describe issues that affect your account's resources, with the affected resources named. The AWS Health Dashboard – Service health page at health.aws.amazon.com/health/status shows public events only. Your account's AWS Health Dashboard, the AWS Health API and EventBridge also carry account-specific events. Public and account-specific events
On Google Cloud, the public Cloud Service Health dashboard communicates broad severe incidents, those affecting many projects or several regions. Incidents with a narrower scope, such as a single location, zone or subset of projects, reach customers through Personalized Service Health only. Google Cloud incident communication
A zonal problem under your database, or a networking fault in one Region that affects some customers, is the kind of event that may appear only in the account-specific view.
AWS: read AWS Health for the account and Region#
Every account can read its AWS Health Dashboard in the console, including an event log of the past 90 days. The AWS Health API requires a Business-level or Enterprise-level AWS Support plan; other accounts get SubscriptionRequiredException. The API is served from a global endpoint whose active Region is us-east-1 by default, so pass --region us-east-1 whatever Region your service runs in, and filter on the service's Region instead. AWS Health Dashboard · AWS Health API endpoints and plans
aws health describe-events \
--profile PROFILE --region us-east-1 \
--filter 'eventTypeCategories=issue,regions=SERVICE_REGION,services=RDS,ECS,EC2' \
--query 'events[].{arn:arn,service:service,code:eventTypeCode,scope:eventScopeCode,
az:availabilityZone,status:statusCode,start:startTime,end:endTime}' \
--output table --no-cli-pager
Include every service on the failing path, such as the database and EC2 for networking; aws health describe-event-types lists the service codes Health uses. describe-event-types Each event carries eventScopeCode (PUBLIC or ACCOUNT_SPECIFIC), statusCode (open, closed or upcoming), an optional availabilityZone, and startTime and endTime. For an account-specific event, list the resources it affects and look for your database instance or load balancer. Event fields · EventFilter · describe-affected-entities
aws health describe-affected-entities \
--profile PROFILE --region us-east-1 \
--filter eventArns=EVENT_ARN \
--query 'entities[].{entity:entityValue,status:statusCode,updated:lastUpdatedTime}' \
--output table --no-cli-pager
If the service runs in a member account of an AWS Organization, the management account or a delegated administrator can read the same events across accounts with describe-events-for-organization, after organizational view is enabled. Organizational events are kept for 90 days. Organizational view
Google Cloud: read Personalized Service Health for the project#
The Service Health dashboard in the Google Cloud console shows incidents relevant to the project, with each affected product and location. The same events come from the Service Health API, which must be enabled on the project. Enabling it after an incident still returns project-level events from the past 90 days. Personalized Service Health · Concepts and API enablement
gcloud service-health events list \
--project PROJECT --location global \
--filter 'category=INCIDENT' \
--format 'yaml(title,relevance,state,detailedState,startTime,endTime,eventImpacts)'
--location global is required for incident events. Each event's eventImpacts lists product.productName and location.locationName pairs, each with its own state and times. relevance says how the incident relates to the project. gcloud service-health events list · Event fields · EventImpact
IMPACTED: confirmed to affect the project.RELATED: affects a product the project uses, in a location the project uses.PARTIALLY_RELATED: affects a product the project uses, possibly in a location it does not use.NOT_IMPACTEDandUNKNOWN.
Service Health also logs events in Cloud Logging, with an entry when an event is created and when it is updated. Service Health logs · gcloud logging read
gcloud logging read \
'resource.type="servicehealth.googleapis.com/Event"
AND jsonPayload.impactedProducts:"Google Cloud SQL"
AND jsonPayload.impactedLocations:"REGION"
AND timestamp>="START_UTC"' \
--project PROJECT --format json
Read the result#
| What you find | What it means | Next move |
|---|---|---|
An account-specific AWS event, or a Google event with relevance IMPACTED or RELATED, for the dependency's service and Region, starting at or before your first errors | Strong evidence of a provider cause | Follow the event's guidance and consider failover; a rollback will not help unless the release raised this service's load on the dependency |
An event for the right service in another Region, or PARTIALLY_RELATED | Weak; possibly unrelated | Check whether any call on the path crosses that Region |
| An event whose start is well after your first errors | The same fault confirmed late, or a coincidence | Rely on your own timeline and the split by revision |
| No event | Not established either way | Keep checking the dependency's own metrics and the release |
Match the event's start against your own first errors, the earliest minute in the old revision's error rate. A provider event that starts after the rollout while your old revision was failing before it is still consistent with a provider fault; one that starts after the rollout while only the new revision fails is not, unless the fault refuses only new connections: then old instances survive on pooled ones, so open a fresh connection from an old instance before ruling the provider out. Timeouts only from new tasks
A worked fixture: one Region, two outcomes#
The fixture is illustrative. A fictional Cloud Run service checkout in us-central1 calls Cloud SQL in the same region. The rollout moved 10% of traffic to checkout-00042 at 14:03 UTC, and the old revision checkout-00041 logged its first Cloud SQL timeouts at 13:52.
| Case A: regional event | Case B: no event | |
|---|---|---|
| Split by revision | Both revisions fail at about 3% | Both revisions fail at about 3% |
| Events returned for the project | category: INCIDENT, detailedState: CONFIRMED, relevance: RELATED | none since 13:00 |
eventImpacts | Google Cloud SQL in us-central1, state: AFFECTED, start 13:47 | |
| Cloud SQL's own metrics | Connection and query latency up from 13:48 | database/postgresql/num_backends at its limit from 13:50 |
In case A the event names the dependency's product and region and starts five minutes before the first timeouts, and the service's Cloud SQL queries per request, its connections, from num_backends_by_application, and its query insights time per query are the same before and after 14:03, and its logs show one connection attempt per new connection, so it is not retrying harder. The release neither caused the fault nor added to it, and the team's time goes to failover and to the event's updates.
In case B nothing from the provider. The database's connections are at their limit, which more callers or longer sessions can both explain, and a slowdown the provider has not reported would also hold sessions open. The absence of an event did not prove the platform healthy. Next, compare connections per caller from num_backends_by_application with query latency: one caller's connections climbing while its calls and work per call rise points at that caller; sessions lasting longer while every caller's issued work stays flat leaves the database or the platform under it. Did our deploy overload a dependency?
The AWS version of case A is a describe-events row with service RDS, the service's Region, eventScopeCode ACCOUNT_SPECIFIC and statusCode open, whose affected entities include your database instance's identifier.
Where this check stops#
Events can lag the impact. Google marks incidents still under investigation as EMERGING, and an event can be confirmed, merged into another or closed after review as a false positive. Read an event's startTime, not the time you first saw it. Event states
An event depends on the provider detecting and confirming the impact. A partial slowdown that only some customers see can fail your requests before an event appears, or without one. Treat the provider's silence as missing evidence.
A provider event and a bad release can coincide. Even with a matching event, check that the old revision fails at the same rate as the new one before you ship the new revision again. Equal rates do not show that the release added nothing: a release that retries harder or queries more makes a degraded dependency worse for every revision. Compare this service's calls and connections per request before and after the rollout. Did our deploy overload a dependency?
Access limits the view. Without a Business-level or Enterprise-level support plan the AWS Health API refuses the call, and the console remains the only account-specific source. Personalized Service Health depends on Google Cloud's own identity services, so in a severe disruption you may be unable to sign in to read it, and the public page becomes the fallback. Fallback guidance
Questions#
Does the AWS Health API work on every AWS account?
No. It requires a Business-level or Enterprise-level AWS Support plan, and other accounts receive SubscriptionRequiredException. Every account can read its own events in the AWS Health Dashboard in the console, including a 90-day event log.
Why does the Google Cloud status page show nothing when Personalized Service Health shows an incident?
Google uses the public Cloud Service Health page for broad severe incidents. Incidents limited to one location, a zone or a subset of projects are communicated through Personalized Service Health only.
If the provider reports an incident, should we still roll back?
A rollback only helps if the release caused the errors. If the provider's event matches the dependency's service and Region, both revisions fail at the same rate, and the release did not raise this service's calls, connections or work per query (per-statement time and rows), a rollback restarts tasks without changing the cause; spend the time on failover and the provider's guidance instead. If the release did raise that load, a rollback can relieve the dependency while the provider recovers.