Find where a deployment regression was reported

When a deploy’s verification records a regression, open the deploy to trace the report to the people and channels that received it.

Check the report and its delivery destinations#

When a deploy regresses, Proofline reports it once. A failed deployment, or a running revision that serves a different commit, reports from any check. A signal regression usually reports from the 20-minute check on, because a new revision's cold start can move the first minutes past tolerance. A landing check reports a signal regression only when higher values are worse and the reading is at least ten times the value at which the signal regresses. Any other landing check whose only regression is a signal records Regressed and reports nothing until a later check finds a regression too.

The report is a sentence that names the environment, the check it reports from and what regressed, such as "Deployment to Production regressed 20 minutes after landing: 5xx responses moved past its tolerance." When the deployment failed after it landed, or the service runs a different commit, the sentence says so after any regressed signals. A failed ECS rollout reads "the ECS rollout of api did not complete", naming each service whose rollout failed. When a landing check that did not report had already found those signals, the sentence says so in one of three ways. It says they moved past tolerance "from landing on" when every check since landing found them too. It ends ", as it also did at landing", or ", as they also did at landing" for several signals, when a check in between did not find them or was not recorded, so the report does not claim the regression lasted the whole time. It says "at landing" when only the landing check found them. When the regressed signal is a Cloud Run service's request count and the check read the service's request log, a second sentence names the failing routes. A report on any other signal, such as a CloudWatch one, names no routes. The report names neither the commit nor a pull request. It goes to three places:

  • Reviews → Notifications, for members who can write to the repository. The row's title is the report, with the repository below it, and selecting it opens the deploy's page. Reports from one environment that describe the same regression share one row, titled with how many deploys it holds, such as "3 deploys to Production regressed 20 minutes after landing: …", without the failing routes. Open it to list each deploy's short commit, which opens that deploy's page. When email delivery is available, members who keep Outcome emails on under Delivery preferences also get an email titled A deploy regressed after it landed. The email states the report, names the repository below it and links the deploy's page with View deployment.
  • GitHub: a comment with the report and a View deployment link, on the pull request that merged the deployed commit, else on the pull request of the review that covered it. Without either, a failed Proofline deploy verification check titled Deploy regressed after it landed on the deployed commit. In shadow mode, nothing is posted to GitHub.
  • The organization's Slack delivery channel: the report, then the repository and a View deployment link. Set up Slack delivery chooses it; without one, nothing is posted to Slack.

A commit is reported once in each environment on GitHub and in Slack, and by email to each member at most once. When several deploys of the same commit regress in one environment, such as separate workflow runs or different deployment tasks, a later report reuses the first one's GitHub comment or check. In Slack, it replies in the first report's thread when that message is in the channel Slack delivery posts to now and Slack still holds it, and the reply names the deploy's GitHub deployment task. Otherwise it posts a new message in that channel.

Each channel is delivered independently, so a Slack failure does not delay the notification or the GitHub comment.

Once a report goes out, the deploy's page lists where it went under Regression report, below the checks. Each of GitHub, Slack and Notifications that recorded a receipt has a row with when it was recorded. A GitHub row links the pull request comment or check run. A channel that did not post the report reads Not delivered with the reason, such as No Slack channel is connected. A channel with no receipt has no row, and a deploy that reported nothing shows no Regression report.

Search help

Tab to a result. Enter to open. Escape to close.