Learning center

Which review delay should a code review trial measure?

An automated reviewer that posts within three minutes can still slow a team down, and one that takes twenty can still be fast enough. Measure the delay between a push and the feedback the author acts on, and compare it with how long the pull request waits for people anyway.

Know the delay you already have#

Human review sets the baseline. Google's study of about nine million code changes reported that authors waited a median of under an hour for initial feedback on small changes and about five hours on very large ones, and that the median time for the whole review was under four hours. Sadowski et al., ICSE-SEIP 2018 Meta measured "time in review", which counts only the time a diff waits for reviewer action, and found that developers' sense that review was slow tracked the slowest quarter of their diffs, those waiting more than about 24 hours. Shan et al., ESEC/FSE 2022

Your team's numbers will differ. Measure them for the trial repository before you measure the reviewer, because a reviewer's delay matters only relative to the wait it shares.

Measure four timestamps per pull request#

For every pull request in the trial, collect:

  1. Pushed: when the commit the reviewer reviewed was pushed. Use the push or pull request event time, not the commit's author date, which can be days earlier on a rebased branch.
  2. Reviewer finished: the submitted_at of the reviewer's review, from GitHub's list reviews endpoint, or the time of its last comment.
  3. First human action: the first review or comment from a person other than the author after that push. Each reviewed push gets its own, so a comment on an earlier revision does not count against the review of a later one.
  4. Merged: the merge time.

The reviewer's own delay is the gap from pushed to reviewer finished. The delay that matters is whether that gap ends before the first human action. A finding that arrives after the human reviewer has already approved either goes unread or reopens a review the team considered finished.

Count late findings, not average minutes#

An average response time hides the cases that cause trouble. Count instead, for each pull request:

Order of eventsWhat happenedCount
Reviewer finished before first human actionThe human reviewer could read the findings31
Reviewer finished after first human action, before mergeThe findings arrived mid-review6
Reviewer finished after mergeThe findings were too late to change the pull request2
Reviewer never finishedA timeout, a skip or an error1

In this fictional trial, three pull requests out of forty were not served by the review at all. Read those three. A review that finishes after merge on large pull requests only is a different problem from one that times out on generated files.

Check the delay after a fix push too#

The second review often matters more than the first. An author who pushes a fix and waits for the reviewer to confirm it is blocked in a way the first review rarely blocks anyone. Record the same timestamps for every push that follows a finding, with the first human action taken after that push, and read them with the follow-through guide. If the reviewer takes twenty minutes per push and the author pushes four times, the reviewer has added over an hour to a pull request that the human reviewer might have approved on the first push.

Decide what delay you can accept#

Set the limit from your own baseline. If your median first human action is two hours after a push, a reviewer that finishes within thirty minutes on almost every pull request rarely delays anyone. If your team reviews within fifteen minutes during working hours, the same reviewer arrives late on many pull requests, and its findings will need a different place in the process, such as a required check before review is requested.

Latency is one column in the decision. A reviewer that is always early and rarely right still costs attention, and the attention cost guide measures that side.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading