Learning center

Keep, narrow or remove: deciding at the end of a code review trial

A review trial ends with one of three decisions: keep the reviewer as it is, narrow it to the changes where it earned attention, or remove it. Make the decision from the per-kind record of the trial, write it down with its evidence, and set the date you will check it again.

Lay out the evidence by kind of change#

A single acceptance share for the whole trial hides the decision. Start from the table the change mix guide produces, and add the costs and misses you recorded:

Kind of changePull requestsReal defects foundIncorrect findingsMinutes judging and answeringKnown misses
Data or access942410
Feature or behavior change1837881
Refactor605360
Dependency or generated2203120
Tests or documentation only1106190

These fictional numbers come from a four-week trial on one repository. The reviewer found seven real defects in 66 pull requests and cost 196 minutes of judging and answering. Every real defect came from the first two rows.

Choose among the three decisions#

Keep the reviewer unchanged when its cost is spread evenly and its findings are useful across most kinds of change. That is not this table.

Narrow the reviewer when the useful findings concentrate in some kinds of change and the cost sits elsewhere. Here, the last three rows produced no real defects and 67 minutes of judging and answering. Most review apps let you restrict reviews by path or file pattern, skip draft or bot-authored pull requests, or run only when someone asks. A narrowed reviewer that skips pull requests opened by the dependency bot, and those that change only tests or documentation, would have kept every real defect in this trial and removed 31 of the 196 minutes. Refactors are harder to exclude, because they touch the same files as features. Their five incorrect findings are a case for an instruction to the reviewer rather than a path filter, and the attention cost guide shows how to group incorrect findings by cause.

Remove the reviewer when no kind of change shows real defects that your existing review and tests would have missed, or when the incorrect findings cost more than the defects were worth. Record what you would need to see to try again, such as a new model version or support for a language you use.

Whichever you choose, check it against the misses. The one known miss on a feature change belongs in the record: it shows a kind of defect that the reviewer did not report in the trial and that your human review still has to cover. The guide to tracing escaped defects describes how to find such misses.

Write the decision with its evidence#

A short decision record lets someone revisit the choice without repeating the trial. Include:

  • The repository, the trial window and the selection rule.
  • The table above, with commit SHAs or pull request links for each real defect.
  • The configuration the reviewer ran with, including model or version where the product exposes it.
  • The decision, the new configuration if narrowed, and the reason in one or two sentences.
  • The limits: kinds of change with too few examples, misses you could not trace, and anything you could not measure.
  • A date to check the decision again.

Check the decision on the next month's pull requests#

A narrowed reviewer is a new configuration. On the review date, count the same columns for the paths it still covers, and look at real defects found by other means in the paths it no longer covers. If a defect in src/billing/** reached production that the trial reviewer would have reported, widen the scope again. If the narrowed reviewer still finds real defects at a cost the team accepts, keep it and set the next review date further out.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading