Learning center

How much engineering time does each accepted review finding cost?

A reviewer's price shows up on an invoice, but most of its cost is the time authors spend reading, checking and answering its comments. Log that time for a sample of findings and divide it by the findings your team accepted.

Log minutes per finding, not per pull request#

Ask each author in the trial to add two numbers to every finding they judge: minutes spent deciding whether it was right and answering it, and minutes spent changing code. Rough values are fine. A spreadsheet column or a short form beside the record of each finding is enough.

Timestamps on GitHub do not measure this. The gap between a review comment and the author's reply includes lunch, meetings and other pull requests. A comment that took two minutes to dismiss can show a reply three hours later, and a finding that took forty minutes to disprove can be answered within the hour.

Keep the time to fix an accepted finding separate. That work would have been needed whoever found the bug, so it belongs to the defect, not to the reviewer. The reviewer's cost is the time to read, judge and answer.

Work through a fictional trial#

A team logged two weeks of findings from one reviewer:

OutcomeFindingsAverage minutes to judge and answerMinutes
Accepted, real defect6530
Accepted, small improvement8216
Dismissed, incorrect129108
Dismissed, style preference14114
Total40168

The 40 findings cost 168 minutes of author attention. Divided by the 14 accepted findings, that is 12 minutes per accepted finding. Divided by the six real defects, it is 28 minutes per defect.

The table shows where the time went. The twelve incorrect findings took nine minutes each because disproving a plausible bug means reading the code it describes, sometimes running it. They used almost two thirds of the total. The fourteen style comments were quick to skip and cost little time, although they added to the number of comments authors learned to skim past.

Compare the cost with the value of what was found#

Twenty-eight minutes per real defect is cheap if the defects would have reached production, and expensive if a test would have caught them in CI an hour later. For each accepted defect, record where it would otherwise have been found: by the human reviewer, by CI, in staging, or by a customer. Staging and customer discoveries are incidents the reviewer prevented. A defect the human reviewer would have found saves only the time that reviewer would have spent finding it, so count that time if you measured it, and nothing for work that merely moved from one reviewer to the other. A defect CI would have caught saves little.

The guide to findings your linter already reports covers the cheapest overlap to remove. If six of the fourteen style comments duplicate an existing lint rule, turning them off costs nothing in signal.

Find the configuration change with the largest saving#

The incorrect findings are where a configuration change pays. Group them by cause: a project convention the reviewer did not know, a generated file it should not have read, or a claim about behavior outside the repository. One cause that accounts for half the incorrect findings is worth one instruction or exclusion. Rerun the same pull requests at the same commits after the change and log the minutes again. A change that removes five incorrect findings and none of the accepted ones saves about 45 minutes over the same two weeks.

Report the cost with its denominator#

Write the result as minutes, findings and window: "168 minutes of judging and answering for 14 accepted findings out of 40 judged, over two weeks". A per-finding average without its count hides how much the estimate could move, and the sample size guide shows how wide that range is at 40 findings. Include the human reviewer's time on the same pull requests if you have it. A reviewer that costs 168 minutes but lets the human reviewer skip a first pass may still save time overall.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading