Pin both reviews to one commit#
When two review apps run on a live pull request, the author usually reads the first review and pushes a fix. The second reviewer then sees different code. If it does not repeat the finding, it looks as though it missed the defect. If it reviews the original commit after the fix, its finding is stale before anyone reads it.
Avoid this by reviewing a fixed commit with both tools before anyone responds:
- Choose a pull request from the trial sample and note its head commit SHA.
- Create a branch at that SHA in a fork or a separate trial repository where both reviewers are installed:
git push trial <sha>:refs/heads/trial/pr-1842. - Push the original base commit to its own branch,
git push trial <base-sha>:refs/heads/trial/base-1842, and open a pull request from the first branch against it. - Let both reviewers finish before anyone reads either review.
If the reviewers cannot run outside the main repository, open a draft pull request at the pinned SHA, mark it as a trial in its title, and push nothing to it. Record each reviewer's configuration and the time its review finished.
Strip the reviewer's name before judging#
Copy each finding into a sheet with a random identifier, the file and line, and the finding text with any product name, badge or formatting removed. Shuffle the rows. A judge who does not know which reviewer wrote a finding cannot favor the expected winner, and the judge reads each finding on its own terms.
Judge every row with the same categories you use for the rest of the trial: a real defect, a valid improvement, incorrect, or unresolved. Record the trigger and consequence for each real defect, because you need them for the next step. The guide to reconstructing an accepted finding shows what that record looks like.
Match findings by defect, not by line#
Two reviewers often describe the same defect on different lines. One comments on the caller that passes a stale value, the other on the function that receives it. Count them as the same finding when the trigger and the consequence match, whichever line they sit on. Count them as different when the same line has two different consequences.
After matching, every real defect falls into one of three groups:
| Real defects | Count | What it tells you |
|---|---|---|
| Reported by both | 5 | Overlap; either reviewer would have caught these |
| Only reviewer A | 3 | What A adds if you already run B |
| Only reviewer B | 1 | What B adds if you already run A |
The two "only" rows answer the question most teams actually have, which is whether a second reviewer earns its place beside the first, or whether one can replace the other. In this fictional example, A found eight of the nine real defects and B found six. Replacing B with A loses one, while adding B to A gains one.
Count the cost on the same rows#
Each reviewer's incorrect and unresolved findings cost the author time, whatever it found. Add those counts to the same table, along with the time each finding took to judge if you logged it. The attention cost guide shows the calculation. A reviewer that adds one defect at the cost of fifteen incorrect findings may still be worth running only on certain paths.
Keep the comparison honest about its size#
Nine real defects across twenty pull requests is a small base. The sample size guide shows how wide the uncertainty is at that scale. Report the counts, the pull requests and the commit SHAs together so another engineer can rerun the comparison after either reviewer changes its configuration or model. Unblinding is the last step: reveal which reviewer wrote each finding only after every row is judged and matched.