Calculate the interval for your own counts#
After a two-week trial, a team judged 20 findings from reviewer A and accepted 12 of them, a 60% share. Reviewer B, run on the same pull requests, had 9 of 20 accepted, 45%. The difference looks decisive in a status update.
The Wilson score interval gives a range of shares that are consistent with those counts. It behaves sensibly for small samples and for shares near 0 or 100%, which is where review trials usually sit. This short script computes the 95% interval:
from math import sqrt
def wilson(accepted, judged, z=1.96):
p = accepted / judged
d = 1 + z * z / judged
center = (p + z * z / (2 * judged)) / d
half = z / d * sqrt(p * (1 - p) / judged + z * z / (4 * judged * judged))
return max(0, center - half), min(1, center + half)
print(wilson(12, 20)) # (0.387, 0.781)
print(wilson(9, 20)) # (0.258, 0.658)
Reviewer A's share could plausibly be anywhere from 39% to 78%. Reviewer B's could be anywhere from 26% to 66%. The ranges overlap across most of their width, so the trial has not established that A's findings are more often accepted. It has established that neither reviewer is accepted nearly always or almost never.
See how the width shrinks#
The interval narrows with the square root of the number of judged findings, so halving its width takes about four times as many findings.
| Accepted of judged | Share | 95% interval |
|---|---|---|
| 6 of 10 | 60% | 31% to 83% |
| 12 of 20 | 60% | 39% to 78% |
| 30 of 50 | 60% | 46% to 72% |
| 60 of 100 | 60% | 50% to 69% |
A team that judges about five findings a day needs roughly four weeks to reach 100. Before extending a trial that long, ask whether the decision needs that precision. "Keep the reviewer if at least a third of its findings are accepted" is answered at 30 of 50, because the whole interval sits above 33%. "Pick the better of two similar reviewers" may never be answered by acceptance share alone.
Read zero carefully#
A trial with no incorrect findings in 30 is encouraging, but the interval for 0 of 30 runs from 0% to 11%. One finding in nine could still be wrong when the reviewer sees more code. The same applies to misses: if a reviewer reported all 8 of the defects you planted, the share it would miss on similar defects could still be as high as 32%.
Check that findings are independent#
The calculation treats each finding as a separate draw. Review findings often are not. If one misunderstanding of a project convention produced eight dismissed comments on a single pull request, those eight are closer to one observation than to eight.
Two diagnostics show clustering. Recompute the share with the pull request as the unit, counting a pull request as a success if at least one of its findings was accepted. Then remove the pull request with the most findings and recompute. Either change moving the share is a sign of clustering, but neither moving does not show independence: twenty pull requests whose ten findings each were all accepted or all dismissed together pass both checks, yet hold twenty observations, not two hundred. When findings on one pull request share causes, report the pull request share from the first diagnostic, with its Wilson interval over the number of pull requests, and say why.
Fix when you will read the result#
Decide three things before the trial: the decision, the threshold that would change it and the number of judged findings or the date at which you will read the result. Read it once, at that point. Stopping as soon as the interval clears the threshold gives the trial many chances to cross it by luck, and the 95% figure no longer holds for a result chosen that way. "13 of 22 findings accepted, 39% to 77%, above the 33% threshold" is a result the team can act on. Report the interval as it stands at the planned point, even when it still spans the threshold. A rounded "about 59%" hides the part that matters.
Shares do not capture the finding that prevents a serious incident. Keep those findings listed by name beside the numbers, and judge them on their own consequences.