Count findings, not comments#
A reviewer may post one comment that raises three problems, or three comments about one. A thread may also contain the reviewer's follow-up replies. Counting comments rewards a reviewer that splits its output and penalizes one that groups related problems.
Count a finding as one claimed defect or improvement, with its trigger and consequence. Split a comment that raises unrelated problems. Merge comments that describe the same problem at several call sites, and note how many sites it named. A reviewer that reports the same broken helper once and lists six callers has given the author one finding to act on.
Decide what "useful" means before you count#
Researchers at Microsoft classified 1.5 million review comments across five projects, using a model trained on a smaller set of comments that their authors rated in interviews or the researchers classified by hand. They found that a comment which triggered a change within one line of the commented lines was very likely to be useful, with 88% precision, and that comments closed as "Won't fix" were mostly not useful. Across the projects, about 65% of comments were classified as useful. Bosu, Greiler and Bird, MSR 2015
That gives a trial a measurable proxy: a finding is accepted when the author changes the lines it concerns. The guide to counting comments that changed code shows how to find those changes. The proxy has known gaps. An author can accept a finding by changing a different file, or by opening a follow-up issue. An author can also change the commented lines for an unrelated reason. Use the proxy to sort findings quickly, then have a person confirm each one.
Separate two kinds of usefulness. A study of more than 1,400 changes in two open-source projects found that about 75% of the changes made in review improved maintainability and about 25% fixed functional problems. Beller, Bacchelli, Zaidman and Juergens, MSR 2014 An industrial study reached a similar 75% for defects that did not affect visible functionality. Mäntylä and Lassenius, IEEE TSE 2009 Both kinds can be worth having. They answer different questions, so count them separately.
Pick the denominator for the question#
Each denominator answers a different question:
| Divide by | Question it answers | Distortion to watch |
|---|---|---|
| Findings | When this reviewer says something, how often is it worth acting on? | Rewards a reviewer that says little |
| Pull requests reviewed | How often does a pull request get at least one useful finding? | Ignores the cost of the other findings on the same pull request |
| Pull requests in the trial, reviewed or not | How much of the team's work does the reviewer help with? | Penalizes a reviewer narrowed to certain paths, which may be intended |
| Known defects | Of the defects we know about, how many did it report? | Depends on how you found the defects; see the replay guide |
A reviewer that comments on two pull requests out of fifty, and is right both times, scores 100% by findings and 4% by pull requests. Neither number is wrong. The first says its comments deserve attention when they appear. The second says it helped with very little of the work.
Keep unresolved findings in the denominator#
Some findings will be neither accepted nor dismissed by the end of the trial: the author never replied, or the team could not decide. Leaving them out raises every share. Report them as a third count: "14 accepted, 19 dismissed, 7 unresolved, of 40 findings". If you must report one share, give the range: between 35% and 53% accepted, depending on how the seven unresolved findings turn out.
Write the definition beside the number#
Every share in a trial report should carry its numerator, its denominator and its definition: "14 of 40 findings accepted, where accepted means the author changed the commented lines or opened a linked issue, as confirmed by the author". The sample size guide shows how much a share at that count can move.