Record the next push for every accepted finding#
For each finding the author accepted in the trial, note the commit that was meant to fix it, then read what the reviewer did on that commit. Five outcomes cover most cases:
| Outcome | Example | Cost to the team |
|---|---|---|
| Confirmed the fix | Replies on its thread that the new commit handles the case, or marks it resolved | None; the thread closes itself |
| Found the fix incomplete | Explains which case the new code still misses | Worth reading; this is a second finding |
| Silent | Does not run on the new commit, or runs without mentioning the thread | A human has to check the fix |
| Repeated the finding | Posts the same comment on the fixed code | An extra dismissal for the author |
| Moved on to new findings only | Reviews the new commit but ignores the old thread | A human checks the fix, and the thread stays open |
"Silent" is not a failure for every team. Some reviewers are designed to run once per pull request, and a human reviewer checks the fix anyway. It is a failure if the team expected the reviewer to close the loop and stopped checking fixes itself.
Test the incomplete fix on purpose#
The most useful follow-through result is a fix the reviewer refuses to accept. Plant one in the trial. Take a real accepted finding, write a fix that handles the example in the comment but not the general case, and push it.
In a fictional example, a reviewer reports that an upload handler reads the first extension, so it accepts report.pdf.exe. The author's first fix blocks the extension in the comment:
-if filename.split(".")[1] in {"pdf", "png", "jpg"}:
+if filename.split(".")[1] in {"pdf", "png", "jpg"} and not filename.endswith(".exe"):
save(upload)
That rejects report.pdf.exe, the example in the comment. It still accepts report.pdf.sh and report.pdf.html, because the handler still reads the first extension. A reviewer that follows through should say what the new line still accepts, on the same thread. One that marks the thread resolved because the quoted example now fails has missed the defect.
Watch for repeated findings across pushes#
Repeated findings are the cheapest cost to measure and the most irritating for authors. Count how many findings appear on more than one commit of the same pull request after the author replied that they were wrong or deferred. Read the dismissed finding guide first: a repeated finding after a reasoned dismissal costs the author the same reasoning twice.
GitHub marks an inline review comment as outdated once the lines it was attached to change. An outdated thread is not a resolved one. It only says the code moved, and a reviewer that posts a fresh copy of the same comment on the moved lines has created a second thread for the same disagreement. The guide to counting comments that changed code shows how to read these states from GitHub's records.
Keep human follow-through in the trial#
During the trial, keep whatever human check of fixes you already have. Score the reviewer's follow-through against it: for each accepted finding, did the reviewer reach the same conclusion about the fix as the human? A reviewer that confirms fixes the human also confirmed, and flags the ones the human flagged, is one you can start to rely on for that step. Until then, a confirmed fix from the reviewer is a hint to look, and the human check stays.