Start from fixes, not from reviews#
Misses appear after the review as a bug report, a failed deploy, a rollback or a pull request titled "fix". Collect every fix that lands in the trial repository from the start of the trial until two to four weeks after it ends, because defects introduced on the last day of a trial take time to surface.
Good sources are pull requests and issues with a bug label, reverts, incident records that name a commit, and commit messages that mention "fix", "regression" or "revert". Expect some noise. A commit called "fix typo" is rarely a missed defect.
Trace each fix to the change that introduced the bug#
The standard technique is SZZ, named after its authors. It takes the lines a fix changed and asks version control which earlier commit last touched those lines. A fix-inducing change, in the paper's words, "is a change that later gets undone by a fix". Ćliwerski, Zimmermann and Zeller, MSR 2005
With Git, a hand-run version for one fix looks like this:
# Lines the fix removed or changed, in the version before the fix
git show --unified=0 FIX_SHA -- src/billing/invoice.py
# Which commit last touched one of those lines before the fix
git blame -L 88,88 FIX_SHA^ -- src/billing/invoice.py
# Which pull request carried that commit
gh api repos/OWNER/REPO/commits/BLAMED_SHA/pulls --jq '.[].html_url'
Do this by hand for each fix in a trial. A few dozen fixes take an afternoon, and reading each one is what makes the result trustworthy.
Know where tracing goes wrong#
Blame points at the last change to a line, which is not always the change that broke it. A later study listed two common errors: line history alone is sometimes "insufficient" to identify the bug-introducing change, and changes to comments, blank lines and formatting get flagged as if they were fixes. Kim, Zimmermann, Pan and Whitehead, ASE 2006 Three habits remove most of these errors in a small trial:
- Ignore whitespace and formatting when blaming, with
git blame -w, and skip lines in the fix that only reformat. - Skip fixes that only add code. A missing check has no earlier line to blame, so read the fix and decide which pull request should have added it.
- Discard a candidate that was merged after the bug was first reported. It cannot have caused the report.
Decide whether the miss counts against the reviewer#
For each traced pull request, record three facts: whether the reviewer reviewed it in the trial, whether the defect was in the diff the reviewer saw, and whether any finding identified it. Use the bar from the replay guide: the finding names the failure or its consequence at the defect, so "consider handling errors" on the right lines does not count.
| Traced pull request | Reviewed in trial | Defect in reviewed diff | Finding identified it | Result |
|---|---|---|---|---|
| Pagination refactor | Yes | Yes | No | Miss |
| Config default change | Yes | No, it was in an unchanged caller | No | Miss outside the diff |
| Timeout increase | Yes | Yes | Yes, dismissed by the author | Reported, dismissed |
| Dependency update | No, opened before the trial | Yes | Not reviewed | Outside the trial |
The third row matters. A defect the reviewer reported and the team dismissed is not a reviewer miss. It is a dismissal to revisit, and the dismissed finding guide shows how to read it. The second row tests context, which the unchanged caller test measures directly.
Report misses next to finds#
A trial report with "7 real defects found" means more beside "2 defects missed in reviewed pull requests, 1 of them outside the diff". Keep the miss list by name, with the fix and the traced pull request linked, and use it at the end of the trial. If the misses cluster in one kind of change, that is where human review must stay thorough whatever the reviewer's results elsewhere. The decision guide uses this list.
Escaped defects are found only when someone notices them. A short trial will undercount misses, and the count says nothing about defects nobody has found yet.