Learning center

How to find the defects a code reviewer missed

A trial that counts only the findings a reviewer posted cannot see what it failed to report. Trace each fix merged during and after the trial back to the pull request that introduced the bug, and check whether that pull request was reviewed in the trial.

Start from fixes, not from reviews#

Misses appear after the review as a bug report, a failed deploy, a rollback or a pull request titled "fix". Collect every fix that lands in the trial repository from the start of the trial until two to four weeks after it ends, because defects introduced on the last day of a trial take time to surface.

Good sources are pull requests and issues with a bug label, reverts, incident records that name a commit, and commit messages that mention "fix", "regression" or "revert". Expect some noise. A commit called "fix typo" is rarely a missed defect.

Trace each fix to the change that introduced the bug#

The standard technique is SZZ, named after its authors. It takes the lines a fix changed and asks version control which earlier commit last touched those lines. A fix-inducing change, in the paper's words, "is a change that later gets undone by a fix". Úliwerski, Zimmermann and Zeller, MSR 2005

With Git, a hand-run version for one fix looks like this:

# Lines the fix removed or changed, in the version before the fix
git show --unified=0 FIX_SHA -- src/billing/invoice.py

# Which commit last touched one of those lines before the fix
git blame -L 88,88 FIX_SHA^ -- src/billing/invoice.py

# Which pull request carried that commit
gh api repos/OWNER/REPO/commits/BLAMED_SHA/pulls --jq '.[].html_url'

Do this by hand for each fix in a trial. A few dozen fixes take an afternoon, and reading each one is what makes the result trustworthy.

Know where tracing goes wrong#

Blame points at the last change to a line, which is not always the change that broke it. A later study listed two common errors: line history alone is sometimes "insufficient" to identify the bug-introducing change, and changes to comments, blank lines and formatting get flagged as if they were fixes. Kim, Zimmermann, Pan and Whitehead, ASE 2006 Three habits remove most of these errors in a small trial:

  • Ignore whitespace and formatting when blaming, with git blame -w, and skip lines in the fix that only reformat.
  • Skip fixes that only add code. A missing check has no earlier line to blame, so read the fix and decide which pull request should have added it.
  • Discard a candidate that was merged after the bug was first reported. It cannot have caused the report.

Decide whether the miss counts against the reviewer#

For each traced pull request, record three facts: whether the reviewer reviewed it in the trial, whether the defect was in the diff the reviewer saw, and whether any finding identified it. Use the bar from the replay guide: the finding names the failure or its consequence at the defect, so "consider handling errors" on the right lines does not count.

Traced pull requestReviewed in trialDefect in reviewed diffFinding identified itResult
Pagination refactorYesYesNoMiss
Config default changeYesNo, it was in an unchanged callerNoMiss outside the diff
Timeout increaseYesYesYes, dismissed by the authorReported, dismissed
Dependency updateNo, opened before the trialYesNot reviewedOutside the trial

The third row matters. A defect the reviewer reported and the team dismissed is not a reviewer miss. It is a dismissal to revisit, and the dismissed finding guide shows how to read it. The second row tests context, which the unchanged caller test measures directly.

Report misses next to finds#

A trial report with "7 real defects found" means more beside "2 defects missed in reviewed pull requests, 1 of them outside the diff". Keep the miss list by name, with the fix and the traced pull request linked, and use it at the end of the trial. If the misses cluster in one kind of change, that is where human review must stay thorough whatever the reviewer's results elsewhere. The decision guide uses this list.

Escaped defects are found only when someone notices them. A short trial will undercount misses, and the count says nothing about defects nobody has found yet.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading