Learning center

How to plant known defects in a code review trial

Planted defects give a review trial an answer key. Use them to learn which kinds of mistake a reviewer reports, and keep the score away from any claim about how many real bugs it will find.

Plant the kinds of mistake your team makes#

Mutation testing tools already define small, mechanical code changes. PIT's mutator list includes a conditionals boundary mutator that turns < into <=, a negate conditionals mutator that turns == into !=, and a void method call mutator that deletes a call to a method that returns nothing. These are a useful vocabulary, but a reviewer trial needs defects that matter to your team, not just the ones a tool can generate.

Start from your last few incidents and post-merge fixes. Each one names a kind of mistake: a boundary in pagination, an authorization check placed after a write, a resource that is opened but not closed on an error path, or two arguments of the same type passed in the wrong order. Plant one example of each kind in an otherwise ordinary change.

Here is a fictional pagination change with one planted boundary defect. The correct comparison is end < len(items); the trial branch carries <=:

 def page(items, number, size):
     start = (number - 1) * size
-    return items[start:start + size]
+    end = start + size
+    has_next = end <= len(items)
+    return items[start:end], has_next

With 20 items and a page size of 10, page 2 now reports a next page, and the client requests a third page that is empty. A reviewer that reads the boundary should say that has_next is true when the last page is exactly full.

Keep the answer key sealed#

Write the planted defects in a file that the reviewer cannot read: outside the repository, or in a private note held by the person who will score the trial. For each one, record:

FieldExample
KindBoundary condition
Locationpagination.py, the has_next line
TriggerNumber of items is an exact multiple of the page size
ConsequenceThe client requests an empty final page
Found ifThe finding names the comparison and the full-last-page case

The "found if" column is the important one. Decide before the review what counts as finding the defect. Otherwise a vague comment on the right line can be scored generously after you know the answer.

Score found, located and noise separately#

Run the reviewer on the trial branch and record three results for each planted defect:

  • Found: the finding states the trigger and consequence in the answer key.
  • Located only: the reviewer commented on the line for some other reason, or asked a question without naming the failure.
  • Missed: nothing on the defect.

Then count the reviewer's other findings on the same change. A trial branch is still code, and some of those findings may be real. Judge them the same way you judge findings on ordinary pull requests. A reviewer that finds every planted defect and also posts fifteen incorrect comments on the surrounding code will cost your authors the same fifteen comments on real work.

Know what the score cannot tell you#

A planted defect is unusually easy to find in some ways and unusually hard in others. It is small and local, so a reviewer that reads only the changed lines can see it. It also sits in code written for the test, which may look different from the code around it. And it has no history: the reviewer cannot learn from an issue, a failing test or a previous fix that a real defect might come with.

So report the result as a set of observations, for example "reported 4 of 6 planted defects; missed both authorization-order defects". Avoid turning it into a percentage of real bugs. To estimate how a reviewer does on defects that actually happened, replay the pull requests that introduced past bugs, and use planted defects for kinds of mistake that have no replayable pull request in your history, or none yet.

Keep trial branches out of the main branch. Name them clearly, close them after the review, and never plant a defect in a pull request that someone might merge.

Use the misses#

A missed kind is the most useful result in the trial. If the reviewer missed both authorization-order defects, write a small test pull request for that kind alone and run it again after any configuration change. If a repository rule or reviewer instruction can describe the check, add it and rerun the same branch at the same commit. The guide to custom review rules shows how to express such a check. The same branch then becomes a regression test for your review setup.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading