Choose a defect the compiler cannot see#
Context is easy to test badly. If the change renames a function, the type checker fails every stale call before any reviewer runs, so a reviewer that reports the broken caller is repeating CI. A useful test changes what a value means while leaving its type alone.
Units are a reliable way to do that. In this fictional TypeScript repository, a pull request changes a retry helper from seconds to milliseconds so it matches the HTTP client's timeout options:
// lib/retry.ts
-/** Delay before the next attempt, in seconds. */
+/** Delay before the next attempt, in milliseconds. */
export function retryDelay(attempt: number): number {
- return Math.min(2 ** attempt, 60);
+ return Math.min(2 ** attempt * 1000, 60_000);
}
The same pull request updates the one caller the author remembered:
// lib/http.ts
-await sleep(retryDelay(attempt) * 1000);
+await sleep(retryDelay(attempt));
A second caller stays outside the diff:
// jobs/webhook-delivery.ts (unchanged)
setTimeout(() => deliver(event), retryDelay(attempt) * 1000);
Before the change, the third retry of a webhook waited 8 seconds. After it, retryDelay(3) returns 8,000 and the job waits 8,000,000 milliseconds, a little over two hours. The cap moves from one minute to 60,000 seconds, almost 17 hours. The pull request updates the retryDelay tests to milliseconds, and they pass, because the helper does exactly what its new comment says.
Score what the reviewer actually located#
Record one of four results for the test pull request.
| Result | What the reviewer wrote | What it shows |
|---|---|---|
| Located | Names jobs/webhook-delivery.ts and the multiplication, and states the new delay | It read the unchanged caller and worked out the consequence |
| Suspected | Says the unit change may affect other callers, without naming one | It noticed the risk. You still have to search for the caller yourself |
| Missed | Comments only on the changed lines, or approves | The review did not reach the defect |
| Wrong | Names a caller that is correct, such as the updated lib/http.ts line | It produced work for the author without finding the problem |
Give credit only for the first row. A generic warning attached to every signature change costs the author a search each time, and a reviewer that suspects everything locates nothing. If the reviewer suggests a fix, check that the fix belongs in the caller. Changing retryDelay back to seconds would undo the purpose of the pull request.
Vary the distance#
One test establishes one distance. Repeat the defect at the distances your codebase actually has, keeping the rest of the change the same:
- The caller in the same package, as above.
- The caller in another package of the same repository, reached through a re-export such as
export { retryDelay } from "./retry". - The caller in configuration rather than code, such as a YAML file that sets
retry_delay: 30and is read as milliseconds after the change. - The caller in another repository that imports a published version of the package.
Expect results to change across these rows. The fourth row tests something most reviewers cannot see from one repository, and a miss there tells you where your team's own cross-repository review still has to happen. Write down which rows matter for your code before you look at the results, so a strong score on an easy row does not stand in for the row you need.
Build the test from your own history#
A fictional pull request shows the mechanism. Your own history shows whether it matters. Search for past fixes whose cause was an unchanged caller: a commit message such as "update remaining callers", or a fix pull request that touched a file its predecessor did not. Recreate the earlier pull request on a branch from its original base commit and run the reviewer on it. That keeps the test realistic and gives you a known answer. The guide to replaying bug-introducing pull requests describes how to find those pairs.
Keep the test pull request out of your main branch, and label it as a trial in its title so nobody approves it by habit. Close it once the reviewer has run.
Read the result with its limits#
A located caller shows that the reviewer reached this file in this repository on this run. It does not establish that every caller will be read on every review. Larger repositories, generated code and files excluded from the reviewer's index can change the result. Run the test again after a configuration change, and keep the commit SHA with the score so a later run compares like with like.
The same test applies to Proofline's code review. Open the test pull request in a repository where it is installed and score the review it posts with the table above. You can run it on your own pull requests with the hosted review offer.