Find a pair: the fix and the pull request it repairs#
Start from fix pull requests that name their cause. Search your repository for descriptions containing "regression from", "introduced in" or "broke in", and for reverts. Each match gives you an answer key that nobody had to invent: the fix shows what was wrong, and the earlier pull request shows the code a reviewer would have seen.
When a fix names no cause, trace it. Blame the lines the fix changed at the commit before the fix, and find the pull request that last touched them. The escaped defects guide describes that tracing and where it goes wrong.
Read a public pair#
Vite's dev server added a check for port conflicts on wildcard hosts in January 2026. The pull request added this code to packages/vite/src/node/http.ts:
// Check if a port is available on wildcard addresses (0.0.0.0, ::)
async function isPortAvailable(port: number): Promise<boolean> {
for (const host of wildcardHosts) {
// Gracefully handle errors (e.g., IPv6 disabled on the system)
const available = await tryListen(port, host).catch(() => true)
if (!available) return false
}
return true
}
function tryListen(port: number, host: string): Promise<boolean> {
return new Promise((resolve) => {
const server = net.createServer()
server.once('error', () => {
// Ensure server is closed even on error to prevent resource leaks
server.close(() => resolve(false))
})
// ...
})
}
The comment on the .catch says that errors such as a disabled IPv6 stack should leave the port available. But tryListen never rejects. Every socket error resolves false, so the .catch never runs, and any error marks the port as taken. Original pull request diff
The pull request was reviewed. A maintainer requested changes about a different port-binding case, then approved the revised version, and it merged on 19 January 2026. Review requesting changes ยท Approval
On 15 February, a user reported "No available ports found between 5173 and 65535" on WSL, with the previous beta working. User report The fix, merged the next day, made only an address-in-use error count as a conflict. Its description says other errors, such as EACCES from Hyper-V port reservations, had caused every port to be reported unavailable. Fix pull request
- server.once('error', () => {
- // Ensure server is closed even on error to prevent resource leaks
- server.close(() => resolve(false))
+ server.once('error', (e: NodeJS.ErrnoException) => {
+ server.close(() => resolve(e.code !== 'EADDRINUSE'))
})
This pair is a good test case because the defect is visible in the diff alone. A reviewer does not need the user's machine to see that the .catch cannot run.
Write the answer key before the replay#
From the fix, write what a reviewer must say to get credit, as you would for a planted defect:
| Field | For this pair |
|---|---|
| Location | isPortAvailable and tryListen in http.ts |
| Trigger | Listening on a wildcard host fails with an error other than address in use |
| Consequence | The port is reported as taken; if every port fails this way, the server finds none |
| Found if | The finding says that tryListen never rejects, so the .catch fallback is dead, or that every error is treated as a conflict |
A finding that says "consider handling errors" on the tryListen lines is located, not found. It names no consequence the author could check.
Replay at the original commit#
Recreate the pull request exactly as the reviewer of the time saw it. In a fork or trial repository where the reviewer is installed, push the pull request's final head commit to one branch and the base commit it was merged onto to another, then open a new pull request between them. Most pull requests keep their head commit reachable at refs/pull/N/head. Run the reviewer once, and score it with the answer key before reading anything else in the review.
Then judge the rest of the review as you would on live work. Incorrect findings on a replayed pull request cost the same author time as they would have at the time, and they belong in the trial's attention cost.
Know what a replay can and cannot show#
Replays measure something a live trial cannot: whether the reviewer reports defects that are known to have reached users. They are also biased in two ways. The defects you can find are the ones someone noticed and fixed, which skews toward visible failures. And a reviewer, or the model behind it, may have seen the public fix during training. Prefer pairs from your own private repositories, and pairs fixed after the reviewer's model was released, when you can tell.
Ten to twenty pairs from the last year of a repository make a useful set. Keep the set and its answer keys, and rerun them whenever you change the reviewer's configuration, so the comparison uses the same pull requests each time.