---
title: "Test a code reviewer on pull requests that already caused bugs | Proofline"
description: "Your repository's history already contains the best test cases for a reviewer: pull requests that were reviewed, merged and later needed a fix."
url: https://proofline.sh/learn/replay-bug-introducing-pull-requests/
---

[Learning center](/learn/)

# Test a code reviewer on pull requests that already caused bugs

Your repository's history already contains the best test cases for a reviewer: pull requests that were reviewed, merged and later needed a fix. Recreate one at its original commit, run the reviewer on it, and score the result against what the fix later showed.

Code review The Proofline team November 20, 2026 4 min read

## Find a pair: the fix and the pull request it repairs

Start from fix pull requests that name their cause. Search your repository for descriptions containing "regression from", "introduced in" or "broke in", and for reverts. Each match gives you an answer key that nobody had to invent: the fix shows what was wrong, and the earlier pull request shows the code a reviewer would have seen.

When a fix names no cause, trace it. Blame the lines the fix changed at the commit before the fix, and find the pull request that last touched them. The [escaped defects guide](/learn/escaped-defects-after-review/) describes that tracing and where it goes wrong.

## Read a public pair

Vite's dev server added a check for port conflicts on wildcard hosts in January 2026. The pull request added this code to `packages/vite/src/node/http.ts`:

```
// Check if a port is available on wildcard addresses (0.0.0.0, ::)
async function isPortAvailable(port: number): Promise<boolean> {
  for (const host of wildcardHosts) {
    // Gracefully handle errors (e.g., IPv6 disabled on the system)
    const available = await tryListen(port, host).catch(() => true)
    if (!available) return false
  }
  return true
}

function tryListen(port: number, host: string): Promise<boolean> {
  return new Promise((resolve) => {
    const server = net.createServer()
    server.once('error', () => {
      // Ensure server is closed even on error to prevent resource leaks
      server.close(() => resolve(false))
    })
    // ...
  })
}
```

The comment on the `.catch` says that errors such as a disabled IPv6 stack should leave the port available. But `tryListen` never rejects. Every socket error resolves `false`, so the `.catch` never runs, and any error marks the port as taken. [Original pull request diff](https://github.com/vitejs/vite/pull/21381/files)

The pull request was reviewed. A maintainer requested changes about a different port-binding case, then approved the revised version, and it merged on 19 January 2026. [Review requesting changes](https://github.com/vitejs/vite/pull/21381#pullrequestreview-3658924395) · [Approval](https://github.com/vitejs/vite/pull/21381#pullrequestreview-3677575571)

On 15 February, a user reported "No available ports found between 5173 and 65535" on WSL, with the previous beta working. [User report](https://github.com/vitejs/vite/pull/21381#issuecomment-3904262666) The fix, merged the next day, made only an address-in-use error count as a conflict. Its description says other errors, such as `EACCES` from Hyper-V port reservations, had caused every port to be reported unavailable. [Fix pull request](https://github.com/vitejs/vite/pull/21642)

```
-    server.once('error', () => {
-      // Ensure server is closed even on error to prevent resource leaks
-      server.close(() => resolve(false))
+    server.once('error', (e: NodeJS.ErrnoException) => {
+      server.close(() => resolve(e.code !== 'EADDRINUSE'))
     })
```

This pair is a good test case because the defect is visible in the diff alone. A reviewer does not need the user's machine to see that the `.catch` cannot run.

## Write the answer key before the replay

From the fix, write what a reviewer must say to get credit, as you would for a [planted defect](/learn/seed-defects-review-trial/):

| Field | For this pair |
| --- | --- |
| Location | `isPortAvailable` and `tryListen` in `http.ts` |
| Trigger | Listening on a wildcard host fails with an error other than address in use |
| Consequence | The port is reported as taken; if every port fails this way, the server finds none |
| Found if | The finding says that `tryListen` never rejects, so the `.catch` fallback is dead, or that every error is treated as a conflict |

A finding that says "consider handling errors" on the `tryListen` lines is located, not found. It names no consequence the author could check.

## Replay at the original commit

Recreate the pull request exactly as the reviewer of the time saw it. In a fork or trial repository where the reviewer is installed, push the pull request's final head commit to one branch and the base commit it was merged onto to another, then open a new pull request between them. Most pull requests keep their head commit reachable at `refs/pull/N/head`. Run the reviewer once, and score it with the answer key before reading anything else in the review.

Then judge the rest of the review as you would on live work. Incorrect findings on a replayed pull request cost the same author time as they would have at the time, and they belong in the trial's [attention cost](/learn/review-attention-cost/).

## Know what a replay can and cannot show

Replays measure something a live trial cannot: whether the reviewer reports defects that are known to have reached users. They are also biased in two ways. The defects you can find are the ones someone noticed and fixed, which skews toward visible failures. And a reviewer, or the model behind it, may have seen the public fix during training. Prefer pairs from your own private repositories, and pairs fixed after the reviewer's model was released, when you can tell.

Ten to twenty pairs from the last year of a repository make a useful set. Keep the set and its answer keys, and rerun them whenever you change the reviewer's configuration, so the comparison uses the same pull requests each time.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [How to find the defects a code reviewer missed](/learn/escaped-defects-after-review/)
    
    Code review The Proofline team November 20, 2026 3 min read
    
-   ### [How to plant known defects in a code review trial](/learn/seed-defects-review-trial/)
    
    Code review The Proofline team November 20, 2026 4 min read
