---
title: "How to tell which review comments changed the code | Proofline"
description: "GitHub records every inline review comment with the commit and line it was written on."
url: https://proofline.sh/learn/count-review-comments-that-changed-code/
---

[Learning center](/learn/)

# How to tell which review comments changed the code

GitHub records every inline review comment with the commit and line it was written on. Compare that line with the pull request's final commit to sort hundreds of comments into those followed by a change and those that were not, then have a person confirm the sorted list.

Code review The Proofline team November 20, 2026 4 min read

## Use a change near the comment as the first signal

A study of review comments at Microsoft found that a comment which triggered a change within one line of the lines it highlighted was very likely to be useful. The rule had 88% precision against a set of comments that their authors rated in interviews or the researchers classified by hand. [Bosu, Greiler and Bird, MSR 2015](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bosu2015useful.pdf) A review trial can apply the same rule to its own pull requests in a few minutes.

The rule finds candidates. It does not settle acceptance on its own. An author can fix a finding in another file, and can change a commented line for a reason that has nothing to do with the comment. Treat "changed" and "unchanged" as two piles to read, not as the result.

## Read the fields GitHub keeps for each comment

The [pull request review comments API](https://docs.github.com/en/rest/pulls/comments#list-review-comments-on-a-pull-request) returns, for each inline comment:

-   `original_commit_id` and `original_line`: the commit and line the reviewer commented on.
-   `commit_id` and `line`: where GitHub places the comment now.
-   `side`: `RIGHT` for a line in the new version of the file, `LEFT` for a deleted line.
-   `in_reply_to_id`: set on replies, so you can separate the reviewer's findings from the conversation.
-   `path`, `user`, `created_at` and `html_url`.

Use the original fields. They identify exactly what the reviewer saw. A comment whose line was later changed is shown as outdated in GitHub's interface, and its current placement no longer points at the code the reviewer read. GitHub's [conversation view](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/commenting-on-a-pull-request) separates unresolved, resolved and outdated threads, but a resolved thread only says someone closed it. It does not say the code changed.

## Run the comparison

Fetch the pull request's comments and its final head commit, and make sure your clone has every commit the comments refer to. A force-push can remove an original commit from the branch. GitHub usually still serves such a commit by its SHA, so fetch each one the comments name; if a fetch fails, judge those comments by hand.

```
gh api --paginate --slurp "repos/OWNER/REPO/pulls/123/comments?per_page=100" | jq 'add' > comments.json
head=$(gh api repos/OWNER/REPO/pulls/123 --jq .head.sha)
git fetch origin pull/123/head
for sha in $(jq -r '.[].original_commit_id' comments.json | sort -u); do
  git cat-file -e "$sha" 2>/dev/null || git fetch origin "$sha"
done
python3 changed_after_comment.py comments.json "$head"
```

The script compares each finding's original commit with the head, file by file, and reports whether any changed hunk touches the highlighted lines or the line on either side of them. A multi-line comment highlights `original_start_line` through `original_line`; a single-line comment has only `original_line`:

```
import json, re, subprocess, sys

# Usage: save the comments as one JSON array in comments.json (see above), then
#        python3 changed_after_comment.py comments.json HEAD_SHA
comments = json.load(open(sys.argv[1]))
head = sys.argv[2]

def changed_ranges(base, path):
    diff = subprocess.run(["git", "diff", "-U0", base, head, "--", path],
                          capture_output=True, text=True, check=True).stdout
    for m in re.finditer(r"^@@ -(\d+)(?:,(\d+))? ", diff, re.M):
        start, count = int(m[1]), int(m[2] or 1)
        yield start, start + max(count, 1) - 1

for c in comments:
    if c.get("in_reply_to_id") or c.get("side") != "RIGHT" or not c.get("original_line"):
        continue  # replies, comments on deleted lines and file-level comments
    end = c["original_line"]
    start = c.get("original_start_line") or end
    hit = any(lo <= end + 1 and hi >= start - 1
              for lo, hi in changed_ranges(c["original_commit_id"], c["path"]))
    print(("changed" if hit else "unchanged").ljust(9), c["user"]["login"].ljust(24),
          f'{c["path"]}:{start}', c["html_url"])
```

Each output line gives the result, the commenter, the file and line, and a link to the comment:

```
changed   review-bot               src/discovery/cluster.rs:105 https://github.com/...
unchanged review-bot               src/pages/cluster.tsx:258    https://github.com/...
```

## Read both piles before you count

Open every comment in the "changed" pile and check that the change answers the finding. The [guide to reconstructing an accepted finding](/learn/reconstruct-accepted-review-finding/) shows what to compare. Then skim the "unchanged" pile for findings that were fixed elsewhere or accepted as follow-up work, and for findings the author answered with a reasoned refusal, which the [dismissed finding guide](/learn/reconstruct-dismissed-review-finding/) covers.

Three cases make the script's answer wrong, and each is easy to recognize once you read the comment:

-   If the author merged the default branch into the pull request after the comment, the diff to the head includes other people's changes. A line changed by someone else's merged work shows as "changed". Compare against the last commit before the merge, or use only the author's commits.
-   If the author moved the commented function to another file, its old lines show as changed even when the code is identical.
-   A finding about a caller that is fixed in the callee shows as "unchanged".

## Count only what a person confirmed

Report the script's piles as a sorting step, and report acceptance only after a person confirms it. In a trial of a few dozen pull requests, the script saves most of the reading and leaves the judgment where it belongs. The [denominator guide](/learn/review-usefulness-denominator/) explains what to divide the confirmed count by.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [What should a code review trial divide by?](/learn/review-usefulness-denominator/)
    
    Code review The Proofline team November 20, 2026 3 min read
    
-   ### [Trace an accepted code review finding to the lines it changed](/learn/reconstruct-accepted-review-finding/)
    
    Code review The Proofline team November 20, 2026 3 min read
