Use a change near the comment as the first signal#
A study of review comments at Microsoft found that a comment which triggered a change within one line of the lines it highlighted was very likely to be useful. The rule had 88% precision against a set of comments that their authors rated in interviews or the researchers classified by hand. Bosu, Greiler and Bird, MSR 2015 A review trial can apply the same rule to its own pull requests in a few minutes.
The rule finds candidates. It does not settle acceptance on its own. An author can fix a finding in another file, and can change a commented line for a reason that has nothing to do with the comment. Treat "changed" and "unchanged" as two piles to read, not as the result.
Read the fields GitHub keeps for each comment#
The pull request review comments API returns, for each inline comment:
original_commit_idandoriginal_line: the commit and line the reviewer commented on.commit_idandline: where GitHub places the comment now.side:RIGHTfor a line in the new version of the file,LEFTfor a deleted line.in_reply_to_id: set on replies, so you can separate the reviewer's findings from the conversation.path,user,created_atandhtml_url.
Use the original fields. They identify exactly what the reviewer saw. A comment whose line was later changed is shown as outdated in GitHub's interface, and its current placement no longer points at the code the reviewer read. GitHub's conversation view separates unresolved, resolved and outdated threads, but a resolved thread only says someone closed it. It does not say the code changed.
Run the comparison#
Fetch the pull request's comments and its final head commit, and make sure your clone has every commit the comments refer to. A force-push can remove an original commit from the branch. GitHub usually still serves such a commit by its SHA, so fetch each one the comments name; if a fetch fails, judge those comments by hand.
gh api --paginate --slurp "repos/OWNER/REPO/pulls/123/comments?per_page=100" | jq 'add' > comments.json
head=$(gh api repos/OWNER/REPO/pulls/123 --jq .head.sha)
git fetch origin pull/123/head
for sha in $(jq -r '.[].original_commit_id' comments.json | sort -u); do
git cat-file -e "$sha" 2>/dev/null || git fetch origin "$sha"
done
python3 changed_after_comment.py comments.json "$head"
The script compares each finding's original commit with the head, file by file, and reports whether any changed hunk touches the highlighted lines or the line on either side of them. A multi-line comment highlights original_start_line through original_line; a single-line comment has only original_line:
import json, re, subprocess, sys
# Usage: save the comments as one JSON array in comments.json (see above), then
# python3 changed_after_comment.py comments.json HEAD_SHA
comments = json.load(open(sys.argv[1]))
head = sys.argv[2]
def changed_ranges(base, path):
diff = subprocess.run(["git", "diff", "-U0", base, head, "--", path],
capture_output=True, text=True, check=True).stdout
for m in re.finditer(r"^@@ -(\d+)(?:,(\d+))? ", diff, re.M):
start, count = int(m[1]), int(m[2] or 1)
yield start, start + max(count, 1) - 1
for c in comments:
if c.get("in_reply_to_id") or c.get("side") != "RIGHT" or not c.get("original_line"):
continue # replies, comments on deleted lines and file-level comments
end = c["original_line"]
start = c.get("original_start_line") or end
hit = any(lo <= end + 1 and hi >= start - 1
for lo, hi in changed_ranges(c["original_commit_id"], c["path"]))
print(("changed" if hit else "unchanged").ljust(9), c["user"]["login"].ljust(24),
f'{c["path"]}:{start}', c["html_url"])
Each output line gives the result, the commenter, the file and line, and a link to the comment:
changed review-bot src/discovery/cluster.rs:105 https://github.com/...
unchanged review-bot src/pages/cluster.tsx:258 https://github.com/...
Read both piles before you count#
Open every comment in the "changed" pile and check that the change answers the finding. The guide to reconstructing an accepted finding shows what to compare. Then skim the "unchanged" pile for findings that were fixed elsewhere or accepted as follow-up work, and for findings the author answered with a reasoned refusal, which the dismissed finding guide covers.
Three cases make the script's answer wrong, and each is easy to recognize once you read the comment:
- If the author merged the default branch into the pull request after the comment, the diff to the head includes other people's changes. A line changed by someone else's merged work shows as "changed". Compare against the last commit before the merge, or use only the author's commits.
- If the author moved the commented function to another file, its old lines show as changed even when the code is identical.
- A finding about a caller that is fixed in the callee shows as "unchanged".
Count only what a person confirmed#
Report the script's piles as a sorting step, and report acceptance only after a person confirms it. In a trial of a few dozen pull requests, the script saves most of the reading and leaves the judgment where it belongs. The denominator guide explains what to divide the confirmed count by.