Learning center

Which review findings does your linter already cover?

A finding your linter, formatter or type checker would report adds nothing a reviewer needs to say. One that contradicts your linter's configuration costs the author an explanation. Sort each trial finding against the tools already in CI before you count what the reviewer added.

See what an overlap costs#

In October 2025, a vLLM maintainer opened a pull request that removed per-file exceptions for Ruff's typing rules and fixed the code those exceptions had hidden. An automated reviewer's summary said the change from Optional[X] to X | None "has been applied inconsistently across the codebase" and added suggestions to convert the remaining files. Pull request and review

The maintainer answered: "Gemini is wrong, the files where | was added had from __future__ import annotations, so we can do this. Other files which do not cannot be updated until we drop Python 3.9." Maintainer's reply

That distinction is the linter's own rule. Ruff's UP045 rewrites Optional annotations to the | syntax. It is enabled when the project targets Python 3.10 or later, and by default also for earlier versions "if from __future__ import annotations is present, as __future__ annotations are not evaluated at runtime." Ruff UP045 The repository's lint configuration had already made the decision the reviewer reopened. The reviewer's suggestions, applied to files without that import, would have used syntax that the project's Python 3.9 support could not run.

So the finding was not merely redundant. It asked for a style the linter would have applied wherever it was valid, in exactly the places it was not.

Sort findings into four groups#

For each finding in the trial, ask which tool in CI already decides the question:

GroupExampleTreatment
Already reported by CIAn unused import that Ruff F401 or ESLint no-unused-vars flagsCount as zero value. CI would have failed the pull request anyway
Decided by CI configurationThe vLLM case: a rule that is deliberately off, or scoped to certain filesJudge it on its consequence like any finding, and record the configuration conflict beside it. The vLLM suggestions are incorrect because they would break Python 3.9, not because the rule was scoped
Could be a lint rule, but is notA naming convention the team enforces only in reviewValid, and a candidate for a new rule
Outside any linter's reachA boundary error, a missing authorization check, a stale callerThe reviewer's own contribution

Only the last two groups measure what the reviewer adds, along with any finding in the second group that names a real defect the configuration let through. In the attention cost guide's fictional trial, fourteen style comments cost little time each but added to the comments authors learned to skim. Taking the first group out of the count, and scoring the second on consequences rather than on how much it repeats the configuration, can change the result more than any other correction.

Check the configuration, not just the rule list#

Run the repository's linters on the reviewed commit, with the repository's own configuration, and keep the output beside the review. A finding matches the linter only when the linter, as configured, reports the same line. A rule that exists in Ruff or ESLint but is disabled in your project does not count as coverage. It counts as a decision. A finding that argues against it lands in the second group, where it is still right if the defect it names is real.

Where the reviewer accepts instructions, tell it which linters and type checkers run in CI and which rules the team turned off on purpose, so it reports a consequence rather than the rule. Then rerun the same pull requests at the same commits and count the first two groups again.

Turn the third group into rules#

Findings that could be a lint rule but are not are worth listing by name. A reviewer that reliably reports the same convention violation is doing work a rule could do more cheaply and on every commit. Write the rule, run it in CI and drop the instruction from the reviewer. The trial then measures the reviewer on what only a reader can find.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading