---
title: "Do a code reviewer's severity labels mean anything? | Proofline"
description: "Many automated reviewers label findings as critical, high, medium or low. Check those labels against what happened to each finding during a trial."
url: https://proofline.sh/learn/review-severity-calibration/
---

[Learning center](/learn/)

# Do a code reviewer's severity labels mean anything?

Many automated reviewers label findings as critical, high, medium or low. Check those labels against what happened to each finding during a trial. A label earns attention only if higher labels are more often real and more often consequential than lower ones.

Code review The Proofline team November 20, 2026 4 min read

## Record the label and the outcome separately

For each finding in the trial, record the reviewer's label exactly as written, then two judgments made without looking at the label:

-   Outcome: a real defect, a valid improvement, incorrect, or unresolved.
-   Consequence if real: what would have happened had it shipped, on a scale your team already uses for incidents or bugs. A four-step scale works: none visible, degraded, broken for some users, broken for everyone or data lost.

Judge the consequence from the code and the trigger, not from the reviewer's wording. A finding that says "this could cause a security vulnerability" about a value that never leaves the server has a small consequence, however it is phrased.

## Lay the labels against the outcomes

A fictional trial produced this table:

| Reviewer's label | Findings | Real defect | Improvement | Incorrect | Real defects with a user-visible consequence |
| --- | --- | --- | --- | --- | --- |
| Critical | 4 | 1 | 0 | 3 | 1 |
| High | 9 | 4 | 1 | 4 | 3 |
| Medium | 21 | 4 | 9 | 8 | 2 |
| Low | 17 | 0 | 12 | 5 | 0 |

Two readings follow. The low label is useful: none of its 17 findings were real defects, so a team could hide low findings, or read them last, and lose no real defect it measured. The critical label is not: three of its four findings were incorrect. Those three each pulled an author away from other work, because a critical label asks for immediate attention. Meanwhile two of the six defects with a user-visible consequence arrived labelled medium.

So the low label filters well, and the critical label cannot be treated as an interrupt, because most of what it marks is wrong. Whether the labels rank real defects by consequence is a separate question. Among real defects, the share with a user-visible consequence rises from medium to high to critical, but four, four and one real defects are too few to establish an order. Record each real defect's step on the consequence scale by label, and decide on prioritizing by label only once those counts are larger.

## Separate "correct" from "critical" in a public case

In September 2025, a vLLM pull request added a Mergify rule to assign a reviewer when quantization files changed. Its conditions listed six file patterns on separate lines. An automated reviewer labelled its comment critical: "The current list of conditions for this rule will be treated as an `AND` condition by Mergify", so the rule would fire only when a pull request changed all six files. [Review comment](https://github.com/vllm-project/vllm/pull/24468#discussion_r2331527559)

A maintainer was unsure and linked the documentation: "I actually don't know if Gemini is hallucinating or not". [Reply](https://github.com/vllm-project/vllm/pull/24468#discussion_r2331531932) The reviewer was right. Mergify's documentation says that conditions on separate lines within the same rule are implicitly joined by `and`. [Mergify conditions](https://docs.mergify.com/configuration/conditions/) The author replied "Good catch! Updated." and the merged rule wraps the patterns in an `or:` block. [Merged change](https://github.com/vllm-project/vllm/pull/24468/files)

Score this finding on both axes. The outcome is a real defect. The consequence, had it shipped, is that one person would not have been assigned automatically to some pull requests: a slower review workflow, nothing a user of vLLM would see. On the four-step scale above, that is "none visible", the lowest step, under the highest label. A team that reads findings in label order would read this one before a medium finding that breaks behavior for users.

## Check one critical finding in detail

When the counts are this small, read each top-labelled finding. Ask what made the reviewer call it critical. It may be a keyword, such as "SQL", "token", "password" or "race", in code where the usual risk does not apply. A query built from constants is not an injection risk, and a lock-free counter used only for a debug log has no race worth fixing. If most incorrect critical findings share a trigger like this, a reviewer instruction that describes the real case can remove them. Rerun the same pull requests at the same commits to confirm.

## Use the labels only where they held

Write down what the trial showed about the labels, by name: "low findings were never real defects in 17; critical findings were incorrect in 3 of 4". Then set the team's handling to match. Hide or batch the label that never mattered. Do not page anyone on a label that was usually wrong. Treat the labels in between as an order to read in, not as a decision.

These are small counts. A label that was never wrong in four findings may be wrong in the next one, and the [sample size guide](/learn/review-trial-sample-size/) shows how wide that uncertainty is. Repeat the table after any change to the reviewer's model or configuration, because a new model can label the same finding differently.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [What should a code review trial divide by?](/learn/review-usefulness-denominator/)
    
    Code review The Proofline team November 20, 2026 3 min read
    
-   ### [How to find the defects a code reviewer missed](/learn/escaped-defects-after-review/)
    
    Code review The Proofline team November 20, 2026 3 min read
