---
title: "Which \"add a test\" review comments deserve a test | Proofline"
description: "A request for a test deserves one when the pull request fixes a bug or changes behavior and no existing test fails on the old code."
url: https://proofline.sh/learn/ai-review-missing-test-comment/
---

[Learning center](/learn/)

# Which "add a test" review comments deserve a test

A request for a test deserves one when the pull request fixes a bug or changes behavior and no existing test fails on the old code. Dismiss it, naming the test or check that already covers the code, when the change is a refactor that existing tests exercise or generated output whose generator is tested. Whatever you add has to fail before the change, or it answers the comment without testing anything.

Code review The Proofline team November 22, 2026 5 min read

## Run the old tests against the old code first

The question behind every "please add a test" comment is whether any test would notice if the change were reverted. You can answer it directly. Check out the parent of the fix, or revert the fix in a scratch branch, and run the tests that touch the changed code. If one fails, the change is already covered and you can say which test covers it. If all of them pass, the comment is right whatever the bot's reasoning was, because nothing protects the fix from being undone.

That check also tells you what the new test needs. It has to reach the input that the old code got wrong, and the old tests have just shown you which inputs they do not reach.

## A bug fix the existing test could not see

In December 2025, a contributor opened a vLLM pull request that changed one call in three `reduce_scatter` implementations, from `input_.movedim(0, dim)` to `input_.movedim(dim, 0)`. The description explained why CI had not caught the bug: the existing `reduce_scatter_test_worker` "tests on 1D tensors (shape `[8]`)" and "explicitly passes `dim=0`". [Pull request](https://github.com/vllm-project/vllm/pull/30610)

The gemini-code-assist reviewer agreed with the fix and asked for a test: "While the fix is logically correct, this change should be accompanied by a regression test to prevent this bug from reappearing in the future." It suggested "a test case that uses a multi-dimensional tensor (e.g., 2D or higher) and a non-zero `dim`". [Review comment](https://github.com/vllm-project/vllm/pull/30610#discussion_r2616331942) A maintainer asked the same thing a day later: "Can you add a test?" [Maintainer's comment](https://github.com/vllm-project/vllm/pull/30610#issuecomment-3651772538)

The author replied that they had found the bug by reading the code: "To verify the bug, I wrote a test script that triggers the error immediately." [Author's reply](https://github.com/vllm-project/vllm/pull/30610#issuecomment-3652615002) The script was posted in the comment, not committed. The pull request's only commit touches no test file, and a bot closed it for inactivity on 16 April 2026 without a merge.

The request was right. This was a bug fix, and the description itself explained why the existing test could not tell the two versions apart. But the bot's suggested shape would not have settled it either. The PyTorch reference describes [`torch.movedim`](https://docs.pytorch.org/docs/stable/generated/torch.movedim.html) this way: "Moves the dimension(s) of `input` at the position(s) in `source` to the position(s) in `destination`." I computed the resulting shapes for the old and new calls in a short Python script:

| Input shape | `dim` | Old `movedim(0, dim)` | New `movedim(dim, 0)` |
| --- | --- | --- | --- |
| `(8,)` | 0 | `(8,)` | `(8,)` |
| `(4, 6)` | 1 | `(6, 4)` | `(6, 4)` |
| `(2, 4, 6)` | 1 | `(4, 2, 6)` | `(4, 2, 6)` |
| `(2, 4, 6)` | 2 | `(4, 6, 2)` | `(6, 2, 4)` |
| `(1, 3, 2)` | 2 | `(3, 2, 1)` | `(2, 1, 3)` |

Moving axis 0 to position 1 and moving axis 1 to position 0 are the same swap, so every 2D tensor and every `dim=1` case gives the same result before and after the fix. A test written to the letter of the bot's suggestion, a 2D tensor with `dim=1`, would pass on the buggy code. Only a tensor of three or more dimensions with `dim` of 2 or more separates the two versions, which is the case the author's script used: shape `(1, 3, 2)` with `dim=2`, where the old code put a leading size of 3 in front of an assertion that it divides evenly across two ranks. I checked shapes only, without running vLLM or a distributed process group.

## When the request deserves a dismissal

Two kinds of change usually do not need a new test, and both need a reason the reviewer can check.

A refactor that moves or renames code without changing what it does is covered if a test fails when the moved code is broken. Break it on purpose: return a constant, delete a branch, swap two arguments. Then run the suite. If a test fails, dismiss the comment and name that test. If nothing fails, the refactor was never covered, and the comment found a gap that predates your change. Add a test that pins current behavior, or open an issue if the gap is outside the pull request's scope.

Generated code, such as protobuf stubs, ORM models or an OpenAPI client, is checked by testing the generator and by a CI step that regenerates the output and fails on a difference. A hand-written test of generated output repeats the generator's tests and breaks whenever the generator changes. Dismiss the request and point to the generator's tests or the regeneration check. If neither exists, the missing check is what the comment found.

## Decide

| What the change does | Evidence you collect | Answer |
| --- | --- | --- |
| Fixes a bug, and no test fails on the parent commit | Test run at the parent commit | Add a test that fails there and passes on the fix |
| Changes behavior on purpose | The tests that assert the old behavior | Update them, or add one if none fail |
| Refactors, and a test fails when you break the moved code | The failing test's name | Dismiss and name the test |
| Refactors, and nothing fails when you break it | The same experiment | Add a test, or defer to an issue if the gap predates the change |
| Regenerates code | The generator's tests or a regeneration check in CI | Dismiss and point to that check |

The vLLM pull request sits in the first row, and it shows the step that is easy to skip: run the new test against the old code before you reply. The [regression test guide](/learn/turn-review-comment-into-regression-test/) works through that step for comments that claim a defect. When the request concerns a test that exists but asserts too little, the [weak assertion guide](/learn/ai-review-weak-assertion-comment/) applies instead.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Settle a reviewer's bug claim with a test that fails first](/learn/turn-review-comment-into-regression-test/)
    
    Code review The Proofline team November 22, 2026 5 min read
    
-   ### [Is the test's assertion too weak to catch the bug?](/learn/ai-review-weak-assertion-comment/)
    
    Code review The Proofline team November 22, 2026 4 min read
