Run the old tests against the old code first#
The question behind every "please add a test" comment is whether any test would notice if the change were reverted. You can answer it directly. Check out the parent of the fix, or revert the fix in a scratch branch, and run the tests that touch the changed code. If one fails, the change is already covered and you can say which test covers it. If all of them pass, the comment is right whatever the bot's reasoning was, because nothing protects the fix from being undone.
That check also tells you what the new test needs. It has to reach the input that the old code got wrong, and the old tests have just shown you which inputs they do not reach.
A bug fix the existing test could not see#
In December 2025, a contributor opened a vLLM pull request that changed one call in three reduce_scatter implementations, from input_.movedim(0, dim) to input_.movedim(dim, 0). The description explained why CI had not caught the bug: the existing reduce_scatter_test_worker "tests on 1D tensors (shape [8])" and "explicitly passes dim=0". Pull request
The gemini-code-assist reviewer agreed with the fix and asked for a test: "While the fix is logically correct, this change should be accompanied by a regression test to prevent this bug from reappearing in the future." It suggested "a test case that uses a multi-dimensional tensor (e.g., 2D or higher) and a non-zero dim". Review comment A maintainer asked the same thing a day later: "Can you add a test?" Maintainer's comment
The author replied that they had found the bug by reading the code: "To verify the bug, I wrote a test script that triggers the error immediately." Author's reply The script was posted in the comment, not committed. The pull request's only commit touches no test file, and a bot closed it for inactivity on 16 April 2026 without a merge.
The request was right. This was a bug fix, and the description itself explained why the existing test could not tell the two versions apart. But the bot's suggested shape would not have settled it either. The PyTorch reference describes torch.movedim this way: "Moves the dimension(s) of input at the position(s) in source to the position(s) in destination." I computed the resulting shapes for the old and new calls in a short Python script:
| Input shape | dim | Old movedim(0, dim) | New movedim(dim, 0) |
|---|---|---|---|
(8,) | 0 | (8,) | (8,) |
(4, 6) | 1 | (6, 4) | (6, 4) |
(2, 4, 6) | 1 | (4, 2, 6) | (4, 2, 6) |
(2, 4, 6) | 2 | (4, 6, 2) | (6, 2, 4) |
(1, 3, 2) | 2 | (3, 2, 1) | (2, 1, 3) |
Moving axis 0 to position 1 and moving axis 1 to position 0 are the same swap, so every 2D tensor and every dim=1 case gives the same result before and after the fix. A test written to the letter of the bot's suggestion, a 2D tensor with dim=1, would pass on the buggy code. Only a tensor of three or more dimensions with dim of 2 or more separates the two versions, which is the case the author's script used: shape (1, 3, 2) with dim=2, where the old code put a leading size of 3 in front of an assertion that it divides evenly across two ranks. I checked shapes only, without running vLLM or a distributed process group.
When the request deserves a dismissal#
Two kinds of change usually do not need a new test, and both need a reason the reviewer can check.
A refactor that moves or renames code without changing what it does is covered if a test fails when the moved code is broken. Break it on purpose: return a constant, delete a branch, swap two arguments. Then run the suite. If a test fails, dismiss the comment and name that test. If nothing fails, the refactor was never covered, and the comment found a gap that predates your change. Add a test that pins current behavior, or open an issue if the gap is outside the pull request's scope.
Generated code, such as protobuf stubs, ORM models or an OpenAPI client, is checked by testing the generator and by a CI step that regenerates the output and fails on a difference. A hand-written test of generated output repeats the generator's tests and breaks whenever the generator changes. Dismiss the request and point to the generator's tests or the regeneration check. If neither exists, the missing check is what the comment found.
Decide#
| What the change does | Evidence you collect | Answer |
|---|---|---|
| Fixes a bug, and no test fails on the parent commit | Test run at the parent commit | Add a test that fails there and passes on the fix |
| Changes behavior on purpose | The tests that assert the old behavior | Update them, or add one if none fail |
| Refactors, and a test fails when you break the moved code | The failing test's name | Dismiss and name the test |
| Refactors, and nothing fails when you break it | The same experiment | Add a test, or defer to an issue if the gap predates the change |
| Regenerates code | The generator's tests or a regeneration check in CI | Dismiss and point to that check |
The vLLM pull request sits in the first row, and it shows the step that is easy to skip: run the new test against the old code before you reply. The regression test guide works through that step for comments that claim a defect. When the request concerns a test that exists but asserts too little, the weak assertion guide applies instead.