---
title: "Is the test's assertion too weak to catch the bug? | Proofline"
description: "Name the defect the test exists to catch, put that defect into the code, and run the test."
url: https://proofline.sh/learn/ai-review-weak-assertion-comment/
---

[Learning center](/learn/)

# Is the test's assertion too weak to catch the bug?

Name the defect the test exists to catch, put that defect into the code, and run the test. If the test still passes, the reviewer is right and the assertion needs to change, in this pull request or in a follow-up issue if the test predates it. If the test fails, dismiss the comment and say which defect you introduced.

Code review The Proofline team November 22, 2026 4 min read

## A count that cannot see the wrong adapter

In August 2025, a vLLM pull request moved a LoRA test into the LoRA test directory. Its stated purpose was to "Move `tests/entrypoints/llm/test_generate_multiple_loras.py` to LoRA-related tests and use smaller models for testing". [Pull request](https://github.com/vllm-project/vllm/pull/23890) The moved test sends two prompts with two LoRA requests and checks the result like this:

```
outputs = llm.generate(PROMPTS, lora_request=lora_request)
assert len(PROMPTS) == len(outputs)
```

The gemini-code-assist reviewer said the test "uses the same LoRA adapter for all prompts and only asserts that the number of outputs matches the number of prompts." It named the defects that would pass: "Applying the wrong LoRA to a prompt" and "Applying only the first LoRA to all prompts in a batch." It suggested using distinct adapters and checking "that the generated output corresponds to the correct LoRA." [Review comment](https://github.com/vllm-project/vllm/pull/23890#discussion_r2308976311)

Nobody replied in that thread. A maintainer asked for the file to be renamed, the author did it, and the pull request merged on 29 August 2025 with the assertion unchanged. The same assertion is still there in `tests/lora/test_llm_with_multi_loras.py` at the v0.20.0 release tag. [File at v0.20.0](https://raw.githubusercontent.com/vllm-project/vllm/v0.20.0/tests/lora/test_llm_with_multi_loras.py)

## Check the claim by breaking the code

The bot named two defects. The question is whether anything in the test would fail if either one were real. Running the real test needs a vLLM install and model downloads, so I checked the logic on a reduced model in Python, where `generate` returns each prompt tagged with the adapter it used:

```
def generate(prompts, loras):
    return [f"{p} -> {lora}" for p, lora in zip(prompts, loras)]

def generate_first_lora_only(prompts, loras):  # mutant
    return [f"{p} -> {loras[0]}" for p in prompts]

def count_test(gen):
    out = gen(["Hello"] * 2, ["Alice", "Bob"])
    return len(out) == 2

def mapping_test(gen):
    out = gen(["Hello"] * 2, ["Alice", "Bob"])
    return out[0].endswith("Alice") and out[1].endswith("Bob")
```

The count test passes on both the correct function and the mutant. The mapping test passes on the correct one and fails on the mutant. The count assertion would stay green even with two distinct adapters, so changing the fixture alone does not answer the comment. The assertion has to read something the defect changes.

Within the file, I also checked whether a neighboring test covers the defect. `test_multi_loras_with_tp_sync` compares Alice's output with an exact expected string, but it sends one prompt per call, so it cannot see a batch that applies the first adapter to every prompt. I did not search vLLM's other LoRA tests.

The reviewer's claim was correct. The pull request only moved the test, so the right answer was a follow-up issue rather than a rewrite in a cleanup change. As far as the thread shows, the comment got neither.

## Assertions that prove less than they look

The same experiment works for each form of the comment:

-   A test with no assertion passes whenever the code returns. Introduce a wrong return value and it stays green.
-   A test that asserts only that no exception was raised catches crashes, which is sometimes the whole point of a smoke test. Introduce a wrong value. If the test is meant to catch only crashes, say so in its name and dismiss the comment on that ground.
-   A snapshot of a whole object or page fails on every unrelated change, so people update it without reading it. Jest's own guidance asks for snapshots that are "focused, short". [Jest snapshot testing](https://jestjs.io/docs/snapshot-testing#1-treat-snapshots-as-code) Introduce the defect and check whether the diff that results is one a reviewer would notice.
-   A test that never checks a subprocess's exit status cannot see a crash. In a February 2026 vLLM pull request, the author found that `test_metrics_exist_run_batch` "passes even when the engine crashes on startup (false positive)" because "`proc.wait()` never checks the return code." [Pull request](https://github.com/vllm-project/vllm/pull/34566)

## Decide

| What you find | Answer |
| --- | --- |
| The defect the comment names leaves the test green | Change the assertion so it reads what the defect changes |
| The same, but the test predates the pull request | Open an issue with the defect and the mutation, and link it in your reply |
| The test fails on that defect | Dismiss and name the mutation you tried |
| The test is a smoke test meant to catch crashes only | Dismiss, and make the name or a comment say so |
| The comment names no defect | Ask what would pass that should fail, or dismiss for lack of a claim |

The first row is the common one, and the vLLM case shows why fixtures and assertions have to change together. If the test's weakness comes from replacing the code under test with a mock, the [over-mocked test guide](/learn/ai-review-over-mocked-test-comment/) covers that variant, and the [regression test guide](/learn/turn-review-comment-into-regression-test/) shows how to run a test against both sides of a fix.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [When a review says the test mocks away the code it tests](/learn/ai-review-over-mocked-test-comment/)
    
    Code review The Proofline team November 22, 2026 4 min read
    
-   ### [Settle a reviewer's bug claim with a test that fails first](/learn/turn-review-comment-into-regression-test/)
    
    Code review The Proofline team November 22, 2026 5 min read
