A count that cannot see the wrong adapter#
In August 2025, a vLLM pull request moved a LoRA test into the LoRA test directory. Its stated purpose was to "Move tests/entrypoints/llm/test_generate_multiple_loras.py to LoRA-related tests and use smaller models for testing". Pull request The moved test sends two prompts with two LoRA requests and checks the result like this:
outputs = llm.generate(PROMPTS, lora_request=lora_request)
assert len(PROMPTS) == len(outputs)
The gemini-code-assist reviewer said the test "uses the same LoRA adapter for all prompts and only asserts that the number of outputs matches the number of prompts." It named the defects that would pass: "Applying the wrong LoRA to a prompt" and "Applying only the first LoRA to all prompts in a batch." It suggested using distinct adapters and checking "that the generated output corresponds to the correct LoRA." Review comment
Nobody replied in that thread. A maintainer asked for the file to be renamed, the author did it, and the pull request merged on 29 August 2025 with the assertion unchanged. The same assertion is still there in tests/lora/test_llm_with_multi_loras.py at the v0.20.0 release tag. File at v0.20.0
Check the claim by breaking the code#
The bot named two defects. The question is whether anything in the test would fail if either one were real. Running the real test needs a vLLM install and model downloads, so I checked the logic on a reduced model in Python, where generate returns each prompt tagged with the adapter it used:
def generate(prompts, loras):
return [f"{p} -> {lora}" for p, lora in zip(prompts, loras)]
def generate_first_lora_only(prompts, loras): # mutant
return [f"{p} -> {loras[0]}" for p in prompts]
def count_test(gen):
out = gen(["Hello"] * 2, ["Alice", "Bob"])
return len(out) == 2
def mapping_test(gen):
out = gen(["Hello"] * 2, ["Alice", "Bob"])
return out[0].endswith("Alice") and out[1].endswith("Bob")
The count test passes on both the correct function and the mutant. The mapping test passes on the correct one and fails on the mutant. The count assertion would stay green even with two distinct adapters, so changing the fixture alone does not answer the comment. The assertion has to read something the defect changes.
Within the file, I also checked whether a neighboring test covers the defect. test_multi_loras_with_tp_sync compares Alice's output with an exact expected string, but it sends one prompt per call, so it cannot see a batch that applies the first adapter to every prompt. I did not search vLLM's other LoRA tests.
The reviewer's claim was correct. The pull request only moved the test, so the right answer was a follow-up issue rather than a rewrite in a cleanup change. As far as the thread shows, the comment got neither.
Assertions that prove less than they look#
The same experiment works for each form of the comment:
- A test with no assertion passes whenever the code returns. Introduce a wrong return value and it stays green.
- A test that asserts only that no exception was raised catches crashes, which is sometimes the whole point of a smoke test. Introduce a wrong value. If the test is meant to catch only crashes, say so in its name and dismiss the comment on that ground.
- A snapshot of a whole object or page fails on every unrelated change, so people update it without reading it. Jest's own guidance asks for snapshots that are "focused, short". Jest snapshot testing Introduce the defect and check whether the diff that results is one a reviewer would notice.
- A test that never checks a subprocess's exit status cannot see a crash. In a February 2026 vLLM pull request, the author found that
test_metrics_exist_run_batch"passes even when the engine crashes on startup (false positive)" because "proc.wait()never checks the return code." Pull request
Decide#
| What you find | Answer |
|---|---|
| The defect the comment names leaves the test green | Change the assertion so it reads what the defect changes |
| The same, but the test predates the pull request | Open an issue with the defect and the mutation, and link it in your reply |
| The test fails on that defect | Dismiss and name the mutation you tried |
| The test is a smoke test meant to catch crashes only | Dismiss, and make the name or a comment say so |
| The comment names no defect | Ask what would pass that should fail, or dismiss for lack of a claim |
The first row is the common one, and the vLLM case shows why fixtures and assertions have to change together. If the test's weakness comes from replacing the code under test with a mock, the over-mocked test guide covers that variant, and the regression test guide shows how to run a test against both sides of a fix.