Learning center

When a review says the test mocks away the code it tests

Ask what change to the production code would make the test fail. If deleting or breaking the unit leaves the test green, because a mock or the test body stands in for it, the comment deserves a change: call the real unit and replace only what it reaches outside the process. If breaking the unit turns the test red, dismiss the comment and say which break you tried, however many mocks the test uses.

A test that performed the step it was testing#

In March 2026, a vLLM pull request added an environment variable, VLLM_RAY_CGRAPH_BUFFER_SIZE_BYTES, so users could raise Ray's compiled-graph buffer size for multi-node pipeline parallelism. The lines that matter here are in the Ray executor:

if envs.VLLM_RAY_CGRAPH_BUFFER_SIZE_BYTES > 0:
    os.environ.setdefault(
        "RAY_CGRAPH_buffer_size_bytes",
        str(envs.VLLM_RAY_CGRAPH_BUFFER_SIZE_BYTES),
    )

Pull request

The first version of test_env_var_mapping compared a string with a copy of itself (comments removed):

vllm_buffer_size = "536870912"
expected_ray_env = vllm_buffer_size
self.assertEqual(vllm_buffer_size, expected_ray_env)

Test at the second commit

A later commit, titled "Address review comments: narrow exception to RuntimeError, switch to logger.info, improve test_env_var_mapping", replaced it with a test that reads the new variable, calls os.environ.setdefault itself, and asserts the result. The last commit, "test: rewrite useless tautological test to actually verify propagation", removed the comments but kept the same body. Test at the last commit

The gemini-code-assist reviewer looked at that last commit and said: "However, the current implementation is tautological; it manually performs the action it's supposed to be testing (os.environ.setdefault) and then asserts the result." It added: "This doesn't actually test the propagation logic within the RayDistributedExecutor." Review comment Its suggested test called the executor's _compiled_ray_dag method with Ray's installation check and graph classes patched out, then checked the environment with the variable set and unset.

Nobody replied. The pull request was marked stale and closed for inactivity on 24 September 2026, so the comment was never answered either way.

Check the claim by deleting the unit#

The comment is easy to check without running Ray. The test file imports os, unittest, unittest.mock.patch and environment_variables from vllm.envs. It never imports the executor. Deleting the executor lines therefore cannot change its result.

I reproduced the structure in Python with a stand-in for those lines and a mutant that drops them:

import os
from unittest import mock

def compile_dag(buffer_size):  # stands in for the executor's new lines
    if buffer_size > 0:
        os.environ.setdefault("RAY_CGRAPH_buffer_size_bytes", str(buffer_size))

def compile_dag_deleted(buffer_size):  # mutant: the new lines removed
    pass

def tautological_test(unit):
    with mock.patch.dict(os.environ, {}, clear=True):
        os.environ.setdefault("RAY_CGRAPH_buffer_size_bytes", str(536870912))
        return os.environ["RAY_CGRAPH_buffer_size_bytes"] == "536870912"

def test_that_calls_the_unit(unit):
    with mock.patch.dict(os.environ, {}, clear=True):
        unit(536870912)
        return os.environ.get("RAY_CGRAPH_buffer_size_bytes") == "536870912"

Run against both units with Python 3, the tautological test returns True for the real lines and for the mutant. The test that calls the unit returns True for the real lines and False for the mutant. The bot's claim was right, and the last commit's message describes a rewrite that the test body does not contain.

Count what is real, not how many mocks there are#

The bot's own suggestion patches several Ray objects. That is fine, because the method under test still runs for real and the patched objects are what it calls out to. Python's unittest.mock documentation puts the principle this way: "you patch where an object is looked up", and the name you patch is "the name used by the system under test". Where to patch The same page warns that tests using mocks of an old API "will still pass. This means your tests can all pass even though your code is broken." Autospeccing With autospec=True, a mocked function keeps the real call signature and raises TypeError when called incorrectly.

So a comment that says "this test mocks too much" is right only when the mocks replace the logic the test is named for. A test with six mocks of the network, the clock and a database can still pin the unit's decisions. A test with one mock can prove nothing if that mock is the unit.

Decide#

What you findAnswer
The test never calls the unit, or calls a mock of itChange it to call the real unit and mock its dependencies
The test repeats the unit's logic in its own bodyChange it the same way. The vLLM case is this row
Breaking the unit makes the test fail, despite many mocksDismiss and name the break you tried
The mocks drift from the real API without failingAdd autospec, or a test against the real dependency
You cannot call the unit without a large harnessTest the smallest real function that holds the logic, or open an issue to extract one

A test that calls the real unit can still check too little, which is the weak assertion guide's subject. To confirm a claimed defect with a test that fails first, see the regression test guide.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading