Learning center

A reviewer says your test is flaky: find the source first

Treat a flakiness comment as a claim about one source of nondeterminism, such as a fixed sleep, the wall clock, test order or a shared port. Change the test when you can point to that source in the code and make it fail by varying it. Dismiss the comment when you can show the named source cannot change what the test asserts.

Name the source before you change anything#

pytest's documentation defines a flaky test as "one that exhibits intermittent or sporadic failure, that seems to have non-deterministic behaviour." pytest: flaky tests A reviewer's flakiness comment is useful only if it names what varies between runs. Find it in the test, then ask whether a different value of it changes the assertion's result. A comment that names no source gives you nothing to check, and rerunning the suite until it passes tells you nothing about it either.

A real source and a proposed one in the same pull request#

In February 2026, a vLLM pull request stabilized two metrics tests after failures in AMD nightly CI. The description named the first source exactly: "test_abort_metrics_reset is flaky due to hardware-dependent fixed sleeps". The test sent completion requests, slept for half a second, then asserted that requests were still running. "On fast hardware, generation completes before the check." The failure log in the description shows the assertion assert 0.0 > 0. Pull request

The fix replaced the fixed sleeps with a helper that polls the metrics endpoint until a condition holds or ten seconds pass, and raised max_tokens from 100 to 500 to widen the window in which requests are running. The varying quantity was generation speed. The fix makes the test wait for the state it asserts instead of guessing how long that takes.

The gemini-code-assist reviewer then raised a second possible problem in the new helper. Its predicate called the metrics endpoint with requests.get() inside an async test: "In an async function, blocking calls can stall the event loop, which can lead to subtle issues and performance degradation in concurrent tests." Review comment

The author declined: "The requests.get hits a localhost metrics endpoint and returns in milliseconds. The event loop stall is negligible compared to the asyncio.sleep(0.1) / asyncio.sleep(0.2) intervals between polls, which is where the actual yielding happens." They added that the original code already made that call from within the same async test, and that making it async would mean updating call sites across several tests. Author's reply A maintainer approved, and the pull request merged on 16 February 2026 with the synchronous call.

The dismissal holds up when you check what the stall could change. The test server is a separate process: RemoteOpenAIServer in tests/utils.py starts it with subprocess.Popen. tests/utils.py at the merge A blocked event loop in the test process pauses the test's own client for as long as one localhost request takes, while generation continues in the server, and the poll still has ten seconds to see the state. The bot described a real cost of blocking calls in async code. It did not name a way for that cost to flip this assertion.

One source remains. The same pull request replaced a hard-coded port 8001 with get_open_port(), which by default binds a socket to port 0, reads the port the operating system assigned, and closes the socket. Another process can take the port before the server binds it. A later vLLM pull request fixed intermittent ZMQError: Address already in use failures in other test fixtures that picked ports this way, and described it plainly: "This is a TOCTOU race." Later pull request That window is narrower than a fixed port that every concurrent run shares, but it is still there.

Reproduce by varying the source#

Repeating a test without varying the source rarely reproduces anything. Run it many times while changing the thing the comment names:

Claimed sourceHow to make it varyWhat a fix looks like
Fixed sleep before an assertionRepeat under CPU load, on slower or faster machines, or with the sleep shortenedPoll for the asserted state with a timeout
Wall clock or time zoneFreeze or shift the clock, run near midnight or a DST changeInject the clock or mock Date.now(), as Jest's guide shows
Test order or shared stateShuffle order: pytest-randomly, go test -shuffle=onReset the state in setup or teardown
Ports or networkRun tests in parallel, with pytest-xdist or several jobs on one hostLet the server bind port 0 and report its port, or stub the network

For plain repetition, pytest-repeat takes --count=N, pytest-flakefinder runs each test 50 times by default with --flake-finder, and Go's -count flag "Run[s] each test, benchmark, and fuzz seed n times". A hundred passes under a varied source is evidence. A hundred passes on an idle laptop is not, because the source never changed.

Decide#

What you findAnswer
The named source is in the test, and varying it produces a failureChange the test to wait for state or control the source
The named source is in the test, but you cannot make it failChange it anyway if the fix is small. Otherwise reply with what you varied and how many runs, and keep the test
The named source cannot change the assertion's resultDismiss and say why, as the vLLM author did
The comment names no sourceAsk which value varies between runs, or dismiss for lack of a claim

A wall-clock source often has the shape of the date off by one guide. If the reviewer's real complaint is that a test passes when the code is broken, the weak assertion guide covers it.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading