---
title: "A reviewer says your test is flaky: find the source first | Proofline"
description: "Treat a flakiness comment as a claim about one source of nondeterminism, such as a fixed sleep, the wall clock, test order or a shared port."
url: https://proofline.sh/learn/ai-review-flaky-test-comment/
---

[Learning center](/learn/)

# A reviewer says your test is flaky: find the source first

Treat a flakiness comment as a claim about one source of nondeterminism, such as a fixed sleep, the wall clock, test order or a shared port. Change the test when you can point to that source in the code and make it fail by varying it. Dismiss the comment when you can show the named source cannot change what the test asserts.

Code review The Proofline team November 22, 2026 4 min read

## Name the source before you change anything

pytest's documentation defines a flaky test as "one that exhibits intermittent or sporadic failure, that seems to have non-deterministic behaviour." [pytest: flaky tests](https://docs.pytest.org/en/stable/explanation/flaky.html) A reviewer's flakiness comment is useful only if it names what varies between runs. Find it in the test, then ask whether a different value of it changes the assertion's result. A comment that names no source gives you nothing to check, and rerunning the suite until it passes tells you nothing about it either.

## A real source and a proposed one in the same pull request

In February 2026, a vLLM pull request stabilized two metrics tests after failures in AMD nightly CI. The description named the first source exactly: "`test_abort_metrics_reset` is flaky due to hardware-dependent fixed sleeps". The test sent completion requests, slept for half a second, then asserted that requests were still running. "On fast hardware, generation completes before the check." The failure log in the description shows the assertion `assert 0.0 > 0`. [Pull request](https://github.com/vllm-project/vllm/pull/34566)

The fix replaced the fixed sleeps with a helper that polls the metrics endpoint until a condition holds or ten seconds pass, and raised `max_tokens` from 100 to 500 to widen the window in which requests are running. The varying quantity was generation speed. The fix makes the test wait for the state it asserts instead of guessing how long that takes.

The gemini-code-assist reviewer then raised a second possible problem in the new helper. Its predicate called the metrics endpoint with `requests.get()` inside an async test: "In an `async` function, blocking calls can stall the event loop, which can lead to subtle issues and performance degradation in concurrent tests." [Review comment](https://github.com/vllm-project/vllm/pull/34566#discussion_r2808083087)

The author declined: "The `requests.get` hits a localhost metrics endpoint and returns in milliseconds. The event loop stall is negligible compared to the `asyncio.sleep(0.1)` / `asyncio.sleep(0.2)` intervals between polls, which is where the actual yielding happens." They added that the original code already made that call from within the same async test, and that making it async would mean updating call sites across several tests. [Author's reply](https://github.com/vllm-project/vllm/pull/34566#discussion_r2808088349) A maintainer approved, and the pull request merged on 16 February 2026 with the synchronous call.

The dismissal holds up when you check what the stall could change. The test server is a separate process: `RemoteOpenAIServer` in `tests/utils.py` starts it with `subprocess.Popen`. [tests/utils.py at the merge](https://github.com/vllm-project/vllm/blob/1e82857/tests/utils.py) A blocked event loop in the test process pauses the test's own client for as long as one localhost request takes, while generation continues in the server, and the poll still has ten seconds to see the state. The bot described a real cost of blocking calls in async code. It did not name a way for that cost to flip this assertion.

One source remains. The same pull request replaced a hard-coded port `8001` with `get_open_port()`, which by default binds a socket to port 0, reads the port the operating system assigned, and closes the socket. Another process can take the port before the server binds it. A later vLLM pull request fixed intermittent `ZMQError: Address already in use` failures in other test fixtures that picked ports this way, and described it plainly: "This is a TOCTOU race." [Later pull request](https://github.com/vllm-project/vllm/pull/67607) That window is narrower than a fixed port that every concurrent run shares, but it is still there.

## Reproduce by varying the source

Repeating a test without varying the source rarely reproduces anything. Run it many times while changing the thing the comment names:

| Claimed source | How to make it vary | What a fix looks like |
| --- | --- | --- |
| Fixed sleep before an assertion | Repeat under CPU load, on slower or faster machines, or with the sleep shortened | Poll for the asserted state with a timeout |
| Wall clock or time zone | Freeze or shift the clock, run near midnight or a DST change | Inject the clock or mock `Date.now()`, as [Jest's guide](https://jestjs.io/docs/snapshot-testing#2-tests-should-be-deterministic) shows |
| Test order or shared state | Shuffle order: [pytest-randomly](https://github.com/pytest-dev/pytest-randomly), `go test -shuffle=on` | Reset the state in setup or teardown |
| Ports or network | Run tests in parallel, with [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) or several jobs on one host | Let the server bind port 0 and report its port, or stub the network |

For plain repetition, [pytest-repeat](https://github.com/pytest-dev/pytest-repeat) takes `--count=N`, [pytest-flakefinder](https://github.com/dropbox/pytest-flakefinder) runs each test 50 times by default with `--flake-finder`, and Go's [`-count`](https://pkg.go.dev/cmd/go/internal/test) flag "Run\[s\] each test, benchmark, and fuzz seed n times". A hundred passes under a varied source is evidence. A hundred passes on an idle laptop is not, because the source never changed.

## Decide

| What you find | Answer |
| --- | --- |
| The named source is in the test, and varying it produces a failure | Change the test to wait for state or control the source |
| The named source is in the test, but you cannot make it fail | Change it anyway if the fix is small. Otherwise reply with what you varied and how many runs, and keep the test |
| The named source cannot change the assertion's result | Dismiss and say why, as the vLLM author did |
| The comment names no source | Ask which value varies between runs, or dismiss for lack of a claim |

A wall-clock source often has the shape of the [date off by one guide](/learn/date-off-by-one-timezone/). If the reviewer's real complaint is that a test passes when the code is broken, the [weak assertion guide](/learn/ai-review-weak-assertion-comment/) covers it.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Is the test's assertion too weak to catch the bug?](/learn/ai-review-weak-assertion-comment/)
    
    Code review The Proofline team November 22, 2026 4 min read
    
-   ### [A date shows one day off: find where the value shifted](/learn/date-off-by-one-timezone/)
    
    Schema conflicts The Proofline team November 20, 2026 5 min read
