---
title: "A reviewer says \"use a set here\": check how often the code runs | Proofline"
description: "An unmeasured micro-optimization is worth taking only when the code runs per request or per item over input that grows."
url: https://proofline.sh/learn/ai-review-performance-suggestion/
---

[Learning center](/learn/)

# A reviewer says "use a set here": check how often the code runs

An unmeasured micro-optimization is worth taking only when the code runs per request or per item over input that grows. Find where the line runs and how large its input can get. If it runs once or on a few items, decline the suggestion, or accept it only because it reads better. If it sits on a path that grows, benchmark the function and then the whole operation before you change it.

Code review The Proofline team November 22, 2026 4 min read

## What the suggestion assumes

Comments such as "use a set for O(1) lookups", "avoid this copy" or "precompile this regex" rest on an unstated premise: that the line runs often enough for its cost to show. The reviewer usually cannot see that premise. It sees the diff, not the call graph or the size of the input, so it reasons from the shape of the code. Your job is to supply the two facts it lacked: how often the line runs, and on what size of input.

## A suggestion that cut both ways

In April 2026, a vLLM pull request converted `tokenizer.all_special_ids`, a list, to a set in several model processors. Its description said: "These are initialization-time calls (`@cached_property`), not per-request hot paths". [Pull request](https://github.com/vllm-project/vllm/pull/40100)

Gemini Code Assist argued against several of the conversions. On a one-off assertion it said: "Creating a set for a single membership check is less efficient than a direct list lookup for small collections". [Review comment](https://github.com/vllm-project/vllm/pull/40100#discussion_r3097947371) On the CLIP processor's `apply()` it said: "The `apply` method is a per-request hot path." It added: "Creating a new `set` from `all_special_ids` on every call introduces unnecessary allocation and hashing overhead." [Review comment](https://github.com/vllm-project/vllm/pull/40100#discussion_r3097947381)

The author agreed: "Agreed — for the CLIP `apply()` path and the single-membership assert, the set allocation overhead isn't worth it for a <10-element list." They kept one change, in SigLIP, where "the membership check runs inside a `range(vocab_size)` loop (32k+ iterations)", adding that "the set really does help there." [Author's reply](https://github.com/vllm-project/vllm/pull/40100#issuecomment-4269616399) Ten days later they closed the pull request unmerged: "this change is too narrow to meet the substantive-work bar on its own." [Closing comment](https://github.com/vllm-project/vllm/pull/40100#issuecomment-4329089721)

## Check each premise against the code

The CLIP check runs only when a request carries image data, and it short-circuits. On `main` in November 2026, it reads `if all(tok in special_tokens for tok in inputs.prompt)`, and CLIP accepts an image only with a prompt made of special tokens. [CLIP processor](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/clip.py) So the scanned prompt is a few tokens long. To see the size of the difference, I timed both forms with Python's [`timeit`](https://docs.python.org/3/library/timeit.html):

```
import timeit
special = [49406, 49407]  # CLIP's start and end tokens
def make(prompt):
    def list_scan():
        return all(t in special for t in prompt)
    def set_per_call():
        s = set(special)
        return all(t in s for t in prompt)
    return list_scan, set_per_call
n = 200_000
for prompt in ([49406, 49407], [49406] + [49407] * 76):
    for f in make(prompt):
        t = min(timeit.repeat(f, number=n, repeat=5))
        print(f"{len(prompt)} tokens, {f.__name__}: {t / n * 1e9:.0f} ns per call")
```

Over five runs on Python 3.11 on a shared machine, the list scan was faster or level with two tokens in every run, by at most about 240 nanoseconds. With 77 tokens, building the set per call was faster in most runs, by up to about 550 nanoseconds. The bot's direction was right for the prompts this path sees, and the difference is a fraction of a microsecond once per image request.

The SigLIP change is in a `@cached_property`, which runs once per processor. Its loop is `next(token_id for token_id in range(tokenizer.vocab_size) if token_id not in all_special_ids)`, and `next()` stops at the first id that is not special. [Diff](https://github.com/vllm-project/vllm/pull/40100/files) The 32,000 iterations are an upper bound the loop reaches only if nearly every id is special. Neither side of the thread was on a hot path, which the description had already said.

## When a maintainer asks for the end-to-end number

A January 2026 vLLM pull request replaced `list.index()` lookups with a dictionary in the LoRA mapping code, and included a microbenchmark of that function. Gemini's review said that "the inclusion of new unit tests, especially the performance test, is excellent for verifying the improvement." [Review](https://github.com/vllm-project/vllm/pull/32354#pullrequestreview-3662961829) A maintainer replied: "Thank you for your contribution, could you please provide the E2E performance comparison？" [Maintainer's comment](https://github.com/vllm-project/vllm/pull/32354#issuecomment-3833618286) No comparison followed, and the pull request closed for inactivity in June.

A function that gets several times faster can still be invisible in the operation that calls it. The maintainer asked for the number that decides whether the review and the extra code are worth having.

## Decide from where the code runs

| Where the line runs | Evidence to cite | Decision |
| --- | --- | --- |
| Once, at startup or inside a cached property | The decorator or the call site | Dismiss, or accept only if the new form reads better |
| Once per request, on input with a small fixed size | The bound on the input | Dismiss, or accept if equally readable. No benchmark needed |
| Per item, on input that grows with data or traffic | The input's realistic size | Benchmark the function, then the whole operation |
| On a path a profiler already shows as hot | Profiler output from [`cProfile`](https://docs.python.org/3/library/profile.html) or your production profiler | Change, with the before and after numbers in the pull request |

When you decline, give the location and the size in one sentence, for example "runs once per processor, in a `cached_property`". When you accept for readability, say so, so that nobody later reads the change as a measured speedup. A reviewer that keeps asking for changes like these on cold paths is a candidate for a [custom review rule](/learn/write-custom-ai-code-review-rules/) that names your hot paths.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [How much engineering time does each accepted review finding cost?](/learn/review-attention-cost/)
    
    Code review The Proofline team November 20, 2026 3 min read
    
-   ### [How to write custom AI code review rules for a repository](/learn/write-custom-ai-code-review-rules/)
    
    Code review The Proofline team November 18, 2026 3 min read
