Learning center

Settle a reviewer's bug claim with a test that fails first

When a review comment claims a defect, write the smallest test that feeds the code the claimed input. Run it on the code before and after the fix. A failure before and a pass after confirms the claim and keeps the fix in place. A pass on both versions means the claim is wrong for that input or the test never reached it, and you should vary the input before you dismiss anything.

A claim the pull request's own tests could not reach#

In September 2025, a gemini-cli pull request added automatic cleanup of old chat sessions, with a maxCount setting that keeps only the most recent sessions. The gemini-code-assist review summary said: "The count-based retention (maxCount) logic is flawed and may not delete the correct number of sessions." Review The inline thread is resolved and its text is hidden on the page, so the summary sentence is all I can quote.

The reviewed code sorted sessions newest first, skipped the active session, and deleted any session at index maxCount or later:

for (let i = 0; i < sortedSessions.length; i++) {
  const session = sortedSessions[i];
  if (session.isCurrentSession) {
    continue;
  }
  // age-based check omitted
  if (retentionConfig.maxCount && i >= retentionConfig.maxCount) {
    shouldDelete = true;
  }

Reviewed file

A later commit in the pull request, titled "Fix count-based retention to include the active session in the count", removes the active session from the list first and keeps maxCount - 1 of the rest. That commit changed only sessionCleanup.ts. The pull request merged on 6 October 2025 with a test named should keep N most recent deletable sessions, which puts the active session first, as the newest, and expects three of six sessions kept with maxCount: 3. Merged diff

Find the input that separates the two versions#

I copied both loops into a Node script and ran them on that test's sessions. Both versions delete the same three sessions and keep three. When the active session is the newest, it sits at index 0, the old loop skips it, and every other index lines up with the new count. The merged test passes on the code the bot flagged, so it cannot say whether the bot was right.

The versions differ only when the active session is older than the sessions maxCount should keep. This is the smallest arrangement I found that separates them, with the new logic simplified for a list that contains an active session:

// Sessions sorted newest first; the active session is the oldest.
const sorted = [
  { id: '1d' }, { id: '2d' }, { id: '3d' }, { id: 'current', isCurrentSession: true },
];
const maxCount = 2;

function oldLogic() {
  const del = [];
  for (let i = 0; i < sorted.length; i++) {
    if (sorted[i].isCurrentSession) continue;
    if (maxCount && i >= maxCount) del.push(sorted[i].id);
  }
  return del;
}

function newLogic() {
  const deletable = sorted.filter((s) => !s.isCurrentSession);
  const maxDeletable = Math.max(0, maxCount - 1);
  return deletable.slice(maxDeletable).map((s) => s.id);
}

console.log('kept by old:', sorted.length - oldLogic().length);
console.log('kept by new:', sorted.length - newLogic().length);

With Node 22 this prints kept by old: 3 and kept by new: 2. The bot's claim holds: with maxCount: 2, the old code kept three sessions. A test with this arrangement would fail on the reviewed commit and pass on the fix. I ran the copied loops, not the repository's test suite, and I did not check how often the CLI's active session ends up older than others.

A test that did settle the claim#

The same week, another gemini-cli pull request fixed dollar signs that the edit tool turned into line breaks, by switching from replaceAll to split and join. Its first unit test replaced 'price is $100' with 'price is $200'. The gemini-code-assist review said "the new unit tests, while a good start, do not cover the specific edge cases (like the $' pattern)". Review A later commit, titled "test(edit): add tests to ensure $ is treated literally in replacements", added a test that replaces 'foo' with "bar$'baz".

MDN lists $' as a special replacement pattern that "Inserts the portion of the string that follows the matched substring", while $n applies only when the pattern is a RegExp. String.prototype.replace So $2 in $200 is literal text when the pattern is a string, and the first test cannot fail. I ran both tests against both implementations in Node 22:

TestreplaceAll (before)split/join (after)
'price is $100' → 'price is $200'passespasses
'foo' → "bar$'baz"fails, returns "barbaz"passes

The bot's claim was right, and the second test is the one that settles it. The pull request merged on 5 September 2025.

Run it on both sides of the fix#

  1. Restate the claim as an input and the wrong output it produces. "The retention logic is flawed" becomes "with the active session oldest and maxCount: 2, three sessions survive."
  2. Write the test at the lowest layer that accepts that input. A unit test of the replacement function settles the $' claim. A test that runs the model end to end adds a source of variation the claim does not need.
  3. Run it on the parent of the fix and on the fix. With Git, git checkout <parent> -- path/to/file gives you the old code without losing the test.
  4. Read the result:
Before the fixAfter the fixWhat it means
FailsPassesThe claim is confirmed. Keep the test
PassesPassesThe claim is wrong for this input, or the test misses it. Try the claim's other inputs before dismissing
FailsFailsThe fix is incomplete
PassesFailsThe test or the fix is wrong
  1. Reply on the thread with the test's name and both results. A reviewer can check that in a minute, and it outlasts the thread.

If the test is hard to write because no layer accepts the input directly, that is a design finding worth an issue of its own. When the comment asks for a test without claiming a defect, start from which test requests deserve a test. To record the result in a review trial, the accepted finding guide shows the row to keep.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading