Learning center

Answer an AI reviewer's question with a change, a test or a reason

A question from a reviewer bot is a claim it could not settle from the diff. Turn it into a check you can run. If the check shows a defect, change the code. If it shows intended behavior, pin that behavior with a test or a recorded reason, and reply with what you ran.

What the question is asking#

Reviewer bots ask questions when they notice behavior that no visible rule explains. They see the diff and some surrounding code, but usually not the callers in other repositories, the product requirement, or the conversation that led to the change. The questions come in two common forms:

  • "Should this also handle X?" asks about an input. It has a factual answer: run the code with X and see what happens, then find out whether any caller can send X.
  • "Is this intended?" asks about a behavior. Its answer is in a requirement, a convention or a decision. If none exists, your answer becomes the record.

A bare "Yes, intended" answers neither form. It tells the bot and the next reader that you looked, but not what you looked at. The bot may also ask again on the next push, because nothing in the code changed.

Work through one function#

A fictional example: a pull request adds a helper for a daily sales report.

def median_order_cents(totals: list[int]) -> int:
    values = sorted(totals)
    mid = len(values) // 2
    if len(values) % 2:
        return values[mid]
    return (values[mid - 1] + values[mid]) // 2

The reviewer asks two questions on it.

The first is "Should this also handle an empty list of orders?" Running it settles what happens. With Python 3.11, median_order_cents([]) raises IndexError: list index out of range. The next question is whether a caller can pass an empty list. The report runs once per store per day, and a store that was closed has no orders, so the empty case is ordinary input. The question deserves a code change and a test.

The second is "Is the floor division intended? statistics.median would return 250.5 for [200, 301]." The reviewer is right about the standard library. Its median uses "the common "mean of middle two" method," and for that input it returns 250.5, while the helper returns 250. statistics.median In this example the report shows whole cents, and the team chose to round down. That choice is intended, but it exists only in someone's memory until the code records it. The question deserves a test that pins the behavior and a docstring that states it.

The changed helper and its tests:

def median_order_cents(totals: list[int]) -> int | None:
    """Median order total in whole cents, rounded down; None when there are no orders."""
    if not totals:
        return None
    values = sorted(totals)
    mid = len(values) // 2
    if len(values) % 2:
        return values[mid]
    return (values[mid - 1] + values[mid]) // 2


def test_no_orders_returns_none():
    assert median_order_cents([]) is None


def test_even_count_rounds_down_to_whole_cents():
    assert median_order_cents([200, 301]) == 250

Both tests pass on Python 3.11. Returning None is a choice for this example. Raising an error, as statistics.median does with StatisticsError on empty data, would also be a defensible answer, provided the caller handles it.

The replies are then short and checkable:

Yes: a closed store has no orders that day. median_order_cents([]) raised IndexError. It now returns None, covered by test_no_orders_returns_none.

Intended: the report shows whole cents, rounded down. The docstring now says so, and test_even_count_rounds_down_to_whole_cents pins [200, 301] to 250.

Each reply names the check and leaves something in the code that answers the question for the next reader. If the bot asks again on a later push, the test is already the answer.

When the answer is a decision someone else owns#

Some questions cannot be settled by running code. In a Home Assistant pull request that changed how failed after_dependencies are handled, Copilot wrote that "a core maintainer should confirm the intended after-dependency contract." Copilot's review No test could answer that. The pull request description already stated the intent, and a maintainer's approval and merge supplied the confirmation. The guide to comments against an intended change follows that thread.

When the decision belongs to someone else, ask them in the thread, and do not answer the bot's question yourself on their behalf. Their reply, linked from the bot's thread, becomes the recorded reason.

Decide what the answer becomes#

What the check showsWhat the answer becomesReply with
The input crashes or misbehaves, and a caller can send itA code change and a testThe test name and the commit
The input misbehaves, but no caller can send itA recorded reason, such as a type, an assertion or a docstringWhat stops the input, such as the caller's validation
The behavior is intended, and a requirement says soA test that pins it, if none existsA link to the requirement
The behavior is intended, and nothing records itA test plus a docstring or commentThe reason in one sentence
Only an owner can decideTheir decisionA link to their reply

The second row needs the most care. "No caller can send it" is a claim too. Name the caller and the check that stops the input, or the next person to add a caller inherits the question unanswered. If the reviewer keeps asking the same question after you answer it, the follow-through guide shows how to count that cost.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading