Why these comments are usually right#
A reviewer that reads a diff sees the function body and the comment above it side by side, so a contradiction between them is one of the few things it can check without any outside context. Such contradictions are common because comments do not fail. A test fails when the code changes and the test was not updated; a comment that has gone wrong compiles and passes.
The work is in the second step. "The docstring no longer matches the code" means one of two things is wrong, and the bot rarely knows which.
Four reviewers and one stale comment#
In February 2026, the owner of a fork of apache/avro-rs opened a copy of upstream pull request 477, which lets Rust types provide default values for Avro record fields, and let four AI reviewers comment on it. The diff added this method to the implementation for fixed-size arrays, [T; N], at a point where the trait already documented that only Option<T> provides a default:
/// If `T` has a field default, this will return an array of elements with that default. Otherwise there is no default.
fn field_default() -> Option<serde_json::Value> {
None
}
All four reviewers flagged it. CodeRabbit wrote "Stale doc comment contradicts the implementation." Comment gemini-code-assist wrote "The comment should be updated to accurately reflect that arrays do not have a default value by default." Comment Claude wrote "The comment describes behaviour that is not implemented — the function always returns None." Comment
The author agreed and pointed to a larger fix: "The docstring is obsolete since now there is no default value by default for all types but std::option::Option. The method could be removed because it does the same as the default implementation in the AvroSchemaComponent trait." Reply
Here the code was right and the comment described a design the pull request had abandoned. The upstream pull request, merged on 25 February 2026, took the larger fix. Its commits include one titled "Remove overwrites of AvroSchemaComponent::field_default() doing the same as the default implementation…", and the merged diff documents the behavior once, on the trait: "Implementations of this trait provided by this crate return None except for Option<T> which returns Some(serde_json::Value::Null)." apache/avro-rs#477 Deleting the override removed the stale comment along with a method that repeated the trait's default implementation.
Checking which side is wrong#
An archivey pull request in October 2026 shows the check done explicitly. The parse_reparse_data docstring said a truncated payload returns None and that callers then leave the archive member as they found it. The author's description says "Both statements were wrong. No test checked what a truncated payload returns." It then records how the author decided: "for the reparse docstring, I checked whether the code or the docstring was right." "The only caller is BaseArchiveReader._apply_reparse_data." "It does not rely on None for a truncated payload." An existing code comment, "a truncated buffer should parse what is there rather than raise", showed the behavior was deliberate. "So the code is right, and only the docstring changed." PR description
The claude bot then reviewed the rewritten docstring and found it was still incomplete. Among its findings: "the caller sentence gives two outcomes where the code has three", and, about the new claim that truncated data parses to a link with no target, "Nothing asserts that." It added, "this docstring has already been wrong once". Review
The author fixed both. The docstring stopped restating the caller's policy and now points at BaseArchiveReader._apply_reparse_data, and the existing truncation test "now asserts the three bands". The reply adds, "It fails when a truncated payload is made to return None." Author's reply The pull request merged on 10 October 2026.
That last step is what turns a docstring fix into something durable. A comment that restates behavior in detail is a second implementation with no tests. Either point at the code that owns the behavior, or pin the claim with a test that fails when the behavior changes.
Decide for this comment#
| What you find | Which side is wrong | Action |
|---|---|---|
| Callers and tests rely on what the code does; the comment describes an older design | The comment | Rewrite or delete the comment in this pull request |
| The comment documents an override that now does the same as the default | The comment, and the code is redundant | Delete the override and its comment, as avro-rs did |
| Callers rely on what the comment says, and the code no longer does it | The code | Treat the finding as a bug: fix the code and add a test |
| Nobody can say which behavior is intended | Unknown | Ask the owner of the caller, and pin whichever answer wins with a test |
| The comment restates details owned by another function | Neither yet, but it will drift | Replace the detail with a reference to the owner |
The same check applies to a README, a usage message or an API reference. Run the documented example. If it fails, decide as above which side moves.
Make the claim checkable#
Where the language supports it, put the documented behavior into an executable example. Python's doctest runs examples written in docstrings, and Rust's documentation tests compile and run code blocks in doc comments as part of cargo test. A docstring example that runs in CI fails the next time the code and the comment drift apart. Then a reviewer, human or bot, has a failing check to point at instead of a contradiction to argue about.