The read path#
def load_order(order_id):
old = read_from_status(order_id) # what the user gets
if random.random() < SAMPLE: # 0.01
new = read_from_state(order_id)
if normalize(old) != normalize(new):
log.warning("shadow mismatch", order_id=order_id)
return old
The new read adds a query to sampled requests. Run it after the response is sent, or with a short timeout, so a slow new source does not slow the old path.
Normalize before comparing#
Two correct reads can differ in representation. We compared one order as each path returned it, in Python 3.11:
| Field | Old path | New path |
|---|---|---|
total | 0.30000000000000004 | 0.3 |
tags | ["gift", "express"] | ["express", "gift"] |
updated_at | 2026-11-22T10:15:03.123456+00:00 | 2026-11-22T10:15:03.123Z |
A plain equality check reports a mismatch. After rounding the total to cents, sorting the tags and comparing timestamps at millisecond precision, the two were equal. Each normalization is a decision about what counts as the same: if the new path truncates microseconds and something depends on them, keep the comparison strict for that field and fix the path.
What sampling finds#
We planted 1,000 wrong rows among 1,000,000 and simulated 100,000 reads an hour, compared at 1%. Reads followed a skewed distribution, as real traffic does, with a few rows read far more than the rest:
| Window | Distinct rows compared | Reports | Bad rows found |
|---|---|---|---|
| 1 hour | 510 | 0 | 0 |
| 24 hours | 7,564 | 15 | 10 |
The hour compared 1,000 reads and the day 24,000; the last column counts the planted rows found, out of 1,000.
With uniform reads, the chance of at least one report in an hour is 1 − (1 − 0.001)¹⁰⁰⁰, about 63%. Under skew the hot rows are compared again and again, and most rows are never compared at all. Zero reports in the first hour said nothing, and 15 reports in a day showed the problem without its size.
When to use which#
Shadow reads answer "would users notice a switch now?" Their rate of mismatches per compared read is the rate users would see. They also exercise the new read path's queries, indexes and timeouts under real load.
A reconciliation query answers "is every row correct?" Find every backfill mismatch with IS DISTINCT FROM covers every row, read or not, including the archive that a support agent opens once a year.
The switch decision#
Switch a first group of users when the reconciliation reports no persistent mismatch and the shadow mismatch rate, after normalization, is zero over at least a full day of traffic. Move them by a stable bucket, as move reads to the new data source by user, not by request explains. Keep shadow reads running in reverse after the switch, serving the new source and comparing the old, until you stop writing the old one.