---
title: "Compare old and new reads on live traffic | Proofline"
description: "A shadow read serves the old source, reads the new one beside it and records any difference."
url: https://proofline.sh/learn/shadow-read-comparison/
---

[Learning center](/learn/)

# Compare old and new reads on live traffic

A shadow read serves the old source, reads the new one beside it and records any difference. It shows what users would have seen after a switch, weighted by what they actually read. It is a poor way to count wrong rows: in our simulation a day of 1% sampled shadow reads found 10 of 1,000 planted mismatches. Use shadow reads to decide whether to switch, and a reconciliation query to decide whether the data is complete.

Deployment failures The Proofline team November 22, 2026 3 min read

## The read path

```
def load_order(order_id):
    old = read_from_status(order_id)              # what the user gets
    if random.random() < SAMPLE:                  # 0.01
        new = read_from_state(order_id)
        if normalize(old) != normalize(new):
            log.warning("shadow mismatch", order_id=order_id)
    return old
```

The new read adds a query to sampled requests. Run it after the response is sent, or with a short timeout, so a slow new source does not slow the old path.

## Normalize before comparing

Two correct reads can differ in representation. We compared one order as each path returned it, in Python 3.11:

| Field | Old path | New path |
| --- | --- | --- |
| `total` | `0.30000000000000004` | `0.3` |
| `tags` | `["gift", "express"]` | `["express", "gift"]` |
| `updated_at` | `2026-11-22T10:15:03.123456+00:00` | `2026-11-22T10:15:03.123Z` |

A plain equality check reports a mismatch. After rounding the total to cents, sorting the tags and comparing timestamps at millisecond precision, the two were equal. Each normalization is a decision about what counts as the same: if the new path truncates microseconds and something depends on them, keep the comparison strict for that field and fix the path.

## What sampling finds

We planted 1,000 wrong rows among 1,000,000 and simulated 100,000 reads an hour, compared at 1%. Reads followed a skewed distribution, as real traffic does, with a few rows read far more than the rest:

| Window | Distinct rows compared | Reports | Bad rows found |
| --- | --- | --- | --- |
| 1 hour | 510 | 0 | 0 |
| 24 hours | 7,564 | 15 | 10 |

The hour compared 1,000 reads and the day 24,000; the last column counts the planted rows found, out of 1,000.

With uniform reads, the chance of at least one report in an hour is 1 − (1 − 0.001)¹⁰⁰⁰, about 63%. Under skew the hot rows are compared again and again, and most rows are never compared at all. Zero reports in the first hour said nothing, and 15 reports in a day showed the problem without its size.

## When to use which

Shadow reads answer "would users notice a switch now?" Their rate of mismatches per compared read is the rate users would see. They also exercise the new read path's queries, indexes and timeouts under real load.

A reconciliation query answers "is every row correct?" [Find every backfill mismatch with IS DISTINCT FROM](/learn/postgres-backfill-reconciliation-is-distinct-from/) covers every row, read or not, including the archive that a support agent opens once a year.

## The switch decision

Switch a first group of users when the reconciliation reports no persistent mismatch and the shadow mismatch rate, after normalization, is zero over at least a full day of traffic. Move them by a stable bucket, as [move reads to the new data source by user, not by request](/learn/switch-reads-by-stable-bucket/) explains. Keep shadow reads running in reverse after the switch, serving the new source and comparing the old, until you stop writing the old one.

## One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. [Eligibility and fair-use terms](/free-code-review/).

[Start free with GitHub](https://proofline.sh/orgs/plan?offer=free_month)

## Related reading

-   ### [Move reads to the new data source by user, not by request](/learn/switch-reads-by-stable-bucket/)
    
    Deployment failures The Proofline team November 22, 2026 2 min read
    
-   ### [Find every backfill mismatch with IS DISTINCT FROM](/learn/postgres-backfill-reconciliation-is-distinct-from/)
    
    Breaking migrations The Proofline team November 22, 2026 3 min read
