The lab#
1,000 carts live in PostgreSQL, and the release mirrors each cart's item count into a Redis hash that the new release reads. We sent 3,000 random "add item" requests and interrupted 1% of them after the database commit and before the Redis write, as a deploy that stops a pod would. We ran this on PostgreSQL 16.15 and Redis 7.0.15 with Python 3.11.
with db.transaction():
items = db.execute("UPDATE carts SET items = items + 1 WHERE id = %s RETURNING items", (cart,)).fetchone()[0]
# the process can stop here
redis.hset(f"cart:{cart}", "items", items)
| Redis write | Interrupted requests | Carts that differ afterwards |
|---|---|---|
HSET of the new total | 20 | 4 |
HINCRBY by one | 20 | 20 |
Writing the total heals a cart the next time the same cart changes, so only carts whose last change was interrupted stayed wrong. An increment never heals: every interrupted request is a permanent difference. Neither result raised an error, and the count of 4 would rise or fall with traffic, which makes it easy to dismiss in a test.
A rolling deploy adds a second source of drift. While the old release runs, its writes reach PostgreSQL only, because it predates the Redis copy.
The outbox#
Write the change and a record of what Redis should hold in one transaction. A relay reads the records in order, writes them to Redis and deletes them:
CREATE TABLE outbox (id bigserial PRIMARY KEY, key text NOT NULL, payload jsonb NOT NULL);
BEGIN;
UPDATE carts SET items = items + 1 WHERE id = $1 RETURNING items;
INSERT INTO outbox (key, payload) VALUES ('cart:' || $1, jsonb_build_object('items', $items));
COMMIT;
def relay():
while True:
with db.transaction():
rows = db.execute("""SELECT id, key, payload FROM outbox ORDER BY id
LIMIT 100 FOR UPDATE SKIP LOCKED""").fetchall()
if not rows:
return
for _, key, payload in rows:
redis.hset(key, mapping=payload) # the relay can stop here
db.execute("DELETE FROM outbox WHERE id = ANY(%s)", ([r[0] for r in rows],))
We interrupted the relay in 1% of record writes, 43 times in the run. Each interrupted batch stayed in the outbox and was written again by the next run. At the end, no cart differed and the outbox was empty.
Two properties make the replay harmless. The payload is the total, so writing it twice gives the same result. The relay sends records in id order, and for one cart that order matches commit order: the outbox insert comes after the UPDATE of the cart row, so a second request for the same cart waits on that row lock and takes its outbox id only after the first commits. Keep the outbox insert after the row update for that reason, and run one relay, or partition keys between relays, so an older total cannot land after a newer one.
Limits#
The outbox delays Redis by the relay's interval, so a request that writes and then reads Redis can see the previous value. Old-release writes still bypass it unless a trigger writes the outbox record, which use a trigger when the old release must keep filling the new column shows for a column. The outbox table grows if the relay stops, so alert on its oldest row.
The check#
Compare every Redis key with its row at the end of a test that interrupts both the request path and the relay. Pass when none differ and the outbox is empty. Find every backfill mismatch with IS DISTINCT FROM shows how to separate in-flight differences from real ones when the check runs during traffic.