Messages that outlive the release#
Release 2026.11.4 moved a payment's amount into an object, {"amount": {"cents", "currency"}}, and marked its messages v=2. Before anyone noticed a problem, it had put two such messages on the stream among five old-format ones. We ran this on Redis 7.0.15 from Python 3.11. First, what was waiting:
counts = {}
for _, fields in redis.xrange("payments"):
counts[fields["v"]] = counts.get(fields["v"], 0) + 1
# {'1': 5, '2': 2}
Then the rollback to 2026.11.3, whose handler reads amount_cents. A failed message stays unacknowledged, and a recovery loop reclaims pending messages with XAUTOCLAIM and tries again, as Redis's consumer group guide suggests for crashed consumers:
def old_handle(m):
return m["amount_cents"] # KeyError on version 2
After the first read and five recovery passes, five payments were charged, and:
pending 1795341878928-2 delivered 6 times
pending 1795341878928-5 delivered 6 times
The two orders are neither charged nor failed, and the loop retries them forever. In a queue that delivers in order to one consumer, such as a single partition, the same message would also block every message behind it.
Decide before the rollback#
The count by version is the size of the problem. With it, choose one of three paths:
- Drain first. If the new consumer works and only something else needs rolling back, let it process the
v=2messages, then roll back. This needs the new consumer to stay attached until the count reaches zero. - Park them. Have the restored consumer move messages it cannot read to a separate stream for the release that can, and acknowledge them on the main stream. A dead-letter path with a delivery limit is the general form: after a few deliveries, move the message aside. Then roll forward and replay the parked stream.
- Convert them. If the old shape can express the message, as
amount_centscan for EUR amounts, a small job rewrites them into new messages and acknowledges the originals.
What not to do is let the old consumer guess. A handler that treats a missing amount_cents as 0 would have acknowledged both payments as charged.
Make the next rollback cheaper#
Mark every message with its format version, as the producer did here, so the count takes one scan. Give the consumer a delivery limit before it moves a message aside, so a format it cannot read never loops. And ship consumers that read both formats before producers send the new one, as change a queue message so old and new consumers both handle it describes; then the rollback target already handles what the newer release produced.
The rollback drill#
In staging, produce messages with the new release, stop it, start the previous consumer, and watch the pending list. Pass when every new-format message ends processed, parked or converted within the delivery limit, and none is acknowledged without its effect.