Learning center

New-format messages still in the queue after a rollback

Rolling back the producer does not roll back the messages it already sent. The restored consumer meets them in the queue, fails on each one and retries it. In our lab, two new-format messages among seven stayed pending after the rollback and were delivered six times each without being processed. Count the waiting messages by version before rolling back, and decide what happens to them.

Messages that outlive the release#

Release 2026.11.4 moved a payment's amount into an object, {"amount": {"cents", "currency"}}, and marked its messages v=2. Before anyone noticed a problem, it had put two such messages on the stream among five old-format ones. We ran this on Redis 7.0.15 from Python 3.11. First, what was waiting:

counts = {}
for _, fields in redis.xrange("payments"):
    counts[fields["v"]] = counts.get(fields["v"], 0) + 1
# {'1': 5, '2': 2}

Then the rollback to 2026.11.3, whose handler reads amount_cents. A failed message stays unacknowledged, and a recovery loop reclaims pending messages with XAUTOCLAIM and tries again, as Redis's consumer group guide suggests for crashed consumers:

def old_handle(m):
    return m["amount_cents"]            # KeyError on version 2

After the first read and five recovery passes, five payments were charged, and:

pending 1795341878928-2 delivered 6 times
pending 1795341878928-5 delivered 6 times

The two orders are neither charged nor failed, and the loop retries them forever. In a queue that delivers in order to one consumer, such as a single partition, the same message would also block every message behind it.

Decide before the rollback#

The count by version is the size of the problem. With it, choose one of three paths:

  • Drain first. If the new consumer works and only something else needs rolling back, let it process the v=2 messages, then roll back. This needs the new consumer to stay attached until the count reaches zero.
  • Park them. Have the restored consumer move messages it cannot read to a separate stream for the release that can, and acknowledge them on the main stream. A dead-letter path with a delivery limit is the general form: after a few deliveries, move the message aside. Then roll forward and replay the parked stream.
  • Convert them. If the old shape can express the message, as amount_cents can for EUR amounts, a small job rewrites them into new messages and acknowledges the originals.

What not to do is let the old consumer guess. A handler that treats a missing amount_cents as 0 would have acknowledged both payments as charged.

Make the next rollback cheaper#

Mark every message with its format version, as the producer did here, so the count takes one scan. Give the consumer a delivery limit before it moves a message aside, so a format it cannot read never loops. And ship consumers that read both formats before producers send the new one, as change a queue message so old and new consumers both handle it describes; then the rollback target already handles what the newer release produced.

The rollback drill#

In staging, produce messages with the new release, stop it, start the previous consumer, and watch the pending list. Pass when every new-format message ends processed, parked or converted within the delivery limit, and none is acknowledged without its effect.

One month of free code review.

New organizations get one calendar month of hosted code review for private and public GitHub repositories. No card or cloud connection required. Eligibility and fair-use terms.

Start free with GitHub

Related reading