Idempotency and safe retries, from first idea to production

You tap “Pay” on a train going into a tunnel. The spinner turns for thirty seconds, then the app says the request timed out. Did the payment go through? The app doesn’t know. You tap “Pay” again, and next morning your bank shows two charges. Nothing was broken. The server did exactly what it was asked, twice.

The fix isn’t to stop retrying: retries are how a client survives a network that drops things. The fix is to make the second attempt harmless. That property is idempotency, and every reliable payment API, webhook receiver and job queue is built on it.

By the end of this article you’ll be able to:

• say exactly why a timeout forces you to choose between losing work and duplicating it;
• make an endpoint safe to retry with unique constraints, upserts, conditional updates and idempotency keys;
• write a retry policy that helps a struggling service instead of finishing it off;
• handle the hard cases: two duplicates at the same instant, queues that redeliver, side effects your database can’t roll back;
• design a payments API that never charges twice, out loud, in an interview.

Start at Beginner if “idempotent” is a word you’ve only heard in meetings. If you already put idempotency keys on POST endpoints, jump to Advanced, where the bugs live. The last section is the real code from ezychat.io, the multi-tenant messaging product I build and run.

Beginner: why the same request arrives twice

What does idempotent mean?

An operation is idempotent if doing it twice leaves things exactly as doing it once did.

Think of the call button for an elevator. You press it, it lights up, an elevator is coming. Somebody impatient presses it four more times. Still one elevator. The button doesn’t mean “send another elevator”; it means “make sure an elevator is coming”, and that’s either true already or it becomes true. Pressing it again changes nothing.

Compare a vending machine’s coin slot. Every coin you put in adds to your credit. Put it in twice, you’ve paid twice. That’s not idempotent.

The difference shows up in code as “set” versus “add”:

account = {"balance": 100, "status": "active"}


def set_status(acct, status):   # idempotent: the second call changes nothing
    acct["status"] = status


def add_credit(acct, amount):   # not idempotent: every call adds again
    acct["balance"] += amount


for _ in range(3):              # the same two requests, delivered three times
    set_status(account, "suspended")
    add_credit(account, 10)
print(account)

# {'balance': 130, 'status': 'suspended'}

Three deliveries of “suspend this account” leave one suspended account. Three deliveries of “add RM10” leave RM30 more than anyone meant. Idempotent doesn’t mean “does nothing”: the first call does real work, and the repeats do nothing.

One more word you’ll need. An operation is safe if it doesn’t change anything at all, like reading a page. Every safe operation is idempotent, but plenty of idempotent operations aren’t safe: “set status to suspended” changes the account the first time.

A timeout is a question nobody answers

Here’s why this matters. When a client sends a request and gets no answer, three very different things might have happened:

1. The request was lost on the way. The server never saw it. Retrying is exactly right.
2. The server crashed partway through the work. Maybe it did some of it, maybe none.
3. The response was lost on the way back. The server finished the job, charged the card, and the confirmation never arrived. Retrying does the job a second time.

The client can’t tell these apart. A timeout only says “I stopped waiting.” It doesn’t say what happened on the other side. So a client that just timed out has two choices, and both are bad on their own:

• Don’t retry, and risk losing the order, the message or the payment when it was case 1.
• Retry, and risk doing it twice when it was case 3.

There’s no third choice on the client side, because you can’t make the network stop losing things. What you can do is make retrying safe, so the client is free to pick the second option every time. That’s the whole article in one sentence: make the operation idempotent, then retry as much as you need.

At most once, at least once

These two failure choices have names, and you’ll see them in every queue and broker’s documentation:

• At-most-once delivery: send it once, never retry. Nothing is ever duplicated, and some things are lost.
• At-least-once delivery: retry until you get an acknowledgement. Nothing is lost (as long as you keep trying), and some things arrive twice.

What everyone wants is exactly-once. Over a network that can lose messages, exactly-once delivery can’t be built: the sender can never be sure the last acknowledgement wasn’t lost. It’s the Two Generals problem, where no number of messengers makes two armies certain they agree, because the last one might have been caught.

So real systems choose at-least-once delivery and make the receiver ignore duplicates. Stripe documents retrying webhook deliveries for up to three days; Meta redelivers WhatsApp webhooks it thinks failed for up to a week; your queue redelivers jobs when a worker dies. If your code assumes each message arrives once, it’s wrong today and you haven’t noticed yet.

HTTP already has rules for this

HTTP was designed with retries in mind, and its standard (RFC 9110) labels methods by whether a repeat is harmless:

PUT /users/42/email with {"email": "a@b.com"} sets the email; send it ten times and the email is a@b.com. DELETE /orders/7 twice leaves the order deleted. The second call might answer 404 instead of 204, which is fine: idempotency is about the state on the server, not the status code.

POST /payments is the dangerous one: each call means “make a new payment.” That’s why the standard tells clients not to retry a non-idempotent request automatically unless they know it’s safe, and why browsers ask “Confirm form resubmission?” when you reload a POST result.

PATCH depends on what you send. {"status": "shipped"} is idempotent. {"stock": "+1"} is not.

The double charge, in code

Here’s the payment story from the opening as a program. The network can lose the request or the response; the server charges a card for every request it receives; the client retries on timeout:

import itertools


class Network:
    """Delivers each call unless told to lose it, the way real networks do on bad days."""

    def __init__(self, faults):
        self.faults = iter(faults)  # one per attempt: "ok", "lose request" or "lose response"

    def call(self, server, request):
        fault = next(self.faults, "ok")
        if fault == "lose request":
            raise TimeoutError      # the server never saw it
        response = server.handle(request)
        if fault == "lose response":
            raise TimeoutError      # the server did the work, and we'll never hear about it
        return response


class PaymentServer:
    def __init__(self):
        self.ledger = []
        self.ids = itertools.count(1)

    def handle(self, request):
        charge = {"id": f"ch_{next(self.ids)}", "amount": request["amount"]}
        self.ledger.append(charge)
        return charge


def pay(network, server, amount, attempts=3):
    for attempt in range(1, attempts + 1):
        try:
            return network.call(server, {"amount": amount})
        except TimeoutError:
            print(f"  attempt {attempt}: timed out")
    raise RuntimeError("gave up")


for faults in (["lose request", "ok"], ["lose response", "ok"]):
    server = PaymentServer()
    print(faults[0])
    print("  paid:", pay(Network(faults), server, 2500))
    print("  charges on the server:", len(server.ledger))

# lose request
#   attempt 1: timed out
#   paid: {'id': 'ch_1', 'amount': 2500}
#   charges on the server: 1
# lose response
#   attempt 1: timed out
#   paid: {'id': 'ch_2', 'amount': 2500}
#   charges on the server: 2

The client’s view is identical in both runs: one timeout, then success. The server’s ledger isn’t. In the second run the customer paid twice, and the client has a receipt for ch_2 with no idea that ch_1 exists.

Try it yourself. The simulator below starts with that exact run already played. Reset it, turn idempotency keys on, and lose the response again:

With keys on, every new payment gets a unique key and every retry of it sends the same key. The server remembers each key and what it answered, so a retry gets the saved answer and the card isn’t touched. That’s an idempotency key, and building one properly is most of the next section.

Intermediate: keys, upserts and retry policy

There are three ways to make an operation idempotent, and you’ll use all three. You can let the database refuse a second copy (a unique constraint). You can make a write conditional on the state it expects (compare-and-set). Or, when the request has nothing unique about it, the client can attach a unique idempotency key. Then you need a retry policy, because a safe retry sent at the wrong moment can still hurt.

Use the id the request already has

The cheapest dedupe is the one where the sender already gave you a unique id. A WhatsApp message from Meta carries a message id. A Stripe webhook carries an event id. A row from another system has its primary key. Make that id unique in your table and let the database say no:

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("""
    CREATE TABLE messages (
        id              INTEGER PRIMARY KEY,
        org_id          TEXT NOT NULL,
        provider_msg_id TEXT NOT NULL,
        body            TEXT,
        UNIQUE (org_id, provider_msg_id)
    )""")


def receive(org_id, provider_msg_id, body):
    row = db.execute(
        """INSERT INTO messages (org_id, provider_msg_id, body) VALUES (?, ?, ?)
           ON CONFLICT (org_id, provider_msg_id) DO NOTHING
           RETURNING id""",
        (org_id, provider_msg_id, body),
    ).fetchone()
    if row is None:
        return "duplicate: already stored, do nothing"
    return f"new: stored as message {row[0]}, now reply to it"


print(receive("org-1", "wamid.ABC", "Are you open on Sunday?"))
print(receive("org-1", "wamid.ABC", "Are you open on Sunday?"))  # the provider retried
print(receive("org-2", "wamid.ABC", "Hello"))                    # same id, different tenant

# new: stored as message 1, now reply to it
# duplicate: already stored, do nothing
# new: stored as message 2, now reply to it

ON CONFLICT DO NOTHING turns “insert this” into “insert this unless it’s already there”, in one statement. RETURNING id tells you which case you’re in: a row back means you’re first, nothing back means it’s a repeat. That’s the hinge for everything after it. Do the follow-up work (reply, bill, notify) only on the first path, and a duplicate becomes a no-op that costs one index lookup.

The same pattern works in PostgreSQL, where the syntax is identical. SQLite is used here because it ships with Python, so you can run this anywhere.

Two details in that table matter more than they look. The key is (org_id, provider_msg_id), not provider_msg_id alone, which scopes it to one tenant. And it’s a constraint in the database, not a check in Python. The Advanced section measures what happens when twenty copies of one request check in Python at the same time.

This is also called an upsert when the conflict branch updates instead of doing nothing (ON CONFLICT ... DO UPDATE SET ...). Upsert is right when the latest copy should win, like a profile sync. Do-nothing is right when the first copy is the truth, like an inbound message.

Idempotency keys: when the request has no natural id

POST /payments {"amount": 2500} has nothing unique in it. Two payments of RM25 by the same customer are perfectly legal. So the client makes something unique up: a random UUID generated once per intent (“I want to pay for order 1042”), sent in a header with every attempt.

Stripe popularized the pattern, and it’s what most payment APIs now accept:

POST /v1/payments HTTP/1.1
Idempotency-Key: 8c1e7f0a-4a55-4a4e-9a51-2f1d1f0a9b11
Content-Type: application/json

{"amount": 2500, "currency": "myr"}

The server remembers three things per key: the key, the request that came with it and the answer. Stripe’s documentation describes exactly that. It saves the status code and body of the first request, returns the same result for later requests with the key, and errors if the parameters differ from the original’s. An IETF draft standardizes the Idempotency-Key header along the same lines.

Here’s the shape of it, with a dictionary as the store:

import hashlib
import itertools
import json


class PaymentsAPI:
    def __init__(self):
        self.saved = {}             # (account, key) -> (fingerprint, response)
        self.ledger = []
        self.ids = itertools.count(1)

    @staticmethod
    def fingerprint(body):
        canonical = json.dumps(body, sort_keys=True, separators=(",", ":"))
        return hashlib.sha256(canonical.encode()).hexdigest()

    def post_payment(self, account, key, body):
        fp = self.fingerprint(body)
        if (account, key) in self.saved:
            saved_fp, response = self.saved[(account, key)]
            if saved_fp != fp:
                return 422, {"error": "Idempotency-Key reused with a different request"}
            return response                       # a retry: replay the answer, don't redo the work
        charge = {"id": f"ch_{next(self.ids)}", "amount": body["amount"]}
        self.ledger.append(charge)
        response = (201, charge)
        self.saved[(account, key)] = (fp, response)
        return response


api = PaymentsAPI()
key = "8c1e7f0a-4a55-4a4e-9a51-2f1d1f0a9b11"
print(api.post_payment("acct_1", key, {"amount": 2500, "currency": "myr"}))
print(api.post_payment("acct_1", key, {"currency": "myr", "amount": 2500}))  # retry, fields reordered
print(api.post_payment("acct_1", key, {"amount": 9900, "currency": "myr"}))  # same key, new amount
print("charges:", len(api.ledger))

# (201, {'id': 'ch_1', 'amount': 2500})
# (201, {'id': 'ch_1', 'amount': 2500})
# (422, {'error': 'Idempotency-Key reused with a different request'})
# charges: 1

Three things to notice:

• The retry gets the original answer, ch_1, not a fresh “already done” message. To the client, a replay is indistinguishable from the first response arriving late, which is exactly what it should look like.
• The fingerprint is canonical. Hashing JSON with sorted keys means {"amount": 2500, "currency": "myr"} and {"currency": "myr", "amount": 2500} match. Hash the raw bytes and an SDK that orders fields differently on a retry would get a spurious 422.
• The same key with a different body is an error, not a replay. That’s a client bug (a key reused across two payments), and quietly replaying the first payment’s answer to the second would tell the customer they’d paid RM99 when they’d paid RM25.

The toy has one serious flaw: checking the dictionary and writing to it are separate steps, so two copies arriving together can both pass the check. The Advanced section fixes that first.

Where the key comes from

The key belongs to the intent, so the client creates it before the first attempt and reuses it for every retry of that intent:

import uuid


def pay_for_order(send, order):
    key = str(uuid.uuid4())                  # once per intent, before any attempt
    return call_with_retries(lambda: send(order, idempotency_key=key))

A good client stores the key with the thing it’s paying for, such as the order row, so it can crash, restart and retry tomorrow with the same key. A key held only in memory survives retries inside one process and nothing else.

Some keys needn’t be random. If the intent has a natural identity, derive the key from it: refund:order-1042, welcome-email:user-42. A deterministic key survives anything, including a client that lost its random one.

Conditional writes: compare-and-set

Not every write is an insert. To make an update idempotent, make it say what state it expects to find. If the state is no longer that, the update does nothing:

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE orders (id TEXT PRIMARY KEY, status TEXT, version INTEGER)")
db.execute("INSERT INTO orders VALUES ('ord_1', 'paid', 7)")


def ship(order_id, seen_version):
    cur = db.execute(
        "UPDATE orders SET status = 'shipped', version = version + 1 "
        "WHERE id = ? AND version = ?",
        (order_id, seen_version),
    )
    return "shipped" if cur.rowcount == 1 else "no change: the order has moved on since you read it"


print(ship("ord_1", 7))
print(ship("ord_1", 7))  # the same request, retried
print(db.execute("SELECT status, version FROM orders").fetchone())

# shipped
# no change: the order has moved on since you read it
# ('shipped', 8)

This is compare-and-set: compare the current value with the one you expect, and set the new one only if they match, in a single statement so nothing can sneak in between. The version column makes it general. Every write bumps the version, so any write based on a stale read is refused. The same idea is called optimistic concurrency control in databases, and HTTP has it built in: send If-Match with the ETag you read, and the server answers 412 Precondition Failed if the resource changed.

You don’t always need a version number. A state transition works as its own guard: UPDATE orders SET status = 'shipped' WHERE id = ? AND status = 'paid'. Only one request can move an order out of paid. The production section has a compare-and-clear on a timestamp that does the same job for an AI reply pipeline.

“Exactly once” is a sum

Put the last few ideas together and you get the most useful formula in this article:

at-least-once delivery + an idempotent receiver = effectively-once processing.

Duplicates still arrive, but each hits a unique constraint, a saved key or a failed compare-and-set and changes nothing, so the effects happen once. When a vendor sells “exactly-once”, this is what’s underneath, inside a boundary they control. Kafka’s exactly-once covers reading from and writing to Kafka; the moment your consumer sends an email, the idempotent half is yours again.

I’d go further: treat “exactly-once” in any design doc as a question, not a guarantee. Ask “where is the dedupe, and what’s its key?” If nobody can answer, it’s at-least-once with optimism.

Which errors should you retry?

A retry is only worth sending if it can succeed and is safe to repeat. That splits errors into three groups:

The rule behind it: retry when the failure is about the moment (a busy server, a dropped connection), not the request (bad input, no permission). Retrying a 400 is a slower way to fail.

Exponential backoff with full jitter

When you do retry, wait first, and wait longer each time. Retrying immediately into a struggling server adds load at the worst moment. Exponential backoff doubles the wait after each failure: 100 ms, 200 ms, 400 ms, up to a cap.

That alone has a nasty failure mode. If a thousand clients all fail at the same moment (because the server hiccuped for all of them at once), they all wait 100 ms, all retry together, all fail together, and all wait 200 ms. The waves never break up. Jitter adds randomness to the wait so they spread out. AWS’s architecture blog compared several versions in “Exponential Backoff And Jitter” (Marc Brooker, 2015) and recommended full jitter: wait a random time between zero and the exponential cap.

I simulated it: a thousand clients call a server at once, the server can answer 50 calls in each 100 ms slot, and every call over capacity fails and backs off.

import heapq
import random
from collections import Counter

BASE, CAP = 0.1, 10.0   # seconds
SLOT = 0.1              # the server works in 100 ms slots...
PER_SLOT = 50           # ...and can answer 50 calls in each one


def no_jitter(attempt, rng):
    return min(CAP, BASE * 2 ** attempt)


def full_jitter(attempt, rng):
    return rng.uniform(0, min(CAP, BASE * 2 ** attempt))


def simulate(policy, clients=1000, seed=7):
    """1,000 clients call at t=0. Calls beyond a slot's capacity fail and retry with `policy`."""
    rng = random.Random(seed)
    queue = [(0.0, i, 0) for i in range(clients)]   # (time, client, attempt)
    heapq.heapify(queue)
    per_slot = Counter()                             # calls that arrived in each slot
    done_at = 0.0
    while queue:
        t, client, attempt = heapq.heappop(queue)
        slot = int(t / SLOT)
        per_slot[slot] += 1
        if per_slot[slot] <= PER_SLOT:
            done_at = max(done_at, t)                # answered
        else:                                        # over capacity: fail, back off, try again
            heapq.heappush(queue, (t + policy(attempt, rng), client, attempt + 1))
    return per_slot, done_at


for name, policy in [("no jitter", no_jitter), ("full jitter", full_jitter)]:
    per_slot, done_at = simulate(policy)
    print(f"{name:12} calls sent: {sum(per_slot.values()):6,}   last client served at {done_at:5.1f} s")

# no jitter    calls sent: 10,500   last client served at 132.7 s
# full jitter  calls sent:  5,033   last client served at   6.5 s

Without jitter the clients stay in lockstep: waves of 950, 900, 850 arrive in the same 100 ms, the server answers 50 and the rest fail together. Getting everyone through takes twenty waves, over two minutes and ten times the useful work. Full jitter halves the calls and finishes in 6.5 seconds. With four other seeds it sent 5,020 to 5,034 calls and finished in 4.8 to 10.5 seconds; without jitter there’s no randomness, so it’s the same 10,500 calls every time.

Here’s a retry helper that puts the policy together: retry only what’s retryable, full jitter, respect Retry-After, and give up when the caller’s deadline would pass:

import random
import time

RETRYABLE = {408, 429, 500, 502, 503, 504}


class HTTPError(Exception):
    def __init__(self, status, retry_after=None):
        super().__init__(status)
        self.status, self.retry_after = status, retry_after


def call_with_retries(call, *, attempts=4, base=0.2, cap=5.0, deadline=10.0,
                      sleep=time.sleep, rng=random.random, clock=time.monotonic):
    started = clock()
    for attempt in range(attempts):
        try:
            return call()
        except (ConnectionError, TimeoutError) as exc:
            error = exc
        except HTTPError as exc:
            if exc.status not in RETRYABLE:
                raise                                   # 400, 401, 403, 404, 422: it will fail again
            error = exc
        if attempt == attempts - 1:
            break
        delay = rng() * min(cap, base * 2 ** attempt)   # full jitter
        if isinstance(error, HTTPError) and error.retry_after:
            delay = max(delay, error.retry_after)       # the server said how long; believe it
        if clock() - started + delay > deadline:
            break                                       # the caller has stopped waiting, so stop too
        sleep(delay)
    raise error


# A fake upstream: two 503s, then success. Record the sleeps instead of sleeping.
replies = iter([HTTPError(503), HTTPError(503), "ok"])


def flaky():
    reply = next(replies)
    if isinstance(reply, Exception):
        raise reply
    return reply


slept = []
print(call_with_retries(flaky, sleep=slept.append, rng=random.Random(42).random),
      [round(s, 3) for s in slept])

calls = []


def invalid():
    calls.append(1)
    raise HTTPError(422)


try:
    call_with_retries(invalid, sleep=slept.append)
except HTTPError as exc:
    print(f"{exc.status} after {len(calls)} call: not retried")

# ok [0.128, 0.01]
# 422 after 1 call: not retried

The deadline matters more than it looks. A user who gave up at ten seconds doesn’t care that your retry at second twelve succeeded. Work for a caller who has gone is waste, and under load waste is what kills you.

Retry budgets

Backoff spaces retries out in time. It doesn’t limit how many there are. If an upstream is completely down, every request still sends its full quota of retries, and a service that normally sends 1,000 calls a second to it now sends 4,000.

A retry budget caps retries as a fraction of normal traffic. Google’s SRE book (in its chapter on handling overload) describes a per-client budget where retries stay under 10% of requests; gRPC has a similar “retry throttling” setting built from tokens. Here’s a small version:

class RetryBudget:
    """Retries may add at most `ratio` extra load. Every request earns `ratio` of a token,
    every retry spends a whole one, and the bank is capped so a quiet hour can't fund a storm."""

    def __init__(self, ratio=0.1, cap=10.0):
        self.ratio, self.cap, self.tokens = ratio, cap, cap

    def on_request(self):
        self.tokens = min(self.cap, self.tokens + self.ratio)

    def can_retry(self):
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False


def calls_sent(requests, retries_each, budget=None):
    """The upstream is completely down: every call fails. How many calls do we send it?"""
    sent = 0
    for _ in range(requests):
        if budget:
            budget.on_request()
        sent += 1
        for _ in range(retries_each):
            if budget and not budget.can_retry():
                break
            sent += 1
    return sent


print("3 retries each, no budget:", calls_sent(1000, 3))
print("3 retries each, 10% budget:", calls_sent(1000, 3, RetryBudget()))

# 3 retries each, no budget: 4000
# 3 retries each, 10% budget: 1109

When everything is healthy, the budget never runs out and costs nothing. When the upstream is down, it turns a 4× load multiplier into about 1.1×. That’s the difference between an outage your upstream recovers from and one it can’t, and the Expert section shows why.

def create_order(session, cart):
    for attempt in range(5):
        key = str(uuid.uuid4())
        try:
            return session.post("/orders", json=cart, headers={"Idempotency-Key": key})
        except Exception:
            time.sleep(1)

def create_order(session, cart):
    key = cart.idempotency_key            # created once, stored with the cart
    return call_with_retries(lambda: session.post(
        "/orders", json=cart.as_json(), headers={"Idempotency-Key": key}))

CREATE TABLE idempotency_keys (
    account_id    uuid        NOT NULL,      -- keys are scoped to one account
    key           text        NOT NULL,      -- what the client sent (cap the length, e.g. 255)
    fingerprint   bytea       NOT NULL,      -- sha256 of the canonical request body
    state         text        NOT NULL,      -- 'started' while working, 'done' when the answer is saved
    response_code smallint,                  -- the saved answer, replayed to every retry
    response_body jsonb,
    order_id      uuid,                      -- what the first request created, for recovery
    locked_until  timestamptz,               -- a lease: who is working on it, and until when
    created_at    timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (account_id, key)
);

Advanced: the cases that break naive idempotency

The Intermediate toy works one request at a time. Production doesn’t. This section covers what goes wrong with concurrency, with time and with work that leaves your database.

Two identical requests at the same instant

A user double-clicks. A client retries after 2 seconds while the first attempt is still running because it’s slow, not lost. A load balancer resends a request to a second server. Now two copies of the same request are in flight together, and the toy’s “is this key in my dictionary?” check runs in both before either writes.

I measured how often that happens with a real database. Twenty threads each send the same request at the same instant (a barrier releases them together), against PostgreSQL 14 on my laptop. The naive handler checks for the key, then inserts:

import threading
import psycopg

DSN = "host=/tmp port=5499 user=postgres dbname=postgres"


def check_then_insert(conn, key):
    seen = conn.execute("SELECT 1 FROM charges_naive WHERE idem_key = %s", (key,)).fetchone()
    if seen is None:                     # "not charged yet", says every request at once
        conn.execute("INSERT INTO charges_naive (idem_key, amount) VALUES (%s, 2500)", (key,))


def insert_on_conflict(conn, key):
    conn.execute(
        "INSERT INTO charges (idem_key, amount) VALUES (%s, 2500) ON CONFLICT (idem_key) DO NOTHING",
        (key,),
    )


def run(handler, n=20):  # table setup and the six-run loop are left out here
    """Release n identical requests at the same instant, like a double-click times twenty."""
    start = threading.Barrier(n)
    conns = [psycopg.connect(DSN, autocommit=True) for _ in range(n)]

    def worker(conn):
        start.wait()
        handler(conn, "order-42")

    threads = [threading.Thread(target=worker, args=(c,)) for c in conns]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

# Six runs of each, starting from empty tables:
# check-then-insert rows per run: [20, 17, 10, 15, 13, 12]
# ON CONFLICT DO NOTHING rows per run: [1, 1, 1, 1, 1, 1]

Twenty copies of one request created between 10 and 20 rows. The check-then-insert has the same shape as the rate limiter’s read-check-write race: between one request’s read and its write, the others read the same “nothing here yet” and all decide to go ahead. Move the decision into the database, where a unique index can only accept one row, and it’s exactly one every time.

That fixes the table. It doesn’t necessarily fix the customer. Try the three ways of guarding a payment in this demo:

The middle option is the trap. A unique constraint on the payments table, written after the card is charged, stops the second row but not the second charge. By the time the database says no, the money has already moved twice. The constraint has to guard the decision to start the work, not the record that it finished.

Claim the key before doing the work

So the order of operations is: claim, work, record.

import hashlib
import json
import time

from psycopg.types.json import Jsonb


def fingerprint(body):
    return hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()


def post_payment(conn, account, key, body):  # tables: idem (PK account, key) and card_charges
    fp = fingerprint(body)
    # 1. Claim the key before doing anything. Exactly one request can win this insert.
    won = conn.execute(
        "INSERT INTO idem (account, key, fingerprint) VALUES (%s, %s, %s) "
        "ON CONFLICT DO NOTHING RETURNING key", (account, key, fp)).fetchone()
    if not won:
        saved_fp, code, saved = conn.execute(
            "SELECT fingerprint, response_code, response_body FROM idem WHERE account = %s AND key = %s",
            (account, key)).fetchone()
        if saved_fp != fp:
            return 422, {"error": "key reused with a different request"}
        if code is None:
            return 409, {"error": "a request with this key is in progress; retry shortly"}
        return code, saved                                   # finished earlier: replay it
    # 2. Only the winner gets here. The side effect happens once.
    time.sleep(0.05)                                          # the card network takes a while
    charge_id = conn.execute(
        "INSERT INTO card_charges (amount) VALUES (%s) RETURNING id", (body["amount"],)).fetchone()[0]
    response = {"id": f"pay_{charge_id}", "amount": body["amount"], "status": "succeeded"}
    # 3. Save the answer so every later retry gets the same one.
    conn.execute("UPDATE idem SET response_code = 201, response_body = %s WHERE account = %s AND key = %s",
                 (Jsonb(response), account, key))
    return 201, response

# Twenty threads at once, then one retry, then the key with a different amount:
# 20 at once: {409: 19, 201: 1}
# a retry later: (201, {'id': 'pay_1', 'amount': 2500, 'status': 'succeeded'})
# same key, new amount: (422, {'error': 'key reused with a different request'})
# card charges: 1

One request won the claim and did the work. Nineteen were told to come back. The retry got the original answer, the reused key got a 422, and the “card network” saw one charge.

What should the losers get? Two reasonable answers:

• 409 Conflict with Retry-After: 1. The client backs off and retries, and by then the answer is saved. This is what I’d pick for a public API: it’s honest, it holds no resources, and clients already know how to retry.
• Wait for the winner. If the claim and the work share one database transaction, PostgreSQL does this for you: a second INSERT ... ON CONFLICT on a key that an uncommitted transaction is inserting blocks until that transaction commits or rolls back, then sees the outcome. It’s tidy for fast, purely-database work. It’s a bad idea when the work includes a slow external call, because now you’re holding a transaction and a connection open for as long as the card network takes.

What if the winner dies?

The claim has a hole. If the server that won the claim crashes before saving a response, the key says “in progress” forever, and every retry gets a 409 until the end of time.

The fix is a lease: the claim records until when it’s held (locked_until), and a retry that finds an expired lease may take over. That creates a new hazard. If the “dead” winner was only slow, two workers now do the same job. So the lease must be longer than the slowest legitimate run, and the work must be safe to resume: split it into steps and save a recovery point after each, so a takeover continues instead of starting over. Brandur Leach’s “Implementing Stripe-like Idempotency Keys in Postgres” is the best long treatment of this design I know.

My opinion on lease length: err long. A stuck request that takes half an hour to recover is an annoyance. A request that runs twice because the lease was shorter than a slow run is the exact bug you built all this to prevent.

What to save, and when not to

Save the response for anything that happened: success, and failures after the work started, like “card declined”. A retry of a declined payment should get the same decline, not a fresh attempt. Stripe’s documentation says it saves results only once an endpoint has begun executing; a request that failed validation, or conflicted with one running concurrently, saves nothing.

Don’t save failures where nothing happened. If the card network was unreachable, release the claim so the client’s retry can actually run. Saving that 503 against the key would replay a transient failure to every retry, forever.

How long should a key live?

Longer than every retry the sender might make, plus a margin. Stripe’s keys become eligible for removal after at least 24 hours. For Meta’s webhooks, which can be redelivered for up to a week, a 24-hour window would let a late redelivery through as a new message.

• The window is part of your contract. A retry after the key expires is a new request. Document it, as Stripe does.
• Don’t keep keys in a store that evicts. A Redis set to evict under memory pressure drops keys exactly when you’re busiest, which is when retries are most common. Use a table, or a Redis with eviction off and a TTL on every key. At high volume, partition the table by day and drop old partitions: dropping one is instant, deleting millions of rows isn’t.

Scope matters as much as lifetime. Scope keys to one account (or tenant, or API key) so one customer’s key can never replay another customer’s response.

The dual write, and the transactional outbox

Here’s a bug almost every event-driven system ships at some point. An order service saves an order, then publishes an “order placed” event to a broker so the email service can send a confirmation:

db.commit()            # the order is saved
broker.publish(event)  # ...and the process dies before this line runs

Two writes to two systems, with no transaction across both: a dual write. Crash between them and the order exists with no event, so no confirmation is ever sent. Swap the order (publish, then commit) and a rollback leaves an event for an order that doesn’t exist.

The transactional outbox removes the second write from the request. The service writes the event as a row in an outbox table in the same transaction as the order. Either both commit or neither does. A separate relay process reads undispatched rows, publishes them, and marks them dispatched.

The relay itself has a dual write: publish, then mark. If it dies between the two, the next run publishes the same event again. So the outbox gives you at-least-once publishing, and the consumer needs an inbox: a table of event ids it has processed, written in the same transaction as the work.

import json
import sqlite3
import uuid

db = sqlite3.connect(":memory:")
db.executescript("""
    CREATE TABLE orders (id TEXT PRIMARY KEY, total INTEGER);
    CREATE TABLE outbox (id TEXT PRIMARY KEY, type TEXT, payload TEXT, dispatched_at TEXT);
    CREATE TABLE inbox  (event_id TEXT PRIMARY KEY);          -- the consumer's memory
    CREATE TABLE emails (order_id TEXT);                       -- the consumer's side effect
""")


def place_order(order_id, total):
    with db:  # one transaction: the order and its event commit together, or neither does
        db.execute("INSERT INTO orders VALUES (?, ?)", (order_id, total))
        db.execute("INSERT INTO outbox (id, type, payload) VALUES (?, 'order.placed', ?)",
                   (str(uuid.uuid4()), json.dumps({"order_id": order_id})))


broker = []  # stand-in for Redis, SQS or Kafka


def relay(crash_before_marking=False):
    rows = db.execute("SELECT id, type, payload FROM outbox WHERE dispatched_at IS NULL").fetchall()
    for event_id, kind, payload in rows:
        broker.append({"id": event_id, "type": kind, "payload": json.loads(payload)})
        if crash_before_marking:
            return  # published, but the process died before it could say so
        with db:
            db.execute("UPDATE outbox SET dispatched_at = datetime('now') WHERE id = ?", (event_id,))


def consume(event):
    with db:  # record the event id and do the work in the same transaction
        seen = db.execute("INSERT INTO inbox VALUES (?) ON CONFLICT DO NOTHING RETURNING event_id",
                          (event["id"],)).fetchone()
        if seen is None:
            return "duplicate, skipped"
        db.execute("INSERT INTO emails VALUES (?)", (event["payload"]["order_id"],))
        return "sent confirmation"


place_order("ord_1", 2500)
relay(crash_before_marking=True)   # first relay run dies after publishing
relay()                            # the next run finds the row still undispatched and publishes again
print("events on the broker:", len(broker))
print([consume(e) for e in broker])
print("emails:", db.execute("SELECT count(*) FROM emails").fetchone()[0])

# events on the broker: 2
# ['sent confirmation', 'duplicate, skipped']
# emails: 1

(The “email” here is a row, so it can share the inbox’s transaction. A real email can’t, and that’s the next problem.)

Queue consumers: acknowledge late, and expect redelivery

Job queues have the same choice at the consumer. When does the worker tell the broker “I’ve got this one”?

• Acknowledge early, when the worker picks the job up. If the worker is killed halfway, the broker has already forgotten the job. That’s at-most-once. Celery does this by default: a task is acknowledged just before it runs.
• Acknowledge late, after the job finishes. A killed worker means the broker still has the job and gives it to someone else. That’s at-least-once, and it means your task can run twice.

from celery import Celery

app = Celery("shop", broker="redis://localhost:6379/0")
app.conf.broker_transport_options = {"visibility_timeout": 3600}  # seconds; longer than any task


@app.task(acks_late=True)       # acknowledge after the work, so a killed worker's task comes back
def send_receipt(order_id):
    ...                         # which means this body can run twice, so it must be idempotent


print(send_receipt.acks_late, app.conf.broker_transport_options)

# True {'visibility_timeout': 3600}

Redis (as Celery uses it) and Amazon SQS can’t see whether a worker is alive. They use a visibility timeout: a job handed to a worker is hidden for that long, and reappears for another worker if it isn’t acknowledged. SQS defaults to 30 seconds; Celery’s Redis transport to one hour. A job that runs longer than the timeout gets handed to a second worker while the first is still running it. Set the timeout above your slowest job, and make the job idempotent anyway.

One more Celery detail. Even with acks_late, if the worker process dies abruptly (an out-of-memory kill, say), Celery acknowledges the task unless you also set task_reject_on_worker_lost. The docs warn that this can cause message loops, and the loop is real: a job that crashes its worker is requeued, crashes the next one, and so on forever. That’s a poison message; the production section handles one without the loop.

Side effects your database can’t roll back

So far the work lived in your database, where one transaction covers the work and the record of it. The hard cases don’t: charging a card, sending an email or a WhatsApp message, calling an LLM. A rollback doesn’t unsend an email, and these are the duplicates customers notice.

In order of preference:

1. Pass a key downstream. If the provider accepts an idempotency key (Stripe does, many payment APIs do), send one derived from your operation, like refund:order-1042. Now a retry is deduped at the place the money actually moves.
2. Record the provider’s id the moment you have it. Many APIs return an id for the thing they created (a video upload, a container, a message). Save it immediately, outside any transaction that might roll back, and on retry resume that object instead of creating a new one.
3. Write a sent marker. If the provider has no key, record “sent” in a store that survives your rollback (a Redis key, a separate committed row) the instant the provider accepts. Check it before sending again.
4. When you can’t tell, ask a person. If a job died after the send and before any record, you can’t know whether the message went out. For a password reset, sending twice is fine. For “your account has been closed”, it’s better to flag it for a human than to guess.

Here’s the sent marker in miniature:

class SmsProvider:
    """A provider with no idempotency key: every call sends a message."""

    def __init__(self):
        self.sent = []

    def send(self, to, text):
        self.sent.append((to, text))
        return f"msg_{len(self.sent)}"


markers = {}  # stand-in for Redis: written outside the database transaction, on purpose


def send_once(job_id, to, text, provider, crash_after_send=False):
    if job_id in markers:
        return f"skipped: already sent as {markers[job_id]}"
    provider_id = provider.send(to, text)
    markers[job_id] = provider_id          # record it the moment the provider accepts it
    if crash_after_send:
        raise RuntimeError("worker killed before the job could commit")
    return f"sent as {provider_id}"


provider = SmsProvider()
try:
    send_once("otp:42", "+60123456789", "Your code is 481920", provider, crash_after_send=True)
except RuntimeError as exc:
    print("first run:", exc)
print("the queue redelivers:", send_once("otp:42", "+60123456789", "Your code is 481920", provider))
print("messages actually sent:", len(provider.sent))

# first run: worker killed before the job could commit
# the queue redelivers: skipped: already sent as msg_1
# messages actually sent: 1

There’s still a gap: a crash after provider.send returns and before the marker is written. You can make it tiny, not zero, without help from the provider. Say so in design reviews. “At most one duplicate SMS in a rare crash” is a fine answer; “it can’t happen” isn’t.

LLM calls deserve their own note. Their results aren’t idempotent (ask twice, get two answers) and every call costs money. So dedupe before the call, and exit early on a duplicate. A replay that discovers it’s a duplicate after the model call has already paid for it.

CREATE TABLE outbound_emails (
    user_id     uuid NOT NULL,
    kind        text NOT NULL,             -- 'welcome'
    state       text NOT NULL,             -- pending -> sending -> sent
    provider_id text,
    updated_at  timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (user_id, kind)
);

Expert: retries at scale, sagas and the interview

Retry storms and metastable failure

Here’s the failure retry budgets exist for. A service runs at 80% of capacity, comfortably. Something slows it for ten seconds: a long garbage collection pause, a noisy neighbor on the disk. Requests queue and time out. Clients retry, so the queue holds the originals and the retries, so more time out, so there are more retries. The slowdown ends, the service is healthy again, and it stays down.

This is a metastable failure, a term from Bronson, Aghayev, Charapko and Zhu’s “Metastable Failures in Distributed Systems” (HotOS 2021), with a follow-up study of real incidents at OSDI 2022. The system has two stable states, working and not, and a temporary trigger can push it from the first to the second. What keeps it there isn’t the trigger, which is gone. It’s a feedback loop, and retries are the most common one.

Here’s a small model: 80 new requests a second against a capacity of 100, which drops to 40 for ten seconds. Clients time out after 3 seconds and retry up to 3 times. The server can’t see that a client gave up, so it still does the work for abandoned requests:

from collections import deque

NEW_PER_SECOND = 80     # fresh requests from users each second
CAPACITY = 100          # what the server can finish each second when healthy
SLOW = range(10, 20)    # the trigger: ten seconds at 40% capacity (a slow disk, a long GC pause)
TIMEOUT = 3             # clients give up after 3 seconds...
RETRIES = 3             # ...and try again, up to 3 times


def run(budget=False, skip_expired=False, seconds=120):
    """Goodput per second: answers that reached a client who was still waiting."""
    queue = deque()     # (sent_at, request id) in arrival order
    waiting = []        # (deadline, request id, attempt) for requests a client still waits on
    answered = set()
    tokens = 10.0       # retry budget: each fresh request earns 0.1 retry, at most 10 banked
    goodput, next_id = [], 0
    for now in range(seconds):
        # 1. Clients whose request timed out try again, if the budget allows.
        still = []
        for deadline, rid, attempt in waiting:
            if rid in answered:
                continue
            if deadline > now:
                still.append((deadline, rid, attempt))
            elif attempt < RETRIES and (not budget or tokens >= 1):
                tokens -= 1 if budget else 0
                next_id += 1
                queue.append((now, next_id))
                still.append((now + TIMEOUT, next_id, attempt + 1))
        waiting = still
        # 2. New users arrive.
        for _ in range(NEW_PER_SECOND):
            tokens = min(10.0, tokens + 0.1)
            next_id += 1
            queue.append((now, next_id))
            waiting.append((now + TIMEOUT, next_id, 0))
        # 3. The server works through the queue, oldest first.
        capacity = int(CAPACITY * 0.4) if now in SLOW else CAPACITY
        good = 0
        while capacity and queue:
            sent_at, rid = queue.popleft()
            if now - sent_at >= TIMEOUT:      # its client already gave up
                if skip_expired:
                    continue                   # dropped for free: no work spent on it
                capacity -= 1                  # answered to nobody: wasted work
                continue
            capacity -= 1
            answered.add(rid)
            good += 1
        goodput.append(good)
    return goodput


def show(name, g):
    recovered = next((t for t in range(20, len(g)) if all(x >= 75 for x in g[t:t + 5])), None)
    print(f"{name:34} healthy {sum(g[5:10]) // 5}/s, slow {sum(g[10:20]) // 10}/s, "
          f"recovered: {'never' if recovered is None else f'{recovered - 20} s after the trigger ended'}")


show("retry every timeout", run())
show("retry budget (10%)", run(budget=True))
show("retry budget + skip expired work", run(budget=True, skip_expired=True))

# retry every timeout                healthy 80/s, slow 20/s, recovered: never
# retry budget (10%)                 healthy 80/s, slow 20/s, recovered: 20 s after the trigger ended
# retry budget + skip expired work   healthy 80/s, slow 40/s, recovered: 0 s after the trigger ended

With unlimited retries the service never recovers in the two simulated minutes. Once everything times out, each user sends four requests, 320 a second against a capacity of 100, and nearly everything the server finishes is for someone who left. The trigger lasted ten seconds; the outage lasts until someone intervenes. A 10% retry budget caps the extra load and the backlog drains in 20 seconds. Skip work whose client has gone as well, and the service recovers the moment the trigger ends.

It’s a toy, but the shape is real. The defenses, roughly in the order I’d reach for them:

• Retry budgets at the layer that retries, so retries can’t multiply load.
• Deadline propagation. Send the caller’s deadline with the request (gRPC does this natively; over HTTP, a header) and have the server drop work that has already expired, before doing it.
• Load shedding. When the queue is long, answer 503 with Retry-After right away instead of queueing, so the work you do accept finishes in time.
• Circuit breakers on the client, so a dependency that’s failing gets a rest instead of a retry.
• Retry at one layer. The amplification exercise above is a metastable failure waiting for its trigger.

Sagas: idempotency across several services

A booking flow charges a card, books a flight and books a hotel, each in a different service with its own database. There’s no transaction spanning all three. If the hotel fails, the flight and the charge need undoing.

A saga (from Garcia-Molina and Salem’s 1987 paper of that name) is a sequence of local steps, each with a compensation that semantically undoes it: refund the charge, cancel the flight. A saga runner executes the steps in order and, on failure, runs the compensations for completed steps in reverse.

Two properties make a saga survive crashes, and both are idempotency:

• The runner’s progress is durable, so a restarted runner resumes instead of starting over.
• Every step and every compensation carries a key derived from the saga id and the step name, so a step that ran just before the crash isn’t repeated, and an undo that’s retried isn’t applied twice.

class Service:
    """A downstream that dedupes on the key it's given, like a payments API with Idempotency-Key."""

    def __init__(self, name, fails=False):
        self.name, self.fails, self.calls = name, fails, {}

    def do(self, key):
        if self.fails:
            raise RuntimeError("no rooms left")
        self.calls.setdefault(key, "applied")

    undo = do


def run_saga(saga_id, steps, log, crash_after=None):
    """Run steps in order; on failure, undo the finished ones in reverse.
    `log` is durable (a table in real life), so a re-run picks up where the last one died."""
    for name, service in steps:
        if log.get(name) == "done":
            continue                                   # finished before the crash: skip it
        try:
            service.do(key=f"{saga_id}:{name}")
        except RuntimeError as exc:
            for done_name, done_service in reversed(steps):
                if log.get(done_name) == "done":
                    done_service.undo(key=f"{saga_id}:{done_name}:undo")
                    log[done_name] = "undone"
            return f"cancelled: {exc}"
        log[name] = "done"
        if name == crash_after:
            raise SystemExit("worker killed")
    return "booked"


card, flight, hotel = Service("card"), Service("flight"), Service("hotel", fails=True)
steps = [("charge card", card), ("book flight", flight), ("book hotel", hotel)]
log = {}
try:
    run_saga("trip-9", steps, log, crash_after="book flight")
except SystemExit as exc:
    print("first run:", exc, log)
print("second run:", run_saga("trip-9", steps, log))
print("log:", log)
print("card saw:", sorted(card.calls))

# first run: worker killed {'charge card': 'done', 'book flight': 'done'}
# second run: cancelled: no rooms left
# log: {'charge card': 'undone', 'book flight': 'undone'}
# card saw: ['trip-9:charge card', 'trip-9:charge card:undo']

The card service saw one charge and one refund, despite a crash in the middle. Remove the keys and a crash between “charge” and “log it as done” would charge twice on resume.

Sagas aren’t transactions. Between the charge and the refund the customer really was charged, and their bank app shows it. Compensations must also be possible: you can refund a payment but not unsend an email, so put irreversible steps last. Workflow engines like Temporal or AWS Step Functions give you the durable log and the retries; the keys are still your job.

Idempotent isn’t the same as order-safe

Dedupe answers “have I seen this before?” It doesn’t answer “is this newer than what I have?” Those are different questions, and webhooks get both wrong regularly, because most providers don’t promise delivery order.

events = [  # what the payment provider sent, in the order the webhooks happened to arrive
    {"id": "evt_2", "created": 1_700_000_060, "plan": "pro"},      # the newer change
    {"id": "evt_1", "created": 1_700_000_000, "plan": "starter"},  # an older one, retried late
    {"id": "evt_2", "created": 1_700_000_060, "plan": "pro"},      # a plain duplicate
]


def apply_dedupe_only(events):
    seen, state = set(), {}
    for e in events:
        if e["id"] in seen:
            continue
        seen.add(e["id"])
        state["plan"] = e["plan"]
    return state


def apply_with_watermark(events):
    state = {"plan": None, "last_event_at": 0}
    for e in events:
        if e["created"] < state["last_event_at"]:
            continue                       # older than what we already applied: stale
        state["plan"], state["last_event_at"] = e["plan"], e["created"]
    return state


print("dedupe only:   ", apply_dedupe_only(events))
print("with watermark:", apply_with_watermark(events))

# dedupe only:    {'plan': 'starter'}
# with watermark: {'plan': 'pro', 'last_event_at': 1700000060}

Dedupe-only applied each event once and ended on the wrong plan, because the late retry of an older event won. A watermark (the newest event time or version applied so far) refuses anything older. Two other tools help:

• State-carrying events instead of deltas. “The subscription is now pro” can be applied any number of times; “upgrade by one tier” can’t. If the producer sends full state, idempotency comes almost free, and you only need the watermark for order.
• Sequence numbers per entity, when the producer can assign them. Apply event n only if you last applied n − 1; buffer or fetch on a gap.

Every dedupe has a window and a scope

Real systems dedupe for you, within limits. Know the limits, because a duplicate outside them is yours to handle:

The Kafka row needs unpacking. With idempotence on (the default since Kafka 3.0), the broker discards a resent batch it already wrote. But the producer id lasts one session: a restarted producer gets a new id, and its resends aren’t recognised. Kafka transactions, with a stable transactional.id, extend that across restarts and let a read, process and write commit together. That’s Kafka’s exactly-once, and it stops at Kafka’s edge.

SQS teaches the same lesson: five minutes covers a producer’s immediate retry and nothing like a nightly job re-sending yesterday’s messages. That’s why the last row is the one you can’t skip.

The interview: a payments API that must never double charge

“Design a payment API” is a favorite because idempotency is the whole question in disguise. Here’s a structure that works.

1. Requirements. Functional: create a payment for an amount and a payment method, get its status, receive the processor’s result asynchronously, refund. Non-functional: never charge twice for one intent, never lose a payment that was charged, p99 under a second for the API call (excluding the processor), and auditable to the cent. Say the first one out loud; it decides everything else.

2. Estimates. Say 20 million payment requests a day: about 230 a second on average, perhaps five times that at peak. I measured a key row shaped like the Intermediate exercise’s table, with a 237-byte saved response, on PostgreSQL 14 on my laptop: 200,000 rows came to 509 bytes per row including the primary key index. That’s about 10 GB a day, or 71 GB for seven days of keys: one ordinary Postgres server. The key table isn’t the scaling problem; the processor’s latency is.

3. API.

POST /v1/payments
Idempotency-Key: 6f1d...           (required; 400 without it)
{"amount": 2500, "currency": "myr", "payment_method": "pm_123", "order_id": "1042"}

201 Created    a new payment, or the saved answer to an earlier request with this key
409 Conflict   the first request with this key is still running; Retry-After: 1
422            the key was used before with a different body

I’d make the key required on create: it’s cheap for clients and removes a whole class of support tickets.

4. Design.

The request path, in phases, each its own short transaction:

1. Claim. Insert (account, key, fingerprint, state='started', locked_until=now()+lease) with ON CONFLICT DO NOTHING. Lost the insert? Load the row and answer 422, 409 or the replay, exactly as in the Advanced section.
2. Create the payment row as pending, and store its id on the key row. This is the recovery point.
3. Call the processor with no transaction open, passing the payment id as the processor’s own idempotency key. If our server dies and the recovery worker retries, the processor dedupes it.
4. Record the result: update the payment, write ledger entries, save the response on the key row, all in one transaction. The ledger has a unique constraint on (payment_id, entry_kind), so even a bug that records twice can’t book money twice.

5. Deep dives the interviewer is likely to pick:

• The processor times out. Don’t create a new charge. The payment stays pending; the recovery worker asks the processor about that payment id, and the processor’s own key makes a re-send safe.
• Processor webhooks. At least once and out of order: dedupe on the event id and only allow forward transitions (pending → succeeded, never back).
• Stuck keys. A sweep hands keys still started past their lease to the recovery worker, which resumes from the recovery point.
• Reconciliation. Daily, compare the ledger with the processor’s settlement report. Idempotency prevents the duplicates you thought of; reconciliation catches the rest.
• Your own retries. One layer, full jitter, a budget, a circuit breaker.
• Key expiry. Seven days, dropped by daily partition, and documented.

6. What breaks first. The processor’s latency and rate limits, not your database. Say what you’d do: a per-merchant concurrency limit toward the processor, and a queue for non-interactive payments like subscriptions.

In production: idempotency in ezychat.io

ezychat.io receives customer messages from WhatsApp, Instagram and Messenger, replies with AI and people, bills through Stripe and publishes posts to social platforms. Nearly every interesting path in it starts with “a provider sent this, and might send it again”. Here’s how the code deals with that, quoted from the source.

Webhooks: the provider’s id is the key

Every inbound message is stored through one function, with Meta’s own message id as the dedupe key, scoped to the organization:

def insert_inbound_message_dedup(session, *, org_id, conversation_id, content_type,
                                 content, raw_payload, channel_msg_id) -> uuid.UUID | None:
    """Insert an inbound Message, short-circuiting cleanly on a duplicate
    `(org_id, channel_msg_id)` (a provider retry). Returns the new message
    id, or None if this delivery was already recorded."""
    stmt = (
        pg_insert(Message)
        .values(
            id=uuid.uuid4(),
            org_id=org_id,
            channel_msg_id=channel_msg_id,
            # ...and the other columns
        )
        # Matches the partial unique index (WHERE channel_msg_id IS NOT NULL,
        # migration 0006) exactly — Postgres requires the ON CONFLICT target
        # to match a partial index's predicate, not just its columns.
        .on_conflict_do_nothing(
            index_elements=["org_id", "channel_msg_id"],
            index_where=Message.channel_msg_id.isnot(None),
        )
        .returning(Message.id)
    )
    return session.execute(stmt).scalar_one_or_none()

The index is partial because an outbound message has no provider id until it’s sent, and the comment records a PostgreSQL detail that bites people: ON CONFLICT must name the partial index’s predicate too.

What happens after the insert matters as much. The ingest code returns on None before anything else runs, and billing sits after that return on purpose:

if message_id is None:
    return  # duplicate delivery — already recorded, short-circuit cleanly
...
# Exact-MAC metering (billing v2.1): first inbound message this month
# marks the contact active. Sits after the dedup short-circuit so a
# provider retry never re-meters; never raises, never blocks ingestion.
billing_metering.note_active_contact(session, org_id=org_id, contact_id=contact_id)

That’s the Intermediate formula in practice. The handbook states the result: a retried delivery “produces exactly one messages row and exactly one message.received event, no matter how many times Meta redelivers it.” Facebook and Instagram comments use the same pattern on (org_id, platform_comment_id).

Deal rooms, which bring WhatsApp group chats in through a separate gateway, add two refinements. The gateway retries until it gets a 2xx, which the code treats as the normal case, not an edge case:

if new_id is None:
    # A redelivery. The gateway retries until it gets a 2xx, so this is the
    # normal path for any event whose first delivery timed out after the
    # write.
    return None

And a photo is expensive to process, so a cheap lookup runs first, with the unique insert still the real guard:

# The insert below is the real dedup guard, but a redelivery has to be
# caught *before* the media fetch: otherwise every retry of a photo costs a
# gateway round-trip and an S3 write whose object is then orphaned by the
# ON CONFLICT. One indexed lookup against the same key.

That’s the right way to use check-then-insert: as an optimization in front of a constraint, never instead of one. The same file handles order too: the room’s last-activity time moves “only forwards”, so a retry draining late can’t re-sort the inbox.

Find-or-create works the same way: conversations and contact identities are inserted with ON CONFLICT DO NOTHING ... RETURNING, and the loser of a race reads the winner’s row. The deal-room docstring says why: “two messages arriving in the same second from a group nobody has written to yet would otherwise each create a conversation, and only one of them would ever be shown again.”

The inbox: write it down, then process it

The Meta webhook handler does almost nothing, deliberately. It caps the body size, verifies the signature over the raw bytes, redacts credentials, inserts the raw payload into webhook_logs and enqueues a task. Then it answers 200. webhook_logs is an inbox: once a body is in it, ezychat owns it.

The task acknowledges late, and the comment above it explains both the choice and the setting it doesn’t use:

# Acknowledged only after the task finishes, so a worker stopped mid-task
# (a deploy, a container kill) leaves the message to be redelivered rather
# than gone for good. Safe because ingestion is idempotent: every write
# dedupes on Meta's own ids, the property Meta's own redeliveries rely on.
#
# Deliberately *not* `reject_on_worker_lost`: a delivery that kills its
# worker process (out of memory, say) would be requeued at once and kill
# the next one, forever. The replay sweep below re-enqueues such a log
# within minutes instead, and abandons it after three attempts.
@celery_app.task(name=PROCESS_WEBHOOK_LOG_TASK_NAME, acks_late=True)
def process_webhook_log(log_id: str) -> None:

That’s the poison-message problem, solved with a counter instead of a loop. A replay sweep runs every two minutes and re-enqueues logs still unprocessed after two minutes. It skips a log whose task is still running, and while the queue has a backlog it skips anything younger than 15 minutes, because that log is “late, not lost”. A log replayed three times without finishing is marked abandoned and logged at CRITICAL, and the age of the oldest unprocessed log goes to a CloudWatch alarm.

webhook_logs itself is deliberately not deduplicated: every signed POST gets its own row, retries included. The log is the audit trail and the message table is the dedupe boundary. One place for dedupe, and knowing which place, keeps the rest simple.

The outbox

Every domain event (a message received, a reply sent) is a row added to the caller’s transaction. The docstring of emit_event describes the outbox exactly: “the event rides the caller’s transaction, committing with the action that produced it and vanishing if that action rolls back.” A post-commit hook nudges a relay task, and the relay publishes to Redis pub/sub and marks rows dispatched in the same transaction. Its docstring calls that idempotent: “a retry finds nothing left to do once that transaction has committed.”

The relay’s failure handling is the clearest statement of the at-least-once trade I’ve seen in a codebase:

except redis.RedisError:
    # The rows published before this point stay published, and the
    # transaction rolls back, so they will be published again by
    # whoever picks the batch up next. Duplicate frames are the
    # correct trade against losing one: the client applies an event
    # it already has, where a lost event is invisible forever.

A missed nudge is caught by a sweep every minute that re-enqueues orgs with events undispatched for over 120 seconds, and logs at WARNING when it does: “a quiet sweep is the healthy case; a noisy one is a signal to go and look at why the nudge is failing.” The oldest undispatched event’s age goes to CloudWatch as OutboxLagSeconds.

Owed replies: compare-and-clear, a replay token and a sent marker

The AI reply pipeline has its own production note, No lost turns, and it’s built from this article’s patterns. When a message makes a reply due, the same transaction stamps the conversation’s ai_pending_since. A turn reads that value at the start and settles exactly that debt:

def _turn_settled(org_id, conversation_id, owed_since):
    """Clear the owed turn this turn read at its start — and only that one.
    A message landing mid-turn has written a newer value, which this leaves
    in place for its own turn (or the sweep) to answer."""
    return (
        update(Conversation)
        .where(
            Conversation.id == conversation_id,
            Conversation.org_id == org_id,
            Conversation.ai_pending_since == owed_since,
        )
        .values(ai_pending_since=None)
    )

That’s compare-and-set with a timestamp as the version. A sweep replays turns owed too long, carrying the owed timestamp as a token, and a replay that finds it changed exits with already_handled before the model is called: dedupe before the LLM call.

The send has two guards. A Redis lock (SET NX, 30-second expiry) stops two turns sending at once, failing open because “the DB re-checks below stay authoritative”. And once the provider accepts a reply, a sent marker is written outside the transaction:

def sent_marker_key(conversation_id):
    """When the AI last got a message accepted by the provider for this
    conversation, as epoch seconds. Written outside the turn's transaction,
    so it survives the rollback of a turn that sent and then failed before
    committing — the one case where replaying the turn would message the
    customer twice. `ai.tasks.sweep_unanswered_turns` reads it for exactly
    that."""
    return f"ai:sent:{conversation_id}"

If the marker shows a reply went out after the turn became owed, the sweep hands the conversation to a person instead of replaying. That’s option 4 from the side-effects list: when you can’t be sure, don’t guess.

Money: one ledger row per key

The prepaid wallet credits money from Stripe webhooks, and only from webhooks; the module docstring is explicit that credit is “never from a success page, which anyone can navigate to.” Every credit carries a key derived from a Stripe object id (invoice_{id}, autotopup_{payment intent id}, dispute_{id}, or the checkout session’s own top-up key), and the ledger enforces it:

class CreditLedgerEntry(Base):
    """One money movement, signed, in sen. Append-only: corrections are
    new entries (`adjustment`), never edits. `idempotency_key` is the
    exactly-once anchor for webhook-driven credits and per-request
    deductions alike — org-scoped, since every caller derives it from an
    id that is already globally unique."""

    __table_args__ = (
        UniqueConstraint("org_id", "idempotency_key", name="uq_credit_ledger_idempotency"),
        ...
    )

credit() locks the org’s wallet row (FOR UPDATE, which “is what serializes the ledger”), returns the existing entry if the key is there, and appends otherwise. The lock makes concurrent duplicates wait; the constraint backs it up. The checkout session’s top-up key is also sent to Stripe as its Idempotency-Key, so the dedupe reaches the place the money moves.

There’s a retry decision here too, and it’s a product one. Auto top-up makes one charge attempt per threshold crossing and never retries a decline: “a silent retry loop against a declining card is how a customer discovers a wall of bank alerts and turns the feature off forever.” Not every failure should be retried, even safely.

Stripe subscriptions: idempotent by construction

Subscription events take a different route to the same place. The webhook module’s docstring:

Mirroring is declarative: customer.subscription.* events carry the full subscription state, so we set the plan and reconcile org_addons to match the event’s item list every time. That makes replays idempotent by construction (same event twice -> same DB state -> no version bump, because the service mutations only bump when a field actually changes) with no dedup table.

That’s the “state-carrying events” idea from Expert. And because full-state events are only safe in order, there’s a watermark. The comment on it explains a subtlety that’s rarely written down: Stripe re-signs a retry when it sends it, so the signature’s timestamp check can’t tell you how old the content is.

if event_at is not None:
    if sub.last_stripe_event_at is not None and event_at < sub.last_stripe_event_at:
        return "ignored: stale event superseded by a newer one"
    if sub.last_stripe_event_at is None or event_at > sub.last_stripe_event_at:
        sub.last_stripe_event_at = event_at

It’s “strictly older only”, because Stripe’s timestamps have one-second granularity and two events in the same second are treated as concurrent, serialized by a row lock rather than dropped. A daily reconcile, which reads Stripe’s current state instead of an event, is deliberately exempt from the watermark: it’s the repair for a divergence, so it must never be blocked by one.

Publishing: a claim gate, a resume pointer and patience

Publishing a post to Facebook or Instagram runs as one Celery task per target. Each task claims its row under a lock, and the claim is the idempotency gate:

# Idempotency gate: a fresh delivery (`pending`) always proceeds; an
# in-flight row (`publishing`) only proceeds when this is the task's
# OWN Celery retry re-entering its own row — any other re-delivery of
# an already-claimed or already-terminal row is a safe no-op.
can_claim = target.status == "pending" or (
    target.status == "publishing" and self.request.retries > 0
)

The adapter call then runs with no database session open, and the finalize steps only touch a row still in publishing (“already finalized by a redelivered task run — no-op”). Video uploads are the side effect that can’t be rolled back, so the moment Meta returns a container or video id it’s saved as provider_upload_ref, “so a retried or recovery-swept task resumes polling the existing container/video instead of re-uploading — the idempotency guard against double-publishing a video.” That’s option 2 from the side-effects list.

The retry policy is two budgets on one counter: transient failures retry five times from 10 seconds, doubling, and Meta’s “still processing” answers get twelve gentler retries from 30 seconds, capped at five minutes. There’s no jitter; each task retries one target on its own schedule, but it’s the first thing I’d add if many targets began failing together. And the threshold for a target stuck in publishing is the lease-length argument from Advanced, settled in a comment:

# A target idle in `publishing` this long means the worker that claimed it
# died before finalizing. Must exceed the LONGEST legitimate in-flight
# window, or the reset spawns a second concurrent publish of the same
# target — the DB converges either way (finalize guards on status), but the
# platform could receive a duplicate post. ... 30 min doubles that worst
# case (covering clock skew between workers too); slower crash recovery is
# the right trade against any chance of double-publishing.
_STUCK_TARGET_THRESHOLD = timedelta(minutes=30)

Every attempt is also recorded in an append-only post_target_attempts table, unique on (post_target_id, attempt_no), numbered by counting existing rows rather than from Celery’s retry counter, which resets on a recovery-swept re-enqueue. The media probe task, which runs ffprobe on uploaded videos, uses the same claim gate on its own status column.

AI tool calls: a key made from the request

When the AI books an appointment through a tool call, the booking key is built from what’s being booked:

idempotency_key = f"ai:{conversation_id}:{service.id}:{requested_start.isoformat()}"

Bookings have a unique index on (org_id, idempotency_key), and the create function returns the existing booking if the key is already there. A model that calls the tool twice for the same slot, or a turn that’s replayed, gets the existing booking back, and the comment above that check spells out what it must not do again: re-sync the calendar or re-send the “you’re booked” confirmation. Manual bookings from the app require the client to send a key (idempotency_key: str = Field(min_length=1, max_length=200)), and public booking-page confirmations derive theirs from the hold being confirmed (public:{hold.id}).

Small in-flight guards

Not every duplicate needs a table. Starting a new WhatsApp chat to a number sends a template message, and a double-click (or two agents at once) would send two. A 10-second Redis guard (SET NX EX) covers it, and its docstring is careful about what it is: “a duplicate-send mitigation, not a security control”. It fails open if Redis is down, because “the residual risk on a Redis outage is one duplicate template message, which is benign.” And it’s released on a failed send, “so a corrected retry isn’t blocked for the rest of the TTL”: the “don’t save failures where nothing happened” rule, at the scale of one key.

What I’d copy

• Dedupe on the sender’s id, in a unique index, scoped to the tenant. Then put everything that costs money or sends something after the insert’s “I was first” answer.
• Make the inbox and the outbox boring. Write the raw input first, write events in the business transaction, and back both with a sweep that’s loud when it finds work.
• Acknowledge late, and cap replays. Late acknowledgement without a replay limit trades lost messages for a poison-message loop.
• For side effects outside the database, save the provider’s id or a sent marker the moment you have it, and hand the ambiguous cases to a person.
• Choose lease lengths and retry counts in comments, with the arithmetic. The next person will want to shorten them, and the comment is what stops them.

Cheat sheet