Idempotency is the only distributed systems concept you need

The network will deliver your message twice. Or zero times. Or once, but you will not find out, so you will send it again.

This is not an edge case. It is the normal operating condition of every distributed system, and idempotency is what makes it survivable.

the fundamental problem#

You send a request. The connection times out.

Did it succeed?

You do not know. There are three possibilities:

1. The request never arrived. Retry is correct.
2. The request arrived and failed. Retry is correct.
3. The request arrived, succeeded, and the response was lost. Retry charges the card twice.

You cannot distinguish these from the client. This is not a limitation of your tooling; it is a theorem about asynchronous networks. There is no protocol that resolves it.

The only resolution is to make retrying safe.

what idempotency means#

An operation is idempotent if performing it multiple times has the same effect as performing it once.

Naturally idempotent:

• PUT /user/123 {"name": "Dom"} — set to a value.
• DELETE /user/123 — the second one finds nothing to do.
• SET balance = 100 — absolute assignment.

Not idempotent:

• POST /orders — creates a new one each time.
• UPDATE accounts SET balance = balance + 100 — relative change.
• Sending an email.
• Incrementing a counter.

the pattern#

For operations that are not naturally idempotent, the client supplies a key and the server remembers it.

POST /payments
Idempotency-Key: 8f14e45f-ea5c-4b0d-9c1a-2f7e3d4a5b6c

{"amount_cents": 4200, "currency": "usd", "source": "card_x"}

Server side:

def create_payment(key, request):
    with transaction():
        existing = lookup(key)
        if existing:
            if existing.request_hash != hash(request):
                raise Conflict("key reused with different parameters")
            return existing.response      # replay, do not re-execute

        result = charge_the_card(request)
        store(key, hash(request), result)
        return result

Four details that matter and are usually missed:

Store the result, not just the key. A retry should return the original response, not a "already processed" error. The client's retry should look like a successful first attempt.

Hash the request. If the same key arrives with different parameters, that is a client bug and you should say so rather than silently returning the wrong result.

Same transaction. The lookup, the work, and the store must be atomic. Otherwise two concurrent retries both find nothing and both execute.

Expire the keys. Days, not forever. Storage is not free and a key from a year ago is not going to be retried.

where else this applies#

Message consumers. At-least-once delivery guarantees duplicates. Same pattern: an idempotency key on the message, a record of processed keys, both in one transaction.

Webhooks you send. Include an event ID. Your receivers will need it, and if you do not provide one they will invent something worse.

Webhooks you receive. Assume duplicates. Every major provider retries and several will send the same event twice under normal operation.

Deployment and provisioning. "Create this resource" should succeed if it already exists in the right state. This is why declarative infrastructure tools work and imperative scripts do not.

Migrations and backfills. They get interrupted. They must be safe to re-run from the beginning.

the design advice#

Prefer absolute over relative. SET balance = 100 is idempotent. ADD 100 is not. When you have the choice, take the absolute form.

Where you cannot — and you often cannot, because concurrent updates need relative operations — use a version or a compare-and-set:

UPDATE accounts SET balance = balance + 100, version = version + 1
WHERE id = $1 AND version = $2;

Zero rows updated means someone else got there first, and you know it.

Let the client generate the key. The client knows whether this is a new request or a retry. The server cannot tell. A server-generated key defeats the entire purpose.

Document it. If your API supports idempotency keys, say so prominently, say how long they are honored, and say what happens on key reuse with different parameters. Consumers who do not know about the mechanism will not use it, and then they will double-charge someone and it will be your incident too.

why this is the one to internalize#

Most distributed systems failures are one of: a duplicate, a message lost, or an out-of-order arrival.

Idempotency makes duplicates safe. Retries make loss survivable — and retries require idempotency to be safe. Which leaves ordering, which is a narrower problem that fewer systems actually have.

So: one property, correctly implemented, defuses the majority of what goes wrong. That is a very good ratio, and it is why this is the thing to get right before anything else.