Idempotent Payment APIs: Exactly-Once Settlement Semantics

Every payment API sits on top of an unreliable network. A client submits a charge, the server debits an account and returns a 200, but the response is lost to a timeout, a proxy restart, or a mobile connection drop. The client, following completely correct retry logic, submits the same request again. If the server treats that second request as a new payment, the customer is charged twice. This is not an edge case — at any meaningful transaction volume, some non-trivial fraction of requests will be retried, and the API has to make retries safe by construction rather than by hoping clients behave.

The naive fix — 'just check if a similar payment happened recently' — does not work. Two legitimate purchases of the same amount, seconds apart, are not duplicates. The system needs a way to distinguish 'the client meant to do this twice' from 'the client is retrying the same intent,' and that distinction has to be made by the client, not inferred by the server.

The constraint set is therefore: the API must be safe to retry an unbounded number of times with identical results (idempotent), it must commit funds movement and its own bookkeeping atomically (no partial states where money moved but the ledger doesn't know), and it must do this under crashes at any point in the request lifecycle — including crashing after debiting an upstream rail but before writing a local record of having done so.

The standard mechanism, popularized by Stripe and now near-universal, is a client-supplied idempotency key: an opaque string the client generates once per logical operation and attaches to every retry of that operation. The server treats the (API key, idempotency key) pair as the true identity of the request, not the payload, not the timestamp.

The data model needs two things bound together in the same transaction boundary: an idempotency record and a payment intent. The idempotency record stores the key, a hash of the request body (to detect a client reusing a key for a different payload, which is a client bug and should be rejected, not silently accepted), the current processing state (received, in-progress, succeeded, failed), and — once resolved — the exact response body to replay verbatim on future hits. The payment intent is a state machine (created, authorized, captured, settled, failed, reversed) with its own row, and it is this row, not the idempotency record, that ever touches the ledger.

Critically, the idempotency record and the first ledger entry it produces must be written in the same atomic unit — typically the same PostgreSQL transaction, using a unique constraint on the key as the concurrency guard. If a second request with the same key arrives while the first is still in-flight, the unique constraint (or a SELECT ... FOR UPDATE on the key row) causes it to either block until the first resolves or fail fast with a 409, depending on how much latency the API is willing to impose on concurrent duplicate submissions. What it must never do is let both requests proceed independently to the point of creating two ledger entries.

A clean write path separates 'record intent' from 'move money' from 'tell the caller.' On request arrival, the server opens a transaction, attempts to insert the idempotency key row with state in-progress; if it already exists and is succeeded, it short-circuits and returns the stored response immediately, without touching the ledger again. If it exists and is still in-progress, that's a concurrent duplicate — reject or park it, don't proceed.

On a fresh key, the server creates the payment intent and the double-entry ledger postings (debit the source account, credit a suspense or destination account) inside that same transaction, then commits. Only after that commit succeeds does the server call out to any external rail — a card network, an ACH gateway, a real-time payments scheme. External calls should never happen inside the database transaction; they're slow, they can hang, and holding a row lock across a network call to a third party is how you turn one degraded dependency into a full outage.

This means the local commit and the external settlement are two separate steps, and the gap between them is exactly where an outbox pattern earns its keep: the local transaction that commits the ledger entry also inserts an outbox event in the same transaction, and a separate worker polls the outbox and drives the external call, updating the payment intent's state as responses come back. The idempotency key travels with that outbox event too, so if the worker crashes and retries, the external call itself is either naturally idempotent (many payment rails support a client-supplied reference id for exactly this purpose) or wrapped in its own dedupe check against the rail's API.

The hardest failure mode is not the database crashing — Postgres transactions handle that correctly by definition — it's the process crashing after the external rail has accepted the payment but before the local system records that success. This is where naive 'call the API, then write local state' designs break: on restart, the worker doesn't know if the external call happened, and calling it again risks a genuine double-execution on a system the API doesn't control.

The mitigation is to make the external call idempotent from the caller's side whenever the rail supports it (a client reference field that the rail itself deduplicates on), and where it doesn't, to move to a reconciliation model: after any ambiguous outcome — timeout, 5xx, connection reset — the worker doesn't retry blindly, it first queries the rail for the status of that reference id, and only submits fresh if the rail confirms it never received the original. This changes the correctness argument from 'we never send twice' (unenforceable across a network boundary you don't own) to 'we never believe we sent twice when we didn't, and we always find out the truth before acting again.'

On the read side, capture and settlement events arriving asynchronously from the rail (webhooks, batch settlement files) go through the same idempotency discipline: each inbound event carries or is assigned a dedupe key, checked against a processed-events table before it's allowed to transition the payment intent's state machine. State transitions themselves should be guarded — a settlement event for an intent that's already settled is a no-op, not an error, because at-least-once delivery from the rail is the norm, not the exception.

Idempotency keys need a retention and expiry policy — typically 24 hours to a few days is enough to cover realistic client retry windows, after which the record can be archived or dropped, keeping the hot table small and its unique-constraint lookups fast. The idempotency table is a natural hot spot under load, so it benefits from being partitioned or sharded by a prefix of the key (many implementations embed a client or account identifier in the key itself for exactly this reason), keeping lock contention local rather than global.

Observability has to treat 'duplicate detected and short-circuited' as a distinct, expected signal, not an error — a dashboard that alerts on idempotency-hit rate crossing an unusual threshold is often the earliest warning of a client-side retry storm or a broken timeout configuration upstream, well before it shows up as user complaints. Reconciliation jobs that compare the internal ledger against rail-reported settlement files on a schedule are the backstop for everything above: idempotency prevents most double-processing at the API boundary, but a reconciliation pass is what catches the rare case where the boundary discipline was itself violated, whether by a bug, an operator running a manual script, or a rail-side anomaly.

We would not rely on client-side deduplication alone — 'the client promises not to double-submit' is not a safety property, it's a hope, and every real payment system eventually meets a client that violates it, intentionally or not. We would not use the request payload itself as the idempotency identity without an explicit client-supplied key; payload hashing alone can't distinguish 'same request, resubmitted' from 'coincidentally identical request, submitted twice on purpose,' and getting that distinction wrong in either direction is a real-money bug.

We would also not try to make idempotency guarantees span all the way through a third-party rail we don't operate, by design — that boundary is inherently at-least-once, and pretending otherwise just hides the reconciliation work instead of doing it. And we would not build the idempotency layer as a generic cross-cutting middleware that's blind to payment semantics: a naive 'cache the HTTP response for N hours' approach fails the moment a payment intent legitimately transitions state after the first response was cached (authorized to captured, for instance), because it will happily replay a stale response instead of reflecting current truth. The idempotency layer has to understand the state machine it's protecting, not just the wire format around it.