Idempotency Keys: Exactly-Once Payments Across Retries

Here is the scenario that keeps payments engineers up at night: you POST a payment, the server creates it, and then the network drops the response before it reaches you. You don't know if it worked. You retry. Now you've sent the money twice. Idempotency keys exist to make that retry safe — so that submitting the same payment request twice creates one payment, not two.

A required key, not an optional one

On KibiPay, the Idempotency-Key header is mandatory on any state-changing request. A middleware requires it on every POST, PUT, PATCH and DELETE (health checks and internal admin routes aside), and a request that omits it is rejected outright with 400 idempotency_required. The payment-creation route enforces the same rule independently, belt and braces. Making the key required rather than optional is a deliberate choice: exactly-once semantics only work if every caller participates, and the surest way to guarantee that is to refuse requests that don't.

The database is the source of truth

It would be fragile to enforce uniqueness in application code with a "check then insert" — two concurrent retries could both check, both find nothing, and both insert. Instead, KibiPay leans on the database. The payments table carries a unique constraint on the pair (tenant_id, idempotency_key). Uniqueness is scoped per tenant, so two different tenants can happen to use the same key string without colliding, but within a tenant a key identifies exactly one payment.

The write itself uses PostgreSQL's INSERT ... ON CONFLICT DO NOTHING on that constraint, followed by a select of the row for (tenant_id, idempotency_key). So the logic is: try to insert the new payment; if a row with this key already exists, the insert quietly does nothing; then read back whichever row is there — the one you just inserted, or the one that was already there — and return it. A duplicate POST therefore returns the original payment's id and details, indistinguishable from the first call, with no second payment created. The race is resolved by the database's atomicity, not by hopeful application checks.

Idempotent all the way down

Creating the payment row is only half the story — a payment also kicks off a workflow to actually move the money. That start is made idempotent too: if the workflow for this payment has already been started, the "already started" error is swallowed rather than surfaced, so a re-POST doesn't launch a second execution. The exactly-once guarantee holds not just for the database record but for the side effect it triggers.

Batches get keys too

Batch submission has an interesting wrinkle. When you POST a batch of up to a hundred payments, the outer Idempotency-Key is the key for the batch as a whole, but each element inside needs its own key or a retry of the batch couldn't safely re-run. The platform derives one per element by suffixing the batch key with the element's index — {batch_key}::{i}. So a retried batch re-inserts each child under its own stable key, and the same insert-or-return logic makes the whole batch replay-safe, element by element.

Why this is the right model

Idempotency keys put the responsibility in the right place. The client, which knows whether two requests are "the same payment" or "two different payments that happen to look alike," supplies the key that says so. The server, which owns the data, enforces uniqueness atomically. Neither has to reason about the network's failure modes — the timeout, the dropped response, the ambiguous error — because the contract is simply: same key, same payment, every time. That's what lets a client retry aggressively, which is exactly what you want a client to do when money is involved and the answer is unclear.

The failure modes it neutralizes

The value of idempotency is easiest to see by enumerating the ambiguities it erases. A request times out — did it land? Retry safely; same key, same payment. A response is lost to a dropped connection — did the payment create? Retry; you get the original back. A client crashes mid-submit and replays its queue on restart — every duplicate collapses to one. A load balancer double-delivers a request — the database's conflict handling makes the second a no-op. In every one of these cases, without an idempotency key the safe-looking action (retry) is the dangerous one (double-pay), and the unsafe-looking action (don't retry) risks losing a legitimate payment. The key dissolves the dilemma: the client can always retry, because the contract guarantees that "the same payment" is defined by the key it chose, not by how many times the bytes reached the server. That's why the header is mandatory rather than best-effort — exactly-once is a property of the whole system only if every writer opts in, and requiring the key is how the platform guarantees they do.