Async vs Sync Standbys: Choosing Between RPO Zero and Never Blocking a Payment
When you run a Postgres primary with a standby, you make one quietly consequential choice: does the primary wait for the standby to confirm each write before telling the client "done"? That single flag is the difference between zero data loss and never freezing a payment — and you usually can't have both. KibiPay's standby is asynchronous, and that is a deliberate decision, not a default we forgot to change.
What synchronous replication buys you
With synchronous replication, the primary holds each commit until the standby acknowledges it has the write-ahead log record. The payoff is a Recovery Point Objective of zero: if the primary dies, the standby is guaranteed to have every committed transaction. Nothing acknowledged to a client can be lost. For some systems, that guarantee is non-negotiable.
What it costs you
The cost is coupling. If the primary must wait for the standby, then the standby's health becomes the primary's health. A standby that is slow, network-partitioned, restarting, or simply gone will make every commit hang — because the primary is faithfully waiting for an acknowledgement that isn't coming. In a payments system that means a standby hiccup can stall every single payment write. You traded one failure mode (rare data loss on primary death) for another (write availability now depends on two nodes instead of one).
Why we chose async
KibiPay's cluster is configured with zero synchronous replicas — replication is asynchronous. The primary commits locally and ships the WAL to the standby right after, without waiting. The consequences, stated plainly:
- Writes never block on the standby. The standby can lag, restart, or fail entirely and payments keep flowing. Write availability depends only on the primary.
- RPO is seconds, not zero. If the primary is lost at the wrong instant, the handful of transactions that hadn't yet streamed to the standby could be lost. In practice that window is small and bounded by replication lag.
For a payments platform, that is the right side of the trade. A stalled payment rail is an immediate, visible, customer-facing outage. A theoretical few-seconds RPO on the rare event of a primary node loss is a risk you manage — and it is exactly what backups exist to bound.
The knob is right there
None of this is baked in. The choice is one setting: the number of required synchronous replicas. Set it to one and you get synchronous quorum and RPO zero — and you accept that a sync-standby outage stalls writes. That would be a separate, conscious decision with its own operational weight, not a flip-of-a-switch. We keep it at zero and cover the residual risk elsewhere.
Async plus backups, not async instead of backups
The standby protects against a node dying. It does not protect against a bad migration, a fat-fingered delete, or corruption — those replicate faithfully to the standby too. That is why the same cluster ships continuous WAL archiving and nightly base backups for point-in-time recovery. The async standby handles fast failover; backups handle "undo the last hour." You want both, and treating them as substitutes is how people lose data.
Zero data loss and never blocking a write pull in opposite directions. We picked "never block a payment" and bounded the rest with backups — the honest choice for a system whose primary job is to keep money moving.
What a failover actually looks like
It helps to picture the failure the async choice is tuned for. A node hosting the primary dies. Because the standby has been streaming the write-ahead log continuously, it is at most seconds behind, and the cluster promotes it to primary. Writes resume against the new primary; the read endpoint follows along. The handful of transactions that had committed on the old primary but hadn't yet streamed across — the few seconds of activity in flight at the instant of death — are the RPO cost, and that's the window backups exist to bound rather than replication. Contrast the synchronous world: there, the failover loses nothing, but the price was paid every single day beforehand, because every commit waited on the standby and any standby wobble stalled payments. The async design pays nothing in the common case and a small, bounded amount in the rare one. For a system whose primary obligation is to keep money moving, a rare few-seconds RPO is a far better trade than a daily exposure to write stalls — and pairing it with continuous WAL archiving means even that small window is recoverable.