The Reconciliation Engine and Its Breaks Queue
In a perfect world, every payment you submit gets a prompt, reliable callback telling you what happened, and your books always match the scheme's. In the real world callbacks get lost, adapters miss a webhook, and positions quietly drift. KibiPay's reconciliation engine exists for that real world: it periodically sweeps for payments and positions that have fallen out of sync, and turns each discrepancy into a tracked break an operator can work.
The sweep
The engine's core is a sweep, exposed at POST /reconciliation/sweep and safe to run on a schedule. It has three knobs: a grace window (older_than_minutes, default 15), a batch limit (default 500), and an auto_apply flag (default on). The grace window matters — you don't want to reconcile a payment that's simply thirty seconds into its normal lifecycle, so the sweep only considers payments older than the window that are still in a non-terminal state: SUBMITTED, ACCEPTED, PENDING, VALIDATED or RESERVED (or any payment explicitly flagged as needing reconciliation). Anything already marked reconciled is skipped.
For each candidate, the engine asks the rail's adapter what it thinks the truth is, via that adapter's status-enquiry endpoint, and compares it to our record. What it does with the answer is the subject of break kinds and auto-apply versus human review. Here we focus on how the findings are recorded.
The breaks queue
Every discrepancy the sweep finds is written to a recon_break table — the operator's queue. Each break carries what you'd need to investigate it: the tenant and payment it concerns, the scheme, its kind, our status versus the scheme's status, an amount and currency, and a free-form detail blob. It starts life OPEN.
Deduplication is the whole trick
The feature that makes the queue usable rather than a firehose is the dedupe key. Every break has a unique key derived from its kind and subject — for example STATUS_MISMATCH:{payment_id} or POSITION_DRIFT:{rail}:{currency}. Writes go through an upsert: insert the break, or on a key conflict just bump its last_seen_at and refresh its details. Run the sweep every five minutes for an hour and a persistently stuck payment produces one break row that keeps getting touched, not twelve identical ones. Operators see a stable list of distinct problems, each with a "first seen" and "last seen" that tells them how long it's been unhappy.
Closures are sticky
There's a subtle but important rule in the upsert: it never reopens a break that's already been resolved. Once an operator (or the auto-apply path) closes a break, a later sweep that stumbles across the same key won't drag it back to OPEN. Resolution is sticky, so working the queue actually shrinks it.
Working the queue
Operators interact with the queue through a small set of endpoints:
GET /reconciliation/breaks— the list, ordered open-first then most-recently-seen, filterable by status, tenant, scheme and kind.GET /reconciliation/breaks/summary— counts by status and open counts by kind, the numbers you'd put on a dashboard.POST /reconciliation/breaks/{id}— update a break, moving it to a terminal state ofRESOLVEDorWRITTEN_OFFwith a resolution note and a stamped resolved-at time.
Those two terminal states encode a real operational distinction: RESOLVED means "we fixed it and the books now agree," while WRITTEN_OFF means "we've accepted this discrepancy and are closing it deliberately." Both get the break out of the open queue; only one implies the underlying truth changed.
Two things it reconciles
The engine works along two axes at once: payment-versus-scheme (is this specific payment's status what the rail says it is?) and position-versus-external (does our ledger position for a rail and currency match the outside world?). Those are genuinely different questions with different remedies, and the platform keeps them distinct — a separation covered in ledger-vs-scheme versus ledger-vs-external. What unifies them is this queue: whatever the axis, a discrepancy becomes a deduplicated, trackable, closeable break.
Safe to run on a schedule
Everything about the sweep is built to be run repeatedly without harm. The candidate query is bounded — a default batch of five hundred, capped well below any runaway size — so a single sweep can't try to reconcile the entire backlog at once. Each run returns a small counts object (scanned, plus a tally of status-mismatches, stuck, unreconcilable, auto-resolved and position-drift breaks), which is exactly the shape you'd emit as metrics to watch the engine's health over time. And because every break is deduplicated by key and closures are sticky, running the sweep every few minutes is idempotent at the queue level: recurring problems keep touching the same row, resolved problems stay resolved. There's also a per-payment path for when you don't want to wait for the next scheduled pass — POST /payments/{id}/reconcile starts a reconciliation workflow for a single payment on demand, useful when an operator is actively chasing one item. The scheduled sweep keeps the whole book honest in the background; the on-demand trigger lets a human reconcile one payment right now.