Shipping stablecoin rails without the pain

Stablecoins promise global, instant, low-cost transfers. The promise is mostly real: the rails themselves work. Almost all of the friction lives in the boring layers around them, and that is where the engineering time actually goes.

We recently helped a client, a regulated digital asset firm, launch a stablecoin-backed remittance corridor between the US and the Philippines. The first version went from concept to production in three weeks. Since then the platform has settled more than $100M across 100,000+ transfers, with peak days above 5,000 transactions, and a typical settlement time of about five seconds on a route where the old correspondent banking path took two to three banking days. This is the condensed retrospective: what settlement actually means to a customer, how we made the custody decision, and why the unglamorous layers of idempotency, reconciliation, and logging turned out to be the real product.

Settlement is a UX choice, not a technical one

On a public chain, USDC settles in seconds. The token mechanics are well documented in Circle's developer docs, and the on-chain leg of a transfer was never our problem. The problem is that "settled" means at least three different things to someone sending money home: is my money safely in the system, what exchange rate am I getting, and when can my family actually spend pesos.

We ended up surfacing three explicit settlement states in the UI, and that single decision removed roughly 70% of support tickets.

The three states

Funds received. We have the sender's dollars and the transfer has been accepted into the system. Compliance screening has passed and the amount is locked in. The FX rate is not yet locked, and the recipient cannot spend anything. This state answers the sender's first question: did my money arrive safely.

Converting. The USDC leg has settled on-chain and the USD to PHP rate is locked. A payout instruction is on its way to the local payout partner. For the customer this is the moment the price becomes fixed: the pesos they were quoted are the pesos that will arrive. It is also the point of no cheap return, because unwinding from here means reversing an FX conversion at whatever the rate is now.

Delivered. The recipient's bank account or e-wallet has been credited and the money is spendable. Not "sent to the bank", credited. We only show this state after the payout partner confirms the credit, because the gap between "we sent it" and "they have it" is exactly where customer trust dies.

Why one "pending" state fails

Collapse those three into a single "pending" and you have one answer for three different questions. A sender asking "did my transfer go through" and a recipient asking "can I withdraw this yet" both see the same word, and neither gets an answer. Refunds make it worse: canceling in funds received is a full refund, canceling in converting involves an FX reversal, and canceling in delivered is not a cancellation at all, it is a new transfer in the opposite direction. If the interface cannot distinguish these states, your support team becomes the state machine, resolving it one ticket at a time.

Custody is the hardest decision you'll keep reopening

Every custody option costs something different, and the costs are not on the same axis. We weighed four:

• Self-custody. You hold the keys. Maximum control and minimal counterparty risk, but you now own hardware security modules, key ceremonies, disaster recovery drills, and the hardest conversation with your regulator. Cheap in fees, expensive in engineering and audit.
• MPC. Key material is split into shares across parties or devices, so no single machine can sign alone. Strong key control with a much better operational story than raw self-custody, but you inherit a vendor dependency and platform costs, and your team still owns signing policy and quorum operations.
• Qualified custodian. A regulated third party holds the assets. Lowest operational burden and the cleanest regulatory posture, at the price of custody fees, withdrawal latency, and concentrated counterparty risk.
• Banked custody. Stablecoin balances held inside a banking partner's platform. The least crypto operations of any option, and the most constrained: you move at the bank's speed and inside the bank's product roadmap.

The axes that actually mattered in the decision: who controls the keys, who carries the operational burden, how much counterparty risk you concentrate in one name, what story you can tell a regulator with a straight face, and what it costs at your expected volume.

We started with a qualified custodian, for speed. With a three-week timeline, "a regulated custodian holds the assets" is a sentence that compliance teams, banking partners, and regulators all understand on the first pass. Nobody had to hire key management staff or rehearse a key ceremony before launch. We accepted the fees and the concentration risk as the price of shipping.

The reason the decision keeps reopening is that the right answer changes with volume. Custody fees that are noise at low volume become a line item worth engineering against later, and the operational maturity you lack at launch is exactly what you build by operating.

An interface the settlement engine never sees through

What made this a decision instead of a trap: custody sits behind an interface. The settlement engine speaks a small vocabulary. Get a deposit address, initiate a withdrawal, fetch a balance, subscribe to confirmation events. It holds opaque account references and never imports a custodian SDK. Maker-checker approval for outbound signing lives behind the same boundary, so the approval workflow does not change when the implementation underneath it does.

That boundary is what let us commit, credibly, to swapping to MPC within a quarter once volume justified the operational investment. The swap plan was mechanical: implement the same interface against the MPC provider, run both implementations in parallel against the reconciliation checks, then cut over. The ledger, the transfer state machine, and the UI never learn that anything changed.

The boring stuff is the actual product

Half the engineering hours went into the parts no one demos. All of them show up at 2am.

Idempotent transfers, or double payouts

Every transfer request carries a client-supplied idempotency key, and every transfer is a state machine with explicit states and a short list of legal transitions. Retry a request and you get the same transfer back, not a second one. This sounds obvious until a mobile client on a weak connection retries a payout request four times, and each retry would have sent pesos.

Under the state machine sits an append-only double-entry ledger with per-asset zero-sum enforcement: every movement is a balanced pair of entries, and the books for each asset must sum to zero at all times. Transfers post two-phase entries, a hold placed on the debit leg when the transfer is accepted, then captured or released once the outcome is known. Replay protection falls out of this structure almost for free, because a duplicate posting against the same key is a no-op by construction. We cover the pattern in more depth in our post on double-entry ledger design.

Reconciliation is three-way or it is theater

We reconcile three sources against each other: on-chain events, our internal ledger, and bank statements from the fiat legs. Two-way reconciliation misses entire classes of failure. Comparing chain to ledger will not catch a bank crediting the wrong amount, and comparing ledger to bank will not catch a deposit that landed on-chain but was never booked. With three sources, every discrepancy has a direction, and the direction tells you which system is lying.

The checks are fail-closed. If an asset's books stop balancing, payouts for that asset halt until a human explains the difference. The same fail-closed posture applies to compliance: party, transaction, and address screening gate every transfer, and a screening outage stops the pipeline instead of waving transfers through. How we structure the matching and the halts is the subject of our post on the payment reconciliation engine.

Postmortem-grade logs

"Postmortem-grade" has a concrete meaning for us: months later, with access to nothing but the logs, you can reconstruct a single transfer's complete history. Every state transition records the idempotency key, the ledger entry ids, the transaction hash, the custodian reference, and the screening decision id. When a customer says a transfer from March arrived short, the answer is one log query and a ledger lookup, not an archaeology project across four dashboards. The test we apply is simple: could an engineer who joined last week write the incident timeline from logs alone.

What we'd do differently

• Build reconciliation on day one. We built it after the first discrepancy, which means the first discrepancy was diagnosed by hand across three exports. The engine was a small build. The manual version of it was one long, unpleasant investigation that the engine would have finished before lunch.
• Name the settlement states in the API from the start. We added the three states to the UI first and retrofitted them into the partner API later. Partners had already built against a single "pending", so we carried a translation layer longer than we wanted to.
• Treat the custodian sandbox as a fiction. Sandbox withdrawal behavior differed from production in both timing and error shapes. Run small real-money transfers through the production path early, before launch traffic finds the differences for you.
• Write the runbook with the feature. Every alert we shipped without a runbook was triaged from scratch the first time it fired.

One forward-looking note. The same discipline, idempotency keys, explicit states, fail-closed gates, is exactly what machine-initiated payments will demand. If software agents start moving money over rails like these, and protocols like x402 suggest they will, the systems that hold up will be the ones already built to be retried and replayed safely. We wrote about that shift in our post on agentic payments and x402.

If you're shipping financial rails of any kind, stablecoin or otherwise, and want a second pair of eyes on the architecture or the compliance trade-offs, that is exactly the kind of conversation we're happy to have.