# CodeDecoders: full content for LLMs

> Engineering studio building production-grade software — fintech, mobile, Web3, custom OS, AR/VR.

CodeDecoders is an engineering studio (Precision Engineering. Digital Excellence.). The blog publishes field reports and opinionated technical deep-dives on payments and money movement, ledgers and financial data integrity, on-chain architecture and smart-contract trade-offs, AI and agentic systems, and shipping durable production software. It is written for technical decision-makers (fintech and Web3 founders, scale-up CTOs) and favors engineering substance over hype.

Source: https://codedecoders.io · 11 articles · generated 2026-08-05

# Core banking architecture: event sourcing and CQRS

URL: https://codedecoders.io/blog/core-banking-architecture-event-sourcing-cqrs
Published: 2026-08-05 | Category: Fintech | Tags: event sourcing banking, CQRS ledger, domain-driven design fintech, open-source core banking, banking domain model

Core banking architecture with event sourcing and CQRS: what the pattern really buys you, why 61 bounded contexts is a warning sign, and where to draw the line.

A core banking architecture built on event sourcing and CQRS records every money movement as an immutable event, then derives balances, statements and reports by folding those events into read models. The write path validates a command against an aggregate and appends events. The read path serves projections built from those events. Nothing ever overwrites a balance.

For the part of a bank that touches money, that is the right default. The hard question is not whether to use event sourcing. It is how many bounded contexts you carve the bank into, because that is the decision you keep paying for: at every schema migration, every audit, every time a new engineer asks where a balance actually comes from.

There is now a concrete reference to argue with. The [FinAegis core banking prototype](https://github.com/FinAegis/core-banking-prototype-laravel), pushed again in the last week, models a bank as 61 domain modules on top of event sourcing and CQRS. Alongside it, [Midaz](https://github.com/LerianStudio/midaz) makes almost the opposite bet with three components. Reading both against each other is the fastest way to work out where your own boundary belongs.

## How event sourcing and CQRS work in a ledger

[Event sourcing](https://martinfowler.com/eaaDev/EventSourcing.html), in Martin Fowler's framing, captures all changes to application state as a sequence of events, so that "application state is purely derivable from the event log". CQRS is the companion move: separate the model you write through from the models you read from. They are independent patterns, but in a ledger they arrive together, because an append-only log is miserable to query directly.

Concretely, a transfer looks like this.

```cd-steps
{ "title": "One transfer through an event-sourced core",
  "steps": [
    { "title": "Command", "body": "TransferMoney arrives with an idempotency key, a source account, a destination account, an amount and a currency." },
    { "title": "Aggregate", "body": "The ledger aggregate rehydrates from its own event stream (or a snapshot plus the tail), then enforces the rules: sufficient funds, matching currency, key not already used, account not frozen." },
    { "title": "Append", "body": "If the rules hold, it appends two facts, MoneyDebited and MoneyCredited. That append IS the commit. There is no second write that could disagree with it." },
    { "title": "Project", "body": "Projectors consume the new events and update read models: account balance, statement rows, end-of-day position, compliance flags, analytics." },
    { "title": "Query", "body": "The API reads projections, never the raw stream. Balance lookups stay cheap because the folding already happened." }
  ] }
```

The payoff is that a balance stops being a number someone typed over and becomes a claim you can prove. Ask "why is this account at 4,182.50?" and the answer is a list of facts that sum to it. That property is what makes [double-entry ledger invariants](/blog/double-entry-ledger-foundations) enforceable rather than aspirational: if debits and credits are events, the sum-to-zero rule is checkable over the whole history, not just the last write.

## The problem it solves is the mutable balance

The alternative most products start with is one statement:

```sql
UPDATE accounts SET balance = balance - 500 WHERE id = 42;
```

That line is fast, obvious, and destroys evidence. After it runs you know the balance changed and nothing about why. Six months later a customer disputes a fee, a regulator asks for the state of an account on a specific date, or you find a 12 cent drift between your ledger and your PSP, and there is no record to reason from. You are reduced to inferring history from application logs, which is exactly the position that makes [payment reconciliation](/blog/payment-reconciliation-fintech) turn into a permanent tax on the team.

Event sourcing removes that class of problem outright: temporal queries, full audit trails and corrective replay all fall out of the log. That is a genuine, durable win, and it is why finance keeps reaching for the pattern even though [the learning curve is real](https://microservices.io/patterns/data/event-sourcing.html).

## Two reference designs that disagree

Here is where the two open-source references part ways.

```cd-compare
{ "columns": ["FinAegis prototype", "Midaz", "LedgerSMB"],
  "rows": [
    { "label": "Core shape", "cells": ["61 domain modules in one Laravel app", "3 components (ledger, tracer, infra)", "One PostgreSQL schema, logic close to the data"] },
    { "label": "Source of truth", "cells": ["Per-domain event streams", "Balance and operation rows", "Accounting tables"] },
    { "label": "Event sourcing", "cells": ["Yes, via Spatie Event Sourcing", "No, immutable hash-chained audit trail instead", "No"] },
    { "label": "CQRS", "cells": ["Yes, projectors to read models", "Yes, hexagonal command and query split", "No"] },
    { "label": "Concurrency control", "cells": ["Append to the aggregate stream", "Optimistic locking on a balance version", "Database transactions"] },
    { "label": "Stack", "cells": ["Laravel 12, PHP 8.4", "Go, PostgreSQL and MongoDB", "Perl 5.38, PostgreSQL 14"] },
    { "label": "Commits", "cells": ["2,254", "7,904", "19,438, lineage from 1999"] },
    { "label": "Stated status", "cells": ["Demonstration platform", "Source-available product", "Production accounting and ERP"] }
  ] }
```

Midaz is the interesting counterweight because it is not event-sourced at all, and says so in its code. Its ledger persists balances as rows and guards them with a version column, updating only where the stored version is older than the incoming one. Money history lives in operation rows and a hash-chained audit trail rather than in a replayable stream. Its hierarchy (organizations, ledgers, assets, portfolios, segments, accounts, transactions, balances) is eight concepts, not sixty-one, and it deliberately pushes rails such as PIX, cards and wires outside the core.

LedgerSMB is the third data point and the least fashionable: a double-entry accounting system whose lineage runs back to 1999, still on Perl and PostgreSQL, with no event sourcing anywhere. It is worth including precisely because it has outlived most of the architectures that would have called it naive.

## What a domain count actually tells you

FinAegis's headline number is 61 DDD domains as self-contained, individually installable modules. Counting the repository tree confirms it: exactly 61 directories under `app/Domain`, holding 2,367 files between them. So the claim is real, not marketing. The question is what those 61 boundaries buy.

Two things stand out once you count files rather than folders.

```cd-chart
{ "type": "bar", "title": "Files per domain folder in the FinAegis prototype (61 total)",
  "xKey": "domain",
  "series": [{ "key": "files", "label": "Files", "color": "#5BBDF9" }],
  "data": [
    { "domain": "AgentProtocol", "files": 208 },
    { "domain": "Exchange", "files": 150 },
    { "domain": "Compliance", "files": 149 },
    { "domain": "Account", "files": 111 },
    { "domain": "Payment", "files": 59 },
    { "domain": "Ledger", "files": 18 },
    { "domain": "Webhook", "files": 10 },
    { "domain": "Newsletter", "files": 8 },
    { "domain": "Auth", "files": 6 },
    { "domain": "Referral", "files": 3 }
  ] }
```

First, the distribution is extremely lopsided. Twelve of the 61 domains hold half the code. The tail is folders, not contexts: `Activity` and `Referral` have 3 files each, `Contact` has 4, `Auth` has 6. A bounded context with three files in it is a namespace someone was optimistic about.

Second, and more telling, the `Ledger` domain has 18 files. In a core banking platform, the ledger is smaller than `Mobile` (44), smaller than `Privacy` (49), and less than a tenth the size of `AgentProtocol` (208). The domain count tells you nothing about where the money model lives, and in this case the money model is not where the weight is.

Then there is the question of what counts as a domain at all. Fowler's [bounded context](https://martinfowler.com/bliki/BoundedContext.html) is a boundary drawn where the language changes: a place where "meter" or "account" means something different to a different group of people. Measured against that, roughly twenty of the 61 are not bounded contexts in any useful sense. `SMS` and `Mobile` are delivery channels. `Webhook` is a transport. `ISO8583` and `ISO20022` are message formats, which is to say adapters. `Auth`, `Security`, `Privacy`, `Monitoring` and `Performance` are cross-cutting concerns that cannot be bounded because they apply everywhere. `Shared` is, by name, the opposite of a bounded context. `VisaCli` is a command line tool.

Naming is not modelling. Putting a concern in its own folder does not give it a ubiquitous language, an aggregate, or an invariant it owns. It gives it an import path. If you cannot say in one sentence which rule about money a context protects, it is not a context yet.

None of this makes the prototype bad. It explicitly calls itself a demonstration platform and says a security audit and compliance review are required before production use, which is a more honest posture than most repos in this space manage. But if you are using it as an argument for modelling your own bank as dozens of contexts, the file counts are the part to read.

## The bill, and when it arrives

Event sourcing's costs are real and they are deferred, which is the worst combination for a startup making architecture decisions at month three.

**Event schemas are forever.** Once an event is in the store, every future version of your code must be able to read it. Adding a field is not a migration, it is an upcasting function you now maintain indefinitely. This is the highest-cost change in the pattern, so decide your versioning approach before the first event lands, not after ten million have.

**Projection rebuilds get slow, then scary.** Rebuilding a read model from scratch is inevitable (you will fix a projector bug, or add a report that needs history). At low volume it takes seconds. At production volume it takes hours, during which that read model is stale or offline. Design the rebuild path, and a way to run a new projection alongside the old one, from day one.

**Replay and the outside world do not mix.** Fowler flags this as the tricky part, and he is right: replaying events must not re-send yesterday's payouts, emails or webhook calls. Every gateway to an external system needs an explicit replay mode. Get this wrong once and you have paid a vendor twice.

**Reads are eventually consistent, and users notice.** The transfer succeeded but the balance still reads old, because the projector has not caught up. In a payments UI that reads as a bug even when the ledger is perfectly correct. You either read the aggregate for the paths that must be immediate, or you design the interface to show pending state honestly.

**Deletion is genuinely hard.** An append-only log and a right-to-erasure request are in direct tension. Crypto-shredding (encrypt personal data per subject, then destroy the key) is the usual answer, and it needs to be in the design before a regulator asks.

One thing event sourcing does make easier: because commands carry keys and the aggregate can see its own history, deduplication has somewhere natural to live. That still does not replace proper [idempotency keys on payment APIs](/blog/idempotency-keys-payment-apis) at the edge, but it does mean the two layers reinforce each other instead of both guessing.

## Where to draw the boundary

The pattern that holds up: **one bounded context owns the money invariant, and everything else is an adapter or a genuinely separate service with its own store.**

Practically, apply three tests before promoting anything to a context of its own. Can you state, in one sentence, the invariant it protects? Does it own data that no other context may write? Would a single team be able to own it end to end? A context that fails all three is a package, and calling it a domain just adds a hop between you and the code.

For most fintech products this collapses to a small core: accounts and balances, postings, plus one aggregate per thing that genuinely has its own lifecycle and rules (a loan, a card, a settlement batch). Compliance, treasury and reporting are usually consumers of the ledger's event stream rather than peers of it. Rails are adapters. Notifications are subscribers. That shape is close to what Midaz ships, and it is also what tends to survive contact with the second and third payment provider you integrate.

## When to skip it

Event sourcing is the wrong call more often than its fans admit. Skip it when the domain is CRUD with an audit column bolted on, when nobody will ever ask what state something was in last March, or when your team does not yet have the operational maturity to run projections, replays and two consistency models at once. Skip it for the parts of the product that are not money: user profiles, content, settings. Mixing a mutable model for those with an event-sourced core is not inconsistency, it is choosing the cost where it pays.

And take LedgerSMB seriously as evidence. Twenty-seven years of double-entry accounting, no event streams, still running businesses. Correct money handling comes from taking invariants seriously, not from the persistence pattern. Event sourcing makes the right invariants cheaper to prove. It does not supply them.

If you are picking this architecture now and want a second opinion on where the boundary should sit before the first million events land, that is a conversation worth having early: our [fintech engineering team](/services/fintech-development) does exactly this kind of review, and you can [get in touch](/contact) with the specifics of your ledger.


# UPI anatomy: how real-time payment rails work

URL: https://codedecoders.io/blog/upi-real-time-payment-rail-anatomy
Published: 2026-07-29 | Category: Fintech | Tags: real-time payment rails, UPI architecture, NPCI switch, instant settlement, payment rail design, PSP

How UPI works under the hood: the PSP to NPCI switch to bank flow behind real-time payment rails, why authorization and settlement split, and what breaks.

A UPI payment feels like one atomic action. You scan, you type a PIN, and two seconds later both phones say it worked. Under that two seconds sit seven parties, a central switch, two separate bank ledger writes, and a settlement that will not actually happen for hours.

That gap is the whole design. Real-time payment rails are not "money moves instantly." They are "the *promise* moves instantly, and the money follows on a schedule." Once you internalize that split, most of the weird behavior in real-time rails (deemed transactions, reversal windows, net debit caps, penalty clauses) stops being weird and starts being the obvious consequence of a deliberate trade-off.

UPI is the largest working example of that trade-off, so it is the best one to dissect. Here is how it is actually wired.

## What a real-time payment rail actually is

A real-time retail payment rail has three properties that separate it from a card network or a batch transfer like ACH or NEFT:

1. **Push, not pull.** The payer's bank initiates the debit. There is no merchant-initiated pull against a stored credential, so there is no "authorize now, capture later" step and no chargeback in the card sense.
2. **Irrevocable and near-instant confirmation.** Both parties get a final answer in seconds, and a successful payment cannot be unilaterally clawed back by the payer.
3. **Deferred net settlement between the banks.** The interbank money movement is batched and netted, then posted through the central bank.

Property 3 is the one everyone forgets. It is also the one that makes properties 1 and 2 affordable. If every ₹40 chai payment required a real gross transfer between two banks through the central bank, the rail would collapse under its own settlement volume.

## Why UPI is the reference implementation

UPI runs at a scale that forces every one of these design decisions into the open. It cleared 23.2 billion transactions worth ₹29.9 trillion in [May 2026](https://www.aninews.in/news/business/upi-hits-new-high-in-may-2026-with-232-billion-transactions-worth-rs-299-trillion-npci-data-shows20260602155337/), which works out to roughly 738 million payments a day, and it accounts for close to half of all real-time payment volume on the planet.

```cd-chart
{ "type": "bar", "title": "UPI monthly transaction volume, 2026",
  "xKey": "month",
  "unit": "B",
  "series": [{ "key": "txns", "label": "Transactions (billions)", "color": "#5BBDF9" }],
  "data": [
    { "month": "Jan", "txns": 21.70 },
    { "month": "Feb", "txns": 20.39 },
    { "month": "Mar", "txns": 22.64 },
    { "month": "Apr", "txns": 22.35 },
    { "month": "May", "txns": 23.20 },
    { "month": "Jun", "txns": 22.72 }
  ] }
```

At that volume, an architecture that is merely "correct" is not enough. It has to be correct while degrading gracefully, because a 1% failure rate is 7 million broken payments a day.

## The parties in a single tap

The mental model most engineers carry ("my app talks to my bank") is wrong by about five hops. A UPI transaction involves:

- **The payer's app (TPAP).** PhonePe, Google Pay, Paytm. It collects intent and captures the PIN, but it never sees the PIN in cleartext. Entry happens inside an NPCI-certified component the app cannot read.
- **The payer's PSP.** A sponsor bank that issues the `@handle` and signs requests. The suffix in `you@ybl` identifies the sponsor bank, not the app you tapped.
- **The payer's (remitter) bank.** The only party that can decrypt the PIN, check the balance, and debit the account.
- **The NPCI switch.** The central router. It resolves addresses, sequences the debit and credit, and keeps the authoritative record of what happened.
- **The payee's (beneficiary) bank.** Credits the receiving account.
- **The payee's PSP and app.** Relay the outcome to the merchant or recipient.

Notice what the switch does *not* do: it never holds funds. It routes messages and computes obligations. The money never sits in an NPCI account mid-flight.

## Anatomy of a transaction: debit first, credit second

The API surface is XML over HTTPS, and the core message is [`ReqPay`](https://s3-ap-southeast-1.amazonaws.com/he-public-data/NPCI%20API%20Descriptionsb9bceb7.pdf), with `RespPay` carrying the outcome back. Address resolution runs through `ReqValAdd`, and status checks through `ReqChkTxn`. The ordering is the interesting part.

```cd-steps
{ "title": "One UPI payment, end to end",
  "steps": [
    { "title": "1. Intent", "body": "The app assembles payee address and amount. The PIN is captured in a certified component and encrypted with a key the app cannot access. The PSP signs the request and sends ReqPay to the NPCI switch." },
    { "title": "2. Resolve", "body": "The switch resolves the virtual address (you@ybl) with the payee's sponsor bank to get a real account. The resolved name is what the payer sees on the confirmation screen before authorizing." },
    { "title": "3. Debit", "body": "The switch asks the payer's bank to debit. Only that bank can decrypt the PIN and check it against the account. This is the authorization decision, and it is the only place it happens." },
    { "title": "4. Credit", "body": "Only after a confirmed debit does the switch request the credit from the payee's bank. Debit-then-credit means the failure mode is a stuck debit, never a phantom credit." },
    { "title": "5. Relay", "body": "The switch tells each sponsor bank the result, and each PSP tells its own app. Neither app hears from the switch directly. Two to three seconds have passed." },
    { "title": "6. Settle", "body": "Hours later, NPCI nets each bank's position and posts the difference through RBI's RTGS. No money moved between the banks until this point." }
  ] }
```

Step 4 is the load-bearing ordering decision. Because credit is strictly conditional on a confirmed debit, the rail can never create money out of a partial failure. It can only *strand* it, which is a recoverable problem. That is the same invariant a [double-entry ledger enforces on every fintech app](/blog/double-entry-ledger-foundations): you would rather have a suspense balance you can reconcile than a credit with no matching debit.

## Authorization is not settlement

Here is the split, stated plainly.

**Authorization** is the payer's bank saying "this PIN is valid, this balance is sufficient, I have debited the account." It happens in step 3, in about a second, per transaction.

**Settlement** is the actual movement of funds between the two banks. UPI runs [10 settlement cycles per business day between 9 AM and 9 PM](https://www.angelone.in/news/personal-finance/new-upi-settlement-rules-to-start-from-3-november), plus two dedicated dispute cycles (DC1 running midnight to 4 PM, DC2 running 4 PM to midnight) that NPCI split out from the authorized cycles in November 2025 so that chargebacks and reversals stop clogging the main path.

In each cycle NPCI computes a **net position** per bank, not a per-transaction transfer. If Bank A's customers sent ₹900 crore to Bank B's customers and Bank B's customers sent ₹850 crore back, one ₹50 crore transfer settles both directions. Those net positions post through the RBI's RTGS system, with the central bank holding the settlement accounts.

So for a window of up to several hours, the payee's bank has credited a customer against money it has not yet received. That is not sloppiness, it is the deliberate extension of intraday credit that makes the rail economically viable. It also means the rail carries genuine settlement risk, which is why participation comes with collateral and net debit cap requirements rather than being open to anyone with an API client.

```cd-compare
{ "columns": ["Card network", "UPI (real-time rail)", "NEFT / ACH (batch)"],
  "rows": [
    { "label": "Direction", "cells": ["Pull from stored credential", "Push from payer's bank", "Push, queued"] },
    { "label": "User-visible latency", "cells": ["1-3s auth, capture later", "2-3s, final", "30 min to next business day"] },
    { "label": "Auth and settlement", "cells": ["Split (auth, then capture, then T+1/T+2)", "Split (instant auth, same-day net cycles)", "Merged into the batch"] },
    { "label": "Reversal model", "cells": ["Chargeback, weeks", "Auto-reversal, T+1 for P2P", "Return file"] },
    { "label": "Cost per txn", "cells": ["Interchange, ~1.5-2%", "Zero for P2P", "Flat, low"] }
  ] }
```

## Where it actually breaks

Three failure classes matter, and they are not equally your problem.

**Business declines (BD)** are the user's side: wrong PIN, insufficient balance, per-transaction or daily limit hit. NPCI's [circular OC-149](https://productgrowth.in/insights/fintech/upi-payment-success-rates/) asks banks to keep these under 5%, and in practice they run near 1 in 10 payments. You cannot engineer these away, only reduce them with better UX (a pre-flight balance hint, and copy that makes clear the UPI PIN is not the ATM PIN).

**Technical declines (TD)** are infrastructure: bank server timeouts, switch overload, a core banking system that folds under a festival spike. The target is under 1%, and the ecosystem has dragged this from 8-10% in 2016 to roughly 0.7-0.8% today. Blended merchant success rates land in the 92-96% range once BD is included.

**Deemed transactions** are the interesting one, and the reason this article exists. A debit succeeded but the credit confirmation never came back. The payer's money is gone and nobody has told anyone anything definitive.

```cd-callout
{ "variant": "warn", "title": "Never poll a pending payment in a tight loop", "body": "NPCI permits the first ReqChkTxn only 90 seconds after authentication, and at most three checks in any two-hour window. A client that retries aggressively gets rate limited, and a client that resubmits the payment instead of checking its status creates a duplicate debit. Status check and retry are different operations." }
```

The rail resolves these itself. NPCI reconciles deemed transactions and posts a verdict, either credit confirmed or debit reversed. For P2P transfers the reversal is guaranteed within one day, and a bank that misses the window owes the customer a per-day penalty. Merchant payments take a few days longer.

For your system, a deemed transaction is a state, not an error. It needs its own status in your model, distinct from both success and failure, and it needs to be reconcilable when the verdict arrives. This is exactly the class of problem [a reconciliation engine has to survive](/blog/payment-reconciliation-fintech): a payment whose true outcome is known to the network before it is known to you.

The other half of the defense is on the write path. If your retry logic can resubmit a payment whose outcome is merely unknown, you will eventually double-debit a real customer. [Idempotency keys on your payment APIs](/blog/idempotency-keys-payment-apis) are what make the difference between "the network was slow" and "we charged them twice."

## What to take from this if you are building a rail

You are unlikely to build a national switch. You are quite likely to build something that sits on one, or that copies its shape for a closed-loop wallet, a marketplace payout system, or an internal transfer product. The transferable decisions:

**Separate authorization from settlement in your own model, explicitly.** Two different objects, two different lifecycles, two different reconciliation processes. Teams that model a payment as one row with a status column discover the problem the first time settlement disagrees with authorization.

**Order your writes so the failure mode is recoverable.** Debit before credit. A stranded debit is a support ticket. A phantom credit is a loss.

**Design the three-way outcome from day one.** Success, failure, and unknown. Unknown is not a transient state you can code around later, it is a permanent citizen of any distributed money system, and every downstream consumer (ledger, notifications, support tooling, accounting export) needs an answer for it.

**Net where you can.** Netting is not an optimization you bolt on at scale. It changes your liquidity requirements, your counterparty exposure, and your collateral math, so it belongs in the design from the start. The same logic drives batching in much smaller systems, including the [sub-cent agent settlement patterns](/blog/nanopayments-sub-cent-agent-settlement) where per-transaction settlement costs more than the transaction.

**Assume bursts, not averages.** 738 million payments a day is the average. Diwali is not. Build for the 10x minute, not the mean hour.

The elegance of UPI is not that it is fast. Plenty of things are fast. It is that it made a clean, honest cut between the part that has to be instant (the user's answer) and the part that does not (the banks' money), and then engineered each side to its own requirements. That cut is the reusable idea.

If you are designing a payment rail or untangling one that has outgrown its original model, our team does this work end to end; [see how we approach payment rails development](/services/payment-rails-development), or [get in touch](/contact) and tell us what is breaking.


# Payment idempotency: prevent duplicate charges

URL: https://codedecoders.io/blog/idempotency-keys-payment-apis
Published: 2026-07-23 | Category: Fintech | Tags: payment idempotency, idempotency keys, duplicate charge prevention, payment retries, idempotency-key header

A practical guide to payment idempotency: design idempotency keys, store responses, and handle retries so a network timeout never double charges a customer.

Payment idempotency is the property that sending the same charge request twice produces one charge, not two. You get it by having the client attach a unique key to each payment intent, and having the server remember what it did with that key. The next time the same key arrives, the server replays the original result instead of moving money again.

If your payment endpoint does not do this, you already have a duplicate charge bug. It just has not fired for a customer who matters yet. This is a how-to for adding payment idempotency to a real endpoint: how to design the key, how to store the result, how to handle two requests racing on the same key, and how to verify the whole thing actually holds.

## Why a retry turns into a double charge

Picture the simplest charge flow. The client POSTs to `/charges`, your server calls the acquirer, the money moves, and you return `200`. Now drop the response on the floor. The acquirer succeeded, but the client's HTTP call timed out at 30 seconds, or the mobile network dropped, or your load balancer killed a slow connection. The client did the reasonable thing and retried. Your server charged the card again.

This is not a rare edge case. It is the normal behavior of every network under load. Timeouts, 502s from a proxy, a user double-tapping "Pay", a job runner retrying a failed step, and now agent-driven traffic that retries on any non-200: every one of these produces two identical requests where you intended one. The client cannot tell "the charge failed" apart from "the charge succeeded but the response was lost", so a correct client retries both.

The fix is not "retry less". Retries are how distributed systems stay available. The fix is to make the second, third, and fourth attempt safe, so that only the first one moves money. That is what payment idempotency buys you.

```cd-callout
{ "variant": "warn", "title": "The client cannot solve this alone", "body": "A dropped response is indistinguishable from a failure at the client. Only the server, which knows whether the charge actually executed, can make the retry safe. Idempotency is a server responsibility." }
```

## The idempotency contract

The mechanism is a single request header. The client generates a unique key per logical operation and sends it on the charge request. The emerging [IETF Idempotency-Key HTTP header draft](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) formalizes exactly this: a client attaches `Idempotency-Key` to a non-idempotent method like POST, and the server is responsible for the key's lifecycle. [Stripe's idempotent requests](https://docs.stripe.com/api/idempotent_requests) are the same idea in production, and they are a good reference for the details.

Three rules make the contract work:

First, the client owns key generation, not the server. Do not derive the key from a hash of the request body. Two genuinely different payments can hash-collide in the ways that matter (same amount, same customer, same minute), and a legitimate second purchase would get silently deduplicated into the first. Have the client mint a fresh [UUID v4](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/) per checkout attempt and reuse that same value across its own retries.

Second, the key identifies a logical operation, not a transport call. The client generates it once, at the top of the operation, and holds it for every retry of that operation. If it generates a new key on each retry, you are back to duplicate charges.

Third, the server stores the outcome against the key and replays it. Stripe saves the status code and body of the first request and returns the identical result on any later request with the same key, including replaying a `500`. That last part matters: replaying the original error is correct, because it tells the client the true outcome of the operation it started rather than starting a new one.

```cd-compare
{ "columns": ["Body hash", "Client UUID", "UUID + user + operation"],
  "rows": [
    { "label": "Who generates it", "cells": ["Server", "Client", "Client"] },
    { "label": "Two real purchases collide?", "cells": ["Yes, silently dropped", "No", "No"] },
    { "label": "Cross-user leak risk", "cells": ["Possible", "Possible", "No"] },
    { "label": "Recommended", "cells": ["No", "OK", "Yes"] }
  ] }
```

Scope the stored key by more than the raw string. A good storage key is the idempotency key combined with the authenticated user or account ID and the operation type. That prevents one tenant's key from ever colliding with another's, and it stops a recycled client-side UUID from matching a payment it has nothing to do with.

## Implementing it on a payment endpoint

The naive version ("look up the key, if missing then charge, then save") has a race in it that this walkthrough closes. Two requests with the same key can both pass the initial lookup before either has written a result, and both then charge. Here is the sequence that is actually safe.

```cd-steps
{ "title": "Idempotent charge, step by step",
  "steps": [
    { "title": "Read the key", "body": "Require the Idempotency-Key header on POST /charges. Reject the request with 400 if it is missing, so you never charge without one." },
    { "title": "First lookup", "body": "Check the idempotency store for (key, user, operation). If a completed record exists, return its saved status and body. Done, no charge." },
    { "title": "Acquire a lock", "body": "Take a distributed lock on the storage key. If another request holds it, the operation is already in flight, so return 409 and let the client retry shortly." },
    { "title": "Re-check under the lock", "body": "Look up the record again now that you hold the lock. A concurrent request may have finished between step 2 and step 3. If so, replay it." },
    { "title": "Charge and persist atomically", "body": "Call the acquirer, then write the charge row and the idempotency record in one database transaction so they commit together or not at all." },
    { "title": "Release and respond", "body": "Release the lock and return the result. Every future request with this key now replays this exact response." }
  ] }
```

Two properties in that sequence are load-bearing, and both are easy to get wrong.

Atomicity is the first. The charge and the idempotency record must commit together. If you charge the acquirer and then crash before saving the record, the next retry sees no record and charges again. Write both in the same database transaction. If the acquirer call sits outside your database, use the outbox pattern: record the intent inside the transaction, then reconcile against the acquirer before releasing the result. This is the same ledger discipline that underpins [double-entry accounting for fintech systems](/blog/double-entry-ledger-foundations), where every money movement has to be recorded exactly once or the books stop balancing.

Durability is the second. Your idempotency store has to survive a restart, because a miss on that store is a duplicate charge. Redis with a default eviction policy is the classic trap: under memory pressure it evicts your keys, and the "cache" you relied on to prevent double charges quietly forgets them mid-incident. Use a durable store (a real table in your primary database, or Redis with persistence and eviction turned off for this keyspace), and treat idempotency records as system-of-record data, not as a cache.

## Pitfalls that quietly reintroduce double charges

The header is easy. The failure modes are where teams lose money.

Reusing keys with different payloads. A buggy or lazy client might send the same idempotency key for a $10 charge and later a $500 charge. If you blindly replay the first result, the customer never gets charged the $500. If you blindly charge, the key did nothing. The right answer, and what Stripe does, is to compare the incoming parameters against the stored request and reject the mismatch (a `422`), so the conflict surfaces loudly instead of corrupting state.

Caching in-flight failures as final. When a request fails validation or collides with a concurrent request on the same key, do not persist that as the final outcome. Stripe explicitly does not save a result in these cases, so the client can retry. Only persist once the operation reached a terminal state.

No expiry policy, or the wrong one. Idempotency records need a TTL. Twenty-four hours is a common baseline and matches Stripe's window. Too short and a client retrying after a long outage double charges. Too long and you store unbounded state. Publish the window so clients know how long a key is safe to reuse, which the IETF draft requires servers to do.

Forgetting the downstream blast radius. A duplicate charge does not stay contained to one row. It flows into settlement files as a phantom transaction, and it becomes an unmatched or duplicated record that your ops team has to chase. If you have read our guide to building a [payment reconciliation engine](/blog/payment-reconciliation-fintech), the "unknown in settlement" bucket is often just an upstream idempotency bug wearing a disguise. Idempotency at the edge is what keeps reconciliation tractable downstream.

Ignoring machine callers. Agents and job runners retry far more aggressively than humans, in tight loops. An autonomous payer with wallet access and a retry bug is an expensive incident, which is why idempotency is table stakes for the [agentic payment rails we have written about](/blog/agentic-payments-x402-ap2-mpp). If you expose a paid endpoint to machine traffic, assume every request arrives at least twice.

## How to verify it actually holds

Do not trust an idempotency layer you have not tried to break. Three tests catch almost every regression.

Fire the same key twice, sequentially. Send a charge, capture the response, send the identical request with the same key, and assert you get the byte-identical response and exactly one charge row in the database. This is the happy path and should be a permanent integration test.

Fire the same key concurrently. Launch two or more requests with the same key at the same instant and assert that exactly one charge occurs and every caller receives a consistent result (the winner's result, or a `409` that resolves to it on retry). This is the test that catches a missing lock, and it is the one most teams skip.

Kill the process mid-charge. Inject a crash between the acquirer call and the commit, then replay the request. Assert there is still exactly one charge. This proves your atomicity story is real and not just a comment in the code, and if you cannot pass it, your charge and your idempotency record are not committing together.

Payment idempotency is a small amount of code and a large amount of discipline. Get the key ownership, the lock, and the atomic write right, and duplicate charges stop being a bug you can ship. If you are hardening a charge path and want a second set of eyes on the failure modes, our [payment rails engineering team](/services/payment-rails-development) does exactly this work, and you can [get in touch](/contact) to talk it through.


# Nanopayments: settling sub-cent agent payments

URL: https://codedecoders.io/blog/nanopayments-sub-cent-agent-settlement
Published: 2026-07-13 | Category: Fintech | Tags: nanopayments, sub-cent payments, agentic payments, usdc micropayments, payment netting

Nanopayments let agents settle USDC amounts smaller than a network fee. How batching, netting and fractional-cent accounting make sub-cent payments work.

A nanopayment is a transfer worth less than the fee it would cost to send it. [Circle's nanopayments](https://www.circle.com/nanopayments) advertise USDC transfers as small as $0.000001, one ten-thousandth of a cent. Send that on Ethereum during a normal 2025 base fee of around $0.53, and the network fee is roughly 53 million times the payment. No rail built for human commerce can settle that. The whole problem of nanopayments is moving value at a size where the transaction cost, not the payment, dominates.

This matters now because agents pay per call. When an autonomous agent buys a vector search, a token of inference, or a single API response, the natural price is a fraction of a cent, metered thousands of times a minute. The answer is not a cheaper transaction. It is to stop settling every payment as its own transaction. This is how sub-cent settlement actually works: aggregation, netting, and payment channels, plus the ledger discipline to account for fractional cents without losing money to rounding.

## Why a per-transaction fee kills a sub-cent payment

Every payment rail carries two costs: a percentage and a fixed floor. The percentage scales down fine. The fixed floor is what breaks micropayments. Card processing runs about 2.9% plus $0.30 for online transactions ([a standard published rate](https://merchantinsiders.com/blogs/stripe-fees/)). On a $0.003 API call the percentage is a rounding error, but the 30 cent floor is 100 times the payment. You would pay $0.303 to move $0.003.

On-chain settlement has the same shape with a different floor. Instead of interchange you pay gas, and gas is priced per transaction regardless of the amount moved. That is why the on-chain data already shows the distortion. [Chainalysis reports](https://www.chainalysis.com/blog/x402-agentic-payments-adoption/) that on Base, x402 payments of a dollar or more grew from 49% of volume in early 2025 to 95% by early 2026, while payments between 10 cents and a dollar collapsed from 46% to 4%. Real sub-cent demand did not disappear. It cannot economically settle one transaction at a time on-chain, so it has to move somewhere else. That somewhere is a batching layer.

## What a nanopayment actually is

A nanopayment is not a tiny on-chain transfer. It is a signed promise to pay, verified instantly off-chain, that settles on-chain later in a group. The payment your agent authorizes and the settlement that hits the chain are two different events, deliberately decoupled.

Circle's implementation builds directly on the [x402 protocol](https://www.x402.org/) and the [EIP-3009](https://eips.ethereum.org/EIPS/eip-3009) "transfer with authorization" standard. An agent signs an authorization for an exact amount. The seller submits that signature to be verified against the agent's available balance, and the amount is deducted immediately. The signed authorizations pile up off-chain, and [Circle Nanopayments](https://www.circle.com/blog/a-deep-dive-into-circle-nanopayments-gas-free-usdc-transfers) periodically submits a single batch transaction that settles thousands of them at once. One on-chain transaction, thousands of payments, gas cost per payment approaching zero. Verification and batching happen inside a Trusted Execution Environment (TEE) that signs the batch, and the on-chain contract checks that signature before updating balances.

The user-facing experience is a sub-second confirmation. The chain sees a commitment minutes later. That gap is the entire trick, and it is also where the risk lives.

## How sub-cent settlement actually works

There are three ways to escape the per-transaction floor, and production stacks combine them.

```cd-compare
{ "columns": ["Aggregation (batching)", "Netting", "Payment channels"],
  "rows": [
    { "label": "Core idea", "cells": ["Sign many payments off-chain, settle them in one on-chain batch", "Cancel opposing debits and credits, settle only the net difference", "Two parties update a signed running balance, settle once on close"] },
    { "label": "Best for", "cells": ["Many small payments, many payers, one settlement window", "Parties who both pay each other (agent marketplaces, mutual APIs)", "A long-lived stream between two counterparties"] },
    { "label": "On-chain cost", "cells": ["One tx per batch, amortized across all payments", "One tx for the net amount per settlement cycle", "Two tx total: open and close the channel"] },
    { "label": "Main trade-off", "cells": ["Trust the batcher between settlements", "Needs bidirectional flow to net anything", "Capital locked in the channel while open"] }
  ] }
```

Aggregation is what Circle ships and what most metered API billing needs: thousands of payers each making tiny payments, collapsed into one settlement transaction. Netting is the accounting move underneath it. If agent A owes B a cent across forty calls and B owes A eight tenths of a cent, you settle two tenths of a cent once, not forty-plus times. Netting shrinks both the number and the size of settlements, which is why it is the backbone of interbank and card settlement too. Payment channels (the Lightning-style model) suit a sustained one-to-one stream: two parties keep a signed balance that only touches the chain when the channel opens and closes.

The batch settlement flow is the same regardless of which model you run underneath:

```cd-steps
{ "title": "How a batch of nanopayments settles", "steps": [
  { "title": "Prefund", "body": "The agent deposits USDC into a balance the settlement layer can draw against." },
  { "title": "Authorize", "body": "For each paid call, the agent signs an EIP-3009 authorization for the exact amount. The seller verifies it and delivers instantly." },
  { "title": "Accrue", "body": "Signed authorizations accumulate off-chain. Balances update in real time; the chain has not been touched yet." },
  { "title": "Net & batch", "body": "At the settlement window, opposing amounts are netted and the remaining payments are bundled into one transaction." },
  { "title": "Commit", "body": "A single on-chain transaction settles the whole batch. Gas is amortized across thousands of payments and receipts are issued." }
] }
```

If you are choosing between rails and protocols for this layer rather than the settlement mechanics, we mapped the landscape in [agentic payments: what x402, AP2 and MPP mean for builders](/blog/agentic-payments-x402-ap2-mpp). The protocols decide how an agent authorizes and proves a payment. The mechanics above decide whether that payment can settle for less than it is worth.

## The accounting problem: fractional cents

Decoupling authorization from settlement creates a ledger problem most fintech stacks are not built for. If your money column stores integer cents, a $0.003 payment is not representable. Round it and you either overcharge every call or leak value on every call. At a few thousand calls a minute, systematic rounding is not a rounding error. It is a revenue line.

The fix is the same discipline that underpins any correct money system: pick a base unit small enough that every amount you handle is an integer, and never store money as a float. USDC has six decimals, so the natural base unit is one micro-USDC (0.000001), and every nanopayment is an exact integer count of those units. Rounding, if it happens at all, happens once at a defined boundary with a documented rule, not implicitly on every arithmetic operation. This is the fractional-unit version of the invariants we cover in [double-entry ledgers: the backbone of every fintech app](/blog/double-entry-ledger-foundations): every authorization is a debit against the agent's prefunded balance and a credit to the seller's accrued balance, and the two sides must always tie out to the batch that eventually settles on-chain.

Reconciliation gets a new failure mode too. You are no longer matching one payment to one settlement. You are matching thousands of off-chain authorizations to one on-chain batch, and any authorization that was verified but excluded from the batch (dropped, expired, or disputed) is a break you have to detect. The matching engine is the same shape as the one in [building a payment reconciliation engine that survives](/blog/payment-reconciliation-fintech), except the "one" side is now a batch commitment and the "many" side is a stream of signatures.

## What you have to get right

Three things separate a nanopayment system that holds from one that quietly loses money.

**Idempotency.** Agents retry, and a signed authorization is replayable. Treat the authorization signature (or its EIP-3009 nonce) as the idempotency key and deduplicate on it, with a cache window wider than your longest retry. A double-counted sub-cent payment is trivial once. Multiplied across a retry storm, it is a reconciliation break you will spend a day chasing.

**Float and prefunding.** Aggregation means the settlement layer holds authorized-but-unsettled value on your behalf between windows. That is float, and float is a custody decision. Who holds the prefunded balance, how is it topped up before it runs dry mid-stream, and what happens to accrued payments if settlement fails? These are the same custody trade-offs we worked through in [shipping stablecoin rails without the pain](/blog/shipping-stablecoin-rails-without-the-pain), now with a real-time drawdown against the balance.

**Trust between settlements.** Between authorization and on-chain commitment, someone is holding your promises. A TEE-signed batch narrows that trust, but it does not remove it.

```cd-callout
{ "variant": "warn", "title": "The window is where the risk lives", "body": "Every unsettled batch is money you have delivered against but not settled on-chain. Cap the batch size and shorten the window so a failed or malicious settlement can only ever put a bounded amount at risk. An unbounded settlement window is an unbounded liability." }
```

## Where this leaves builders

Nanopayments are not a cheaper transaction. They are the admission that per-transaction pricing and sub-cent payments cannot coexist, so you settle in batches and keep an exact off-chain ledger in between. The pattern is old (netting predates blockchains by a century) but the trigger is new: software that pays software thousands of times a minute has finally made sub-cent settlement a load-bearing requirement rather than a novelty.

If you are designing the settlement, float, and accounting layer for an agent-native product and want a second opinion on the mechanics, that is the kind of problem we work through with teams building [agentic payment infrastructure](/services/agentic-payments). [Get in touch](/contact) if you want to pressure-test the design before it meets production volume.


# Double-entry ledgers: the backbone of every fintech app

URL: https://codedecoders.io/blog/double-entry-ledger-foundations
Published: 2026-07-06 | Category: Fintech | Tags: double-entry accounting, ledger system design, fintech ledger, append-only ledger

A double-entry ledger is the 500-year-old accounting model under every fintech app. How it works, the invariants that keep money correct, and why it matters.

A double-entry ledger is the data model that records every movement of money as two matching halves: a debit from one account and a credit to another. The two halves always sum to zero. That single rule is why a real ledger can tell you not just that your balance changed, but exactly where every cent came from and where it went.

If you are building a fintech product (a wallet, a payments app, a lending platform, a marketplace that holds funds) you will end up building a ledger whether you plan to or not. The only question is whether you build the right one on purpose or a broken one by accident. This is what a double-entry ledger is, how it works, and the handful of invariants that separate a ledger you can trust from a spreadsheet that lies to you at scale.

## A 500-year-old idea, codified in 1494

Double-entry bookkeeping is not a software pattern. It is a 500-year-old accounting model. Venetian merchants were already using it in the 13th and 14th centuries, but it was the mathematician Luca Pacioli who codified it in print. In November 1494 he published the Summa de Arithmetica, a 615-page encyclopedia of mathematics that included a 27-page section, Particularis de Computis et Scripturis, describing the Venetian method of recording trade.

That section is the first published description of double-entry bookkeeping. It spread the method across Italy, then Europe, then the world, and it underpinned the rise of modern commerce. The reason it survived 500 years unchanged is that it solves a problem that never goes away: how do you record value moving between parties so that the records can never silently drift out of balance.

Software did not improve on the idea. We just rediscovered, usually the hard way, that it was right all along.

## What single-entry gets wrong

Most products start with single-entry. You have a `users` table with a `balance` column, and when money moves you add to one row and subtract from another. It works in a demo. It falls apart in production.

The failure mode has a name in the wild: dancing cents. Money goes missing not because it was stolen, but because nobody can reconstruct why a balance is what it is. A widely cited account from engineer Alvaro Duran describes exactly this: a single-entry system where cents disappeared into the application with no way to trace them. His verdict is blunt. Single-entry ledgers are undebuggable. They tell you what a balance is, but never why.

The deeper anti-pattern is treating balance as a property of a domain object: an `orders` row with a `price`, an `accounts` row with a mutable `balance`. The moment you do that, two things break. Reconciliation becomes guesswork, because there is no record of the individual movements that produced the number. And reporting collapses under its own weight, because every summary has to recompute from scattered, mutable rows. Overnight jobs that once took minutes start taking hours.

```cd-compare
{ "columns": ["Single-entry", "Double-entry"],
  "rows": [
    { "label": "Records", "cells": ["One mutable balance per account", "Two immutable entries per movement"] },
    { "label": "Source of truth", "cells": ["The balance column", "The append-only entry log"] },
    { "label": "Can it drift?", "cells": ["Yes, silently", "No, debits must equal credits"] },
    { "label": "Debuggable?", "cells": ["No, the history is gone", "Yes, every movement is traceable"] },
    { "label": "Audit trail", "cells": ["Reconstructed, if you are lucky", "Built in, by construction"] }
  ] }
```

## The three entities that make it work

A double-entry ledger needs exactly three concepts, strictly separated. Modern ledger infrastructure (Modern Treasury, SDK.finance, TigerBeetle) converges on the same three, which is a good sign you are looking at something fundamental rather than a vendor opinion.

- **Accounts** are pools of value, each representing one point of view on how value changes. A customer wallet, a fee account, a treasury account, a settlement account. An account is a bucket, not a balance.
- **Entries** are the individual debits and credits. An entry never travels alone. Every movement of value produces at least one debit and one matching credit, posted against different accounts.
- **Transactions** are the orchestrators. A transaction groups the entries for a single business event and guarantees they commit together or not at all. This is where atomicity lives: no transaction can leave the ledger half-posted.

Here is a single deposit moving through the model. Watch how the value never appears from nowhere and never vanishes. It only moves.

```cd-steps
{ "title": "A $100 deposit, double-entry style",
  "steps": [
    { "title": "Event", "body": "A customer deposits $100 from their bank into your platform." },
    { "title": "Debit the source", "body": "Debit the bank settlement account $100. That is where the money came from." },
    { "title": "Credit the destination", "body": "Credit the customer wallet account $100. That is where the money landed." },
    { "title": "Balance the transaction", "body": "Debits ($100) equal credits ($100). The transaction commits as one atomic unit." },
    { "title": "Derive the balance", "body": "The customer's balance is now the sum of their entries. Nothing was overwritten." }
  ] }
```

Notice the customer balance was never set to a value. It was derived by summing entries. That is the whole trick.

## The one invariant that catches everything

There is exactly one rule you can never break: across all posted entries, total debits must equal total credits. If that equation holds, your ledger is internally consistent. If it ever fails, you have a bug, and the ledger will tell you immediately rather than six months later in a regulator's office.

This invariant is not accounting tradition for its own sake. It is your only mechanism for catching where missing cents actually went. Because every movement has a matched pair, a discrepancy cannot hide. The sum is either zero or it is not, and "not zero" is an alarm.

```cd-callout
{ "variant": "warn", "title": "Never store a balance you can edit", "body": "A balance is a derived value: the sum of an account's entries. The moment you cache it in an editable column without the entries to back it, you have reinvented single-entry and lost the ability to prove the number is correct." }
```

Two related rules fall out of this:

**Immutability.** Posted entries are never edited or deleted. If something was wrong, you post a correcting entry, which itself is a new, traceable movement. The log is append-only. This is the same discipline that makes the [stablecoin settlement work we shipped](/blog/shipping-stablecoin-rails-without-the-pain) auditable: every state transition is a new record, never an overwrite.

**Normal balance over signed numbers.** Do not use positive and negative numbers to indicate direction. Instead, each account declares whether it is normally debit-balanced or credit-balanced. This removes the constant ambiguity of "is this negative number an error or a legitimate overdraft." The account type answers the question.

## Why ledger infrastructure is back in the spotlight

For years, building a correct, scalable ledger meant years of effort and a team of senior engineers. Modern Treasury describes consumer payment companies discovering a few million dollars of unattributed balance every month, not stolen, just untraceable, because their ledger could not keep up. That is the cost of getting this layer wrong at scale.

Two forces have pushed ledgers back into the conversation. The first is purpose-built databases. TigerBeetle, for example, models the entire problem with two entities (accounts and transfers) and one invariant (every debit has an equal and opposite credit), enforces balance limits inside the database to avoid round-trips, and treats immutability as non-negotiable: reversals are new transfers, never deletions. It is the 1494 model, compiled for OLTP throughput.

The second force is agentic payments. When autonomous agents start moving money at machine speed and machine volume, every weakness in your ledger gets hit thousands of times an hour. We covered the [protocols agents use to settle payments](/blog/agentic-payments-x402-ap2-mpp) separately, but the ledger underneath them has to obey the exact same 500-year-old rule. An agent with a retry bug and a single-entry ledger is a very expensive way to lose track of money.

The lesson is the same whether your money moves over card rails, stablecoins, or an agent's API call. The ledger is not the boring part you bolt on later. It is the part that determines whether you can ever prove your numbers are right. If you are designing the money layer of a product and want a second set of eyes on the ledger model before it hardens into production, [tell us what you are building](/#contact) and we will dig in.


# Upgradeable smart contracts: proxy patterns, trade-offs

URL: https://codedecoders.io/blog/upgradeable-smart-contracts-proxy-patterns
Published: 2026-06-29 | Category: Web3 | Tags: proxy pattern solidity, UUPS proxy, transparent proxy, diamond pattern, contract storage collision

How upgradeable smart contracts work, the transparent, UUPS, and diamond proxy patterns, the storage collisions and admin-key risks, and when not to use them.

A deployed smart contract is supposed to be immutable. That is the entire pitch: the code on chain is the code that runs, forever, and nobody can change it on you. Upgradeable smart contracts deliberately break that promise so teams can patch bugs and ship new features after launch. The mechanism is a proxy: a thin contract that holds your data and forwards every call to a separate logic contract you can swap out later.

That swap is the whole point, and it is also where the risk lives. Upgradeability buys you flexibility and trades away the immutability guarantee your users thought they had. It introduces an admin key that can rewrite your contract's behavior, and a class of storage bugs that can silently corrupt every balance you hold. This is a foundational tour of the three proxy patterns in common use (transparent, UUPS, and diamond), what each one trades off, and when you should not make a contract upgradeable at all.

## How a proxy actually works

The pattern splits one logical contract into two deployed contracts. The **proxy** is the address users interact with, and it holds all the state: balances, owners, mappings, everything. The **implementation** (or logic) contract holds the functions but stores nothing meaningful itself. When a call hits the proxy, its `fallback` function uses `delegatecall` to run the implementation's code in the proxy's own storage context. The logic is borrowed; the memory stays with the proxy.

`delegatecall` is the load-bearing primitive. A normal call runs the target's code against the target's storage. A `delegatecall` runs the target's code against the *caller's* storage. So when the proxy delegatecalls into the implementation, the implementation's `transfer` function reads and writes the proxy's slots. Upgrading is then just pointing the proxy at a new implementation address. The data never moves, the proxy address never changes, and users see a zero-downtime hot swap.

```cd-steps
{ "title": "What happens on a single proxy call",
  "steps": [
    { "title": "User calls the proxy", "body": "A wallet sends a transaction to the proxy address, the only address users ever know." },
    { "title": "Fallback fires", "body": "The proxy has no matching function, so its fallback() runs and reads the current implementation address from a fixed storage slot." },
    { "title": "delegatecall", "body": "The proxy delegatecalls the implementation. The logic executes, but every SLOAD and SSTORE hits the proxy's storage, not the implementation's." },
    { "title": "Return", "body": "Return data is bubbled back to the user. To upgrade later, an admin just writes a new implementation address into that fixed slot." }
  ] }
```

## The three proxy patterns

The patterns differ mainly in *where the upgrade logic lives* and *how flexible the swap is*. Here is the trade-off at a glance, then the detail.

```cd-compare
{ "columns": ["Transparent", "UUPS", "Diamond (EIP-2535)"],
  "rows": [
    { "label": "Upgrade logic lives in", "cells": ["The proxy", "The implementation", "A facet behind the proxy"] },
    { "label": "Per-call gas overhead", "cells": ["Higher (admin check every call)", "Lower", "Selector lookup per call"] },
    { "label": "Bricking risk", "cells": ["Low", "Higher (forget the upgrade fn, lose upgradeability)", "Medium (facet management is complex)"] },
    { "label": "24KB size limit", "cells": ["Constrained", "Constrained", "Bypassed (split across facets)"] },
    { "label": "Best for", "cells": ["Simple, stable contracts", "Most new projects", "Large modular systems"] }
  ] }
```

### Transparent proxy

In the transparent pattern, the admin upgrade functions live in the proxy itself. The proxy inspects `msg.sender` on every call: if the caller is the admin, it runs the proxy's own admin functions; if not, it delegates the call to the implementation. That `msg.sender` routing exists to prevent a *function selector clash*, where an admin function and a business function happen to share the same 4-byte selector and the proxy cannot tell them apart. The cost is real: the proxy loads the admin address from storage on every single call, so every interaction pays a little extra gas. It is the oldest and most battle-tested pattern, and a reasonable default for a simple contract that rarely changes.

### UUPS proxy

UUPS (Universal Upgradeable Proxy Standard, EIP-1822) moves the upgrade logic into the implementation contract instead. The proxy becomes a minimal forwarder, so normal calls are cheaper, and the selector-clash problem disappears because the Solidity compiler refuses to let one contract define two functions with the same selector. OpenZeppelin now recommends UUPS over transparent for most new projects.

The trade-off is a sharper edge. Because the `upgradeTo` function lives in the implementation, every future implementation must keep including it. Ship a version that forgets the upgrade function and the proxy is frozen at that logic forever, with no way to upgrade out of the mistake. UUPS gives you cheaper calls in exchange for a footgun you have to remember not to pull.

### Diamond pattern

The diamond pattern (EIP-2535) generalizes the idea. Instead of one implementation, the proxy holds a mapping from function selectors to multiple implementation contracts called *facets*. A call looks up which facet owns its selector and delegatecalls that one. This buys two things: you sidestep the 24KB contract size limit (EIP-170) by spreading logic across facets, and you can upgrade a single function without redeploying everything. The price is complexity. Facet management, shared storage layout across facets, and the tooling around it are significantly harder to reason about and audit, so diamonds make sense for large modular systems and are overkill for most.

## Storage collisions: the bug that eats your state

The most dangerous failure mode in upgradeable contracts is the storage collision, and it is dangerous precisely because nothing reverts. Solidity assigns state variables to sequential 32-byte slots starting at slot 0. The proxy and the implementation share that same slot numbering because they share storage. So two rules govern everything.

First, the implementation must never declare a state variable in a slot the proxy uses for its own bookkeeping (like the implementation address). This is solved by ERC-1967, which mandates that the implementation address, admin, and beacon live in specific pseudo-random slots derived from a hash, far away from slot 0 where your business variables sit. Every modern proxy library uses these slots, so you rarely think about it, but it is the reason the two contracts can share storage without stepping on each other.

Second, between versions, you can only *append* new variables. If V2 inserts a new variable at the top of the list or reorders existing ones, every variable below it shifts to a different slot. Your `balances` mapping now reads from where `owner` used to be. No error, no revert, just silently corrupted state across every account. Two safeguards help: declare a `uint256[50] __gap` in upgradeable base contracts to reserve slots for future fields, and run OpenZeppelin's upgrade plugin, which diffs the old and new layouts and blocks an upgrade that would shift a slot.

There is one more trap that has cost the industry real money: the uninitialized implementation. Upgradeable contracts cannot use constructors (a constructor runs in the implementation's storage, not the proxy's), so they initialize through an `initialize()` function instead, guarded to run only once. If the implementation contract itself is left uninitialized, an attacker can call its `initialize()` directly, become its owner, and then trigger a `selfdestruct` while a `delegatecall` is in flight, wiping the logic the proxy depends on.

```cd-callout
{ "variant": "warn", "title": "Initialize your implementation, or someone else will", "body": "The 2017 Parity multisig freeze locked roughly 150 million dollars of ETH after an attacker initialized an uninitialized library contract and called selfdestruct. Wormhole, Harvest Finance, and others hit variants of the same uninitialized-proxy bug. Lock the implementation with _disableInitializers() in its constructor." }
```

## When a contract should not be upgradeable

Upgradeability is not free, and treating it as a default is a mistake. Every upgradeable contract has an admin key, and that key can rewrite the rules: change fees, mint tokens, redirect funds. To your users, "this contract is upgradeable" means "the team can change what this does after I deposit." For a contract that custodies value, that admin key is now the single most attractive target in your system, and how you protect it (a multisig, a timelock, a governance vote) becomes part of your security model whether you planned for it or not.

So weigh it honestly. A contract whose rules must be credibly fixed (a token with a capped supply, a trustless escrow, a vault whose terms users are relying on) is often better off immutable, with new versions shipped as new contracts and users migrating by choice. Upgradeability earns its keep where the logic is genuinely expected to evolve and the team is trusted to govern the key: protocol contracts under active development, systems with regulatory requirements that shift, anything where a bug fix without a full redeploy and migration is worth the added attack surface. The same trade-off logic shows up across on-chain design. We worked through it for asset issuance in [real-world asset tokenization architecture](/blog/real-world-asset-tokenization-architecture), where custody and oracle assumptions decide whether the design holds, and for money movement in [shipping stablecoin rails without the pain](/blog/shipping-stablecoin-rails-without-the-pain), where the boring infrastructure is what actually makes it work.

The honest framing: upgradeability is a governance decision wearing an engineering costume. Pick the simplest pattern that fits (transparent for stable contracts, UUPS for most new work, diamond only when size or modularity forces it), lock down the storage layout and the admin key, and be deliberate about which contracts get to change at all. If you are weighing those trade-offs for a system that custodies real value and want a second set of eyes before you commit, [get in touch](/#contact).


# Payment reconciliation: build an engine that survives

URL: https://codedecoders.io/blog/payment-reconciliation-fintech
Published: 2026-06-22 | Category: Fintech | Tags: multi-currency reconciliation, settlement file matching, payment reconciliation engine, acquirer reconciliation, payments data integrity

A practical guide to building a payment reconciliation engine that survives partial settlement files, PSP fees, and FX, plus the matching ladder it needs.

Most payments stacks ship without a reconciliation engine. The team integrates an acquirer, money starts moving, and reconciliation becomes a spreadsheet someone updates on Mondays. That works until you add a second processor, a second currency, and a finance lead who asks why the settled amount is 0.4% short of what the dashboard says. Then reconciliation stops being a chore and becomes an architecture problem.

This is a practical guide to building a payment reconciliation engine that holds up at scale: the data model that makes matching possible, the matching ladder that handles missing IDs, and the failure modes that bite once you cross processors, currencies, and time zones.

## What reconciliation actually has to prove

Reconciliation answers one question for every transaction: did the money you expected actually arrive, in the amount you expected, net of the fees you agreed to? You have two sides. On one side, your own ledger of charges (what you told the customer they paid). On the other, the settlement files your acquirers send (what the money movement actually was, after fees and FX). The engine's job is to match those two sides and explain every row that does not line up.

The reason this is hard is not the matching. It is that the two sides speak different dialects. Your ledger thinks in your transaction IDs and your presentment currency. The settlement file thinks in the acquirer's IDs, its own fee structure, its settlement currency, and a value date that can lag the transaction by days. Reconciliation is a translation problem before it is a matching problem.

## Step 1: normalize every file into one event table

Do not try to reconcile files directly against your ledger. Settlement files differ by acquirer, by region, and by year (acquirers change their layouts without warning). Instead, parse every file into a single normalized table, often called `settlement_events`, and reconcile against that.

Each row should carry:

- Source identity: acquirer, file name, external transaction ID, original row stored as JSON so you can re-parse later.
- Money in minor units: gross, fees, and net as integers (cents, not floats). Floating point on money is a bug waiting for a rounding audit.
- Currency context: transaction currency, settlement currency, and the FX rate, stored as three separate fields.
- Time: the event timestamp and the value date (when funds actually settle).

Keeping the raw row as JSON matters more than it looks. When an acquirer quietly adds a column or shifts a fee into a new field, you re-parse history instead of begging them to resend six months of files. Version your parsers and keep the old ones, so a file from March still parses in September after the layout changed.

From this one table you project two read models: your internal charges and the settled transactions. Now reconciliation is a SQL join between two clean shapes, not a fight with raw files.

## Step 2: match on IDs first, then climb a ladder

The happy path is an exact match on the acquirer's external transaction ID. In a healthy integration that covers the large majority of rows. The interesting work is what you do for the rest, because external IDs go missing constantly: truncated fields, fees delivered in a separate file with no ID, refunds that reference the original transaction under a different column name.

Do not jump straight to fuzzy matching. Build a deterministic ladder and only descend when the rung above fails.

```cd-steps
{ "title": "The reconciliation matching ladder",
  "steps": [
    { "title": "Exact external ID", "body": "Join settlement rows to your charges on the acquirer's transaction ID. This is the bulk of a healthy integration." },
    { "title": "Composite key", "body": "When the ID is missing, match on amount + currency + acquirer within a value-date window (T+2 is common), tightened with card last-4 when present." },
    { "title": "Aggregate match", "body": "For split or batched rows, sum settlement events per transaction before comparing, so one charge can match many rows." },
    { "title": "Exception queue", "body": "Anything still unmatched lands in a typed exception bucket with a runbook, not a silent drop." }
  ] }
```

A fallback that combines amount, currency, a value-date window, acquirer, and card last-4 will recover the overwhelming majority of rows that lack a usable external ID. The point is that each rung is still deterministic and explainable. You are not guessing, you are widening the key in a controlled way and recording which rung made the match, so a human can audit any decision later.

Machine learning has a place here, but later. Rules break when patterns change and need manual upkeep; ML learns from historical matches and copes with messier data. But a learned matcher you cannot explain is a liability in a financial control, and most teams hit an accuracy ceiling around 85 to 92% not because the algorithm is weak but because the system lacks context (the parent transaction, the fee schedule, the prior match history). Add the context layer before you reach for the model.

```cd-compare
{ "columns": ["Exact ID", "Composite key", "ML-assisted"],
  "rows": [
    { "label": "How it matches", "cells": ["Acquirer transaction ID", "Amount + currency + window + last-4", "Learned from match history"] },
    { "label": "Explainable", "cells": ["Fully", "Fully", "Partially"] },
    { "label": "Breaks when", "cells": ["ID missing or truncated", "Many same-amount txns in window", "Patterns drift without retraining"] },
    { "label": "Use it for", "cells": ["The default path", "Missing-ID recovery", "Long-tail exceptions, with a human in the loop"] }
  ] }
```

## Step 3: model fees, FX, and refunds as first-class events

Three things turn a clean matcher into a wrong one if you model them lazily.

**Fees.** They arrive inline on the transaction row or in a separate fee file, depending on the acquirer. Normalize both as distinct event types and aggregate them per transaction in the read model. Do not maintain two parallel code paths, one for inline fees and one for separate ones. One event shape, aggregated.

**FX.** This is where money silently disappears. Store the transaction currency, the settlement currency, and the conversion rate as three fields, and never store only the converted amount. The moment you collapse FX into a single settled number, you lose the ability to prove where a 0.3% gap came from, and you will be asked.

```cd-callout
{ "variant": "warn", "title": "Never store only the converted amount", "body": "Keep transaction currency, settlement currency, and FX rate as separate fields, and apply one rounding rule (banker's rounding) once and permanently. Re-deriving an FX figure later, with a different rounding step, manufactures the exact discrepancies reconciliation exists to catch." }
```

**Refunds and chargebacks.** Treat them as new events with their own external IDs, not as updates to the original row. The original charge stays immutable; the refund references it. Acquirers name that parent reference inconsistently (`original_transaction_id`, `orig_txn`, and worse), so normalize the field on ingest. An append-only event log here is not academic purity, it is what lets you reconstruct the full history of a disputed transaction months later.

If you are reconciling on-chain settlement too, the same append-only discipline carries over. We wrote about keeping fiat and on-chain ledgers honest in [shipping stablecoin rails without the pain](/blog/shipping-stablecoin-rails-without-the-pain), and the integrity rules are the same on either rail.

## Step 4: bucket the exceptions, do not hide them

A reconciliation run should never produce a single pass or fail. It should produce a diff that sorts every unmatched row into a typed bucket, each with its own owner and runbook:

- Missing in settlement: in your ledger, not in the file (money you expected that has not arrived).
- Unknown in settlement: in the file, not in your ledger (money that arrived you cannot explain).
- Currency mismatch: matched transaction, wrong settlement currency.
- Gross mismatch: amounts disagree beyond tolerance.
- Fee mismatch: the fee differs from the agreed schedule.
- Matched: the boring, healthy majority.

Set an amount tolerance so legitimate rounding from FX and fee math does not flood the queue. A band like plus or minus 0.01% or 0.05 in minor units, whichever is smaller, absorbs noise without hiding real gaps. Anything outside the band is a real exception, not rounding.

## Step 5: verify with three metrics, every day

You cannot eyeball reconciliation health at scale. Put three numbers on a dashboard and alert on them:

1. Match rate by the T+1 close. A healthy integration sits above 99%. A drop is the first sign an acquirer changed a file layout.
2. Age of the oldest unmatched item per bucket. Reconciliation debt is like any debt: the old items are the dangerous ones.
3. Net currency delta per acquirer. It should stay inside your rounding tolerance. A drift means an FX or fee assumption is wrong somewhere.

When one of these moves, it is usually not a data glitch. It is a dependency you did not model. The most useful framing we have seen is that persistent reconciliation exceptions are a diagnostic: they signal [hidden coupling in distributed systems](https://dev.to/doomhammerhell/hidden-coupling-in-distributed-financial-systems-dependencies-you-didnt-know-you-had-3hc6), where "completed" means one thing to your ledger and another to the acquirer, or where a settlement window shifted under load and broke an ordering assumption nobody wrote down.

## Why this matters more every quarter

The reconciliation surface is growing, not shrinking. Stripe's Sessions 2026 push into multicurrency treasury, instant settlement, and adaptive local pricing makes multi-currency, multi-rail money movement the default rather than the exception. Every new currency and every new rail multiplies the edge cases your engine has to absorb. The same pressure shows up in agentic commerce, where machine-driven micropayments add volume and new settlement paths; we covered those rails in [what x402, AP2 and MPP mean for builders](/blog/agentic-payments-x402-ap2-mpp), and the tokenized-asset world raises the same integrity bar we discussed in [real-world asset tokenization architecture](/blog/real-world-asset-tokenization-architecture).

The teams that treat reconciliation as a core service from day one (a normalized event table, a deterministic matching ladder, typed exceptions, and three honest metrics) spend their time resolving real discrepancies. The teams that bolt it on later spend their time arguing about whose number is right. If you are designing the part of your payments stack nobody architects for, we would be glad to [talk through your reconciliation architecture](/#contact) before the second processor and the second currency force the question.


# Real-world asset tokenization: architecture and trade-offs

URL: https://codedecoders.io/blog/real-world-asset-tokenization-architecture
Published: 2026-06-15 | Category: Web3 | Tags: RWA tokenization, asset-backed tokens, on-chain RWA, tokenized treasuries, stablecoin RWA

A field guide to real-world asset tokenization architecture, the token standards, oracle and custody assumptions, and the trade-offs that decide if it works.

Real-world asset tokenization is the practice of representing an off-chain asset, a Treasury bill, a building, a private credit loan, as a transferable token on a blockchain. The pitch is liquidity and programmability: fractional ownership, 24/7 settlement, and assets that plug straight into on-chain finance. The reality is more sober. The token is the easy part. Everything that makes the token mean something, custody, compliance, valuation, and the legal right to redeem, lives off-chain and has to be wired in without recreating the problems you were trying to escape.

This is a foundational guide to how RWA tokenization is actually built: the architecture layers, the token standards worth knowing, the oracle and custody assumptions baked into every design, and the trade-offs that decide whether tokenization buys you anything real.

## Why this is suddenly worth understanding

For two years tokenized assets were a slide in a pitch deck. The numbers moved in 2025. Tokenized RWAs (excluding stablecoins) grew from roughly 5.5 billion dollars in early 2025 to around 18.6 billion by year end, and sat near 30 billion by April 2026. Tokenized US Treasuries alone went from about 5 billion in late 2024 to roughly 12.9 billion by early April 2026. BlackRock's BUIDL fund crossed 2.5 billion dollars in assets, runs across nine chains, and became usable as collateral on a major exchange.

```cd-chart
{ "type": "area", "title": "Tokenized RWA value on-chain (excl. stablecoins), $B",
  "xKey": "period",
  "series": [{ "key": "value", "label": "On-chain RWA ($B)", "color": "#5BBDF9" }],
  "data": [
    { "period": "Q1 2025", "value": 5.5 },
    { "period": "Q3 2025", "value": 11 },
    { "period": "Q4 2025", "value": 18.6 },
    { "period": "Q2 2026", "value": 30 }
  ] }
```

The growth is real, but the figures are dominated by the boring end of the spectrum: short-dated government debt and money-market funds. That is not an accident. Treasuries are the asset where the off-chain machinery (a regulated issuer, a known custodian, a daily net asset value) is simplest to attest to on-chain. The further you move from that, into real estate or illiquid credit, the harder the architecture gets. Keep that ordering in mind. It explains most of what follows.

## The three-layer architecture

Strip away the marketing and almost every serious RWA system separates into three layers, each solving a different problem.

The **token layer** holds balances and transfer logic. The **compliance layer** decides who is allowed to hold or move the token. The **oracle layer** keeps the on-chain representation in sync with off-chain reality: price or net asset value, proof that the underlying reserves exist, and triggers for events like a coupon payment or a redemption.

The mistake teams make is treating this as a token project. It is a systems-integration project where a token happens to be the user-facing object. As one practitioner put it bluntly, the token is usually the smallest part of the system; the real work is provenance, custody changes, and valuation updates.

```cd-steps
{ "title": "A transfer in a compliant RWA token",
  "steps": [
    { "title": "Transfer requested", "body": "A holder calls transfer to send tokens to another address." },
    { "title": "Compliance check", "body": "The contract queries an on-chain identity registry: are sender and receiver both verified and eligible in their jurisdiction?" },
    { "title": "Rule evaluation", "body": "Holding limits, lockups, and max-holder caps are checked. Any failure reverts the transfer." },
    { "title": "Settle", "body": "Only if every rule passes does the balance update on-chain." }
  ] }
```

That compliance gate is the structural difference between an RWA token and a plain ERC-20. In a normal token, a transfer is unconditional. In a security token, the transfer function asks permission first.

## Token standards: ERC-20, ERC-1400, ERC-3643

You can tokenize an asset with a plain ERC-20 and a whitelist mapping, and plenty of tutorials do exactly that: override the internal transfer hook so only KYC-verified addresses can hold the token. It works for a demo. It does not scale to regulated issuance, because the compliance rules live as ad hoc code inside one contract rather than as a reusable, auditable standard.

Two standards exist for the regulated case. ERC-1400 introduced partitions, separate tranches inside one token contract, which suits structured products where classes of the same asset carry different rights. ERC-3643 (originally T-REX) has become the de facto standard for permissioned tokens: an open suite of contracts with a built-in on-chain identity framework (ONCHAINID) so that only addresses meeting predefined conditions can ever hold the token. Its momentum is institutional, not just technical. A major post-trade infrastructure provider joined the ERC-3643 association in 2025, and a US regulator referenced it as a compliance-aware protocol in a no-action letter permitting a tokenized-securities pilot.

```cd-compare
{ "columns": ["ERC-20 + whitelist", "ERC-1400", "ERC-3643"],
  "rows": [
    { "label": "Built for", "cells": ["Demos, simple cases", "Structured securities", "Regulated permissioned assets"] },
    { "label": "Compliance", "cells": ["Custom code per contract", "Transfer hooks + documents", "On-chain identity + rules engine"] },
    { "label": "Identity model", "cells": ["Address whitelist mapping", "Per-partition controls", "ONCHAINID decentralized identity"] },
    { "label": "Tranches/partitions", "cells": ["No", "Yes", "Via modules"] },
    { "label": "Ecosystem", "cells": ["Universal tooling", "Moderate", "Growing institutional support"] }
  ] }
```

The takeaway: pick the standard for the buyer, not the demo. If accredited investors and a regulator will ever touch the asset, the identity-and-rules model of ERC-3643 is doing work that a whitelist mapping cannot.

## Custody and the legal wrapper, the part that is not code

Here is where most of the risk actually sits, and where it is invisible in the smart-contract repo. A token is a claim. For the claim to be worth anything, someone off-chain must hold the real asset and honor redemption. That someone is usually a special purpose vehicle (SPV) or a licensed custodian, and the token represents a beneficial interest in what the SPV holds, not direct legal ownership of the asset.

This creates a bifurcation of authority. On-chain, the contract tracks balances perfectly. Off-chain, a custodian holds the asset, an SPV holds the legal title, and a court, not a smart contract, resolves disputes. If the SPV is poorly documented or its bankruptcy-remoteness is challenged, token holders can find their redemption rights are not enforceable, no matter how clean the Solidity is. Smart contracts cannot litigate.

This is the same uncomfortable seam we wrote about in [shipping stablecoin rails without the pain](/blog/shipping-stablecoin-rails-without-the-pain): the on-chain leg is fast and deterministic, the off-chain leg is where the real failure modes hide. RWA tokenization inherits that seam and adds a legal one on top.

## Where it breaks: the trade-offs

Tokenization does not remove custody and compliance risk. It relocates it, and adds a few new failure modes of its own. The honest version of the architecture is a list of the assumptions you are taking on.

**Oracle and proof-of-reserves risk.** The token only reflects reality through an oracle. Net asset value comes from a price feed; the claim that reserves exist comes from a proof-of-reserves attestation. But a proof-of-reserves proves a historical balance at attestation time. Reserves can be drawn down between attestations, and a feed that is manipulated or stale will mint, burn, or value tokens against numbers that are wrong. The decentralization of the token does not decentralize the data feeding it.

**Redemption under stress.** Many designs use asynchronous redemption with a settlement delay (request now, settle in a few days). That is fine until everyone redeems at once. A redemption queue plus a delay is the on-chain shape of a bank run: early redeemers get full value, later ones absorb the loss if the underlying has moved. Model the queue before you ship it.

**Cross-chain settlement.** Burn-on-source, mint-on-destination bridging is convenient and dangerous. A source-chain reorg after the burn, a stuck messaging layer, or a mismatch between burn and mint can lock, lose, or duplicate value. For high-value RWAs, bridge desync is a real custody risk, not a theoretical one.

```cd-callout
{ "variant": "warn", "title": "The oracle is the weakest link", "body": "Proof-of-reserves attests to a balance at a moment in time, not continuously. Treat every off-chain feed (NAV, reserves, identity revocation) as a trust assumption you are inheriting, and design for it being wrong or late." }
```

There is also an identity-portability trap. If a holder's verification is revoked off-chain (a failed AML re-check) but that revocation does not propagate to every chain the token lives on, an ineligible address stays eligible somewhere. Compliance that is not synchronized is not compliance.

## When tokenization actually buys you something

Strip it down and the decision is simple. Tokenization is worth it when the on-chain benefits, fractional ownership, programmable settlement, composability with other on-chain finance, outweigh the cost of building and trusting the off-chain bridge. That is why tokenized Treasuries took off first: the asset is liquid and standardized, the issuer and custodian are regulated, and net asset value is easy to attest. The bridge is short and the payoff (yield-bearing collateral that moves at on-chain speed, usable in the same systems as agentic and machine-to-machine payments, like the rails behind [agentic payments with x402, AP2 and MPP](/blog/agentic-payments-x402-ap2-mpp)) is large.

The further your asset sits from that profile, illiquid, hard to value, dependent on a single custodian or a contested legal structure, the more the architecture is just an expensive wrapper around the same old custody and compliance problem. Tokenize the asset where the bridge is short and the composability is worth it. Be skeptical everywhere else.

If you are weighing whether tokenization is the right call for a specific asset or settlement flow, that trade-off (bridge cost versus on-chain payoff) is exactly the conversation worth having before you write a line of Solidity. If that is on your roadmap, [get in touch](/#contact) and we are happy to pressure-test the design.


# Claude Fable 5 and the long-horizon agent problem

URL: https://codedecoders.io/blog/claude-fable-5-long-horizon-agents
Published: 2026-06-10 | Category: AI/ML | Tags: claude fable 5, long-horizon agents, ai agent autonomy, agent reliability, agentic payments

Claude Fable 5 keeps agents coherent for hours, not minutes. What changed, why it matters for fintech automation, and where you still need a human gate.

On June 10 2026, Anthropic shipped Claude Fable 5, and the launch post hit the Hacker News front page near 1,524 points the same day. The headline is not another benchmark record. It is duration. Claude Fable 5 is built to stay coherent across long-running, ambiguous, multi-step tasks, the kind that used to drift into nonsense after twenty minutes of autonomy.

For anyone building agentic payment or fintech workflows, that one property, how long an agent stays on task before it loses the thread, decides what you can safely hand to a machine and where you still need a person in the loop. Here is what actually shipped, why the longer horizon matters, and what to change if your agents touch money.

## What actually shipped

Two models, one engine. Claude Fable 5 (model id `claude-fable-5`) is the widely released one. Claude Mythos 5 (`claude-mythos-5`) is the same capability behind Project Glasswing, an invite-only program for partners and biology researchers. Both carry a 1M-token context window and up to 128K tokens of output.

Pricing is $10 per million input tokens and $50 per million output tokens, which Anthropic notes is less than half what the earlier Mythos Preview cost. Vercel added it to its AI Gateway the same week as `anthropic/claude-fable-5`, with no platform markup on inference. So it is cheap to reach, but the per-token rate is roughly double Opus-tier, and a long autonomous run burns far more tokens than a single prompt.

A few API behaviors changed enough to break code written for the Opus family:

- **Thinking is always on.** Omit the `thinking` parameter (or send `{type: "adaptive"}`). An explicit `disabled` or a fixed `budget_tokens` returns a 400. You control reasoning depth with `output_config.effort` (`low` through `xhigh` and `max`).
- **New tokenizer.** The same text costs roughly 30% more tokens than on Opus 4.8, so re-baseline your token budgets with `count_tokens` instead of reusing old numbers.
- **A `refusal` stop reason.** Safety classifiers can decline a request with an HTTP 200 and `stop_reason: "refusal"`, so check `stop_reason` before you read `content` or you will index into an empty array.
- **30-day data retention is required.** Fable 5 is not available under zero data retention; non-conforming orgs get a 400.

```cd-compare
{ "columns": ["Claude Fable 5", "Claude Opus 4.8"],
  "rows": [
    { "label": "Best for", "cells": ["Long-horizon, ambiguous, multi-step work", "Fast, high-quality coding and agentic loops"] },
    { "label": "Sustained autonomy", "cells": ["Hours per task, parallel sub-agents", "Minutes to tens of minutes"] },
    { "label": "Thinking config", "cells": ["Always on, omit the param", "Adaptive, can be disabled"] },
    { "label": "Relative cost", "cells": ["$10 / $50 per 1M tokens", "About half the per-token cost"] },
    { "label": "Tokenizer", "cells": ["New, ~30% more tokens per text", "Opus-tier"] },
    { "label": "Data retention", "cells": ["30 days required", "Supports zero data retention"] }
  ] }
```

## What "long-horizon" actually means

The interesting claims are about endurance, not raw IQ. In Anthropic's own writeup, Stripe used Fable 5 to compress a 50-million-line Ruby migration that normally takes more than two months into a single day. Ethan Mollick's hands-on account describes the model building research software it named "Concord" that ran for nine and a half hours straight against a 19-page design spec, then spinning up adversarial groups of sub-agents that researched and checked each other's results.

That is the step change. Earlier agents were sprinters. They did one well-scoped thing and handed back. Fable 5 is closer to a contractor you brief and leave alone for an afternoon. Mollick's line captures the shift: "I no longer steer; I commission." Vercel positions it the same way, for "long-running, ambiguous, multi-step tasks" that previously needed frequent human oversight, dispatching parallel sub-agents and holding output quality across a multi-day run.

For a builder, the useful question is not "is it smarter." It is "how far can the leash extend before the work goes sideways." Fable 5 moves that leash from minutes to hours.

## Why a longer horizon moves the automation boundary

A reliable horizon is the real input to any automation decision. You automate the part of a workflow the machine can finish before it drifts, and you gate the rest behind a human. When the coherent horizon was twenty minutes, an agent could draft a reconciliation report or propose a payout batch, then a person had to take over. Push the horizon to several hours and the agent can plan a migration, run it, test it, and surface only the exceptions.

In money movement that is exactly the work that was stuck. Reconciling a day of stablecoin settlements across three providers, chasing a mismatched ledger entry through five systems, or migrating a payments service off a legacy schema are all multi-step, ambiguous, and long. They are also where the three [agentic payment standards we covered, x402, AP2, and MPP](/blog/agentic-payments-x402-ap2-mpp), are pushing real spend through autonomous agents. A longer horizon means more of that pipeline can run unattended.

The catch is that the blast radius scales with the leash. An agent that can act for hours against live systems can also be wrong for hours against live systems.

```cd-callout
{ "variant": "warn", "title": "The leash and the blast radius grow together", "body": "An agent with wallet access that stays coherent for hours is also an agent that can move money in the wrong direction for hours before anyone looks. A longer horizon is a capability and a liability at the same time. Scope the wallet, not just the prompt." }
```

## The new failure mode: you commission, you don't steer

The honest part of Mollick's account is the opacity. The details of the model's decision making are not shown, so it makes hundreds of judgment calls across a multi-hour run with no human visibility into any of them. He also notes the guardrails trip at the faintest hint of a security problem, which is the over-cautious mirror of the same black box. You get a finished result and a `refusal` you cannot always explain.

For money movement that trade is dangerous in a specific way. A confident, wrong agent that ran for six hours is worse than one that quit after five minutes, because it had time to compound the mistake across many steps and you have no trace of where it went wrong. Longer autonomy does not remove the need for verification. It raises the stakes on it. The lesson is the same one we learned [shipping stablecoin rails](/blog/shipping-stablecoin-rails-without-the-pain): the model is fast, the boring infrastructure around it is what keeps it safe.

## What to do if your agents touch money

Treat the longer horizon as more rope, not less supervision. Concretely:

```cd-steps
{ "title": "A safe agentic money-movement loop",
  "steps": [
    { "title": "Plan, then dry-run", "body": "Let the agent produce a plan and a simulated result first. Diff the simulation against expectations before anything executes." },
    { "title": "Gate the irreversible step", "body": "Put a human approval (or a hard policy check) in front of any action that moves real funds above a threshold. The agent prepares; a person or rule commits." },
    { "title": "Execute idempotently", "body": "Every money-moving call carries an idempotency key. An agent that retries a long task must never double-pay." },
    { "title": "Reconcile against the ledger", "body": "After the run, reconcile actual transfers against the plan. Unmatched entries are exceptions for a human, not silent successes." }
  ] }
```

On the API itself, a few settings matter more than they did:

- **Spend `effort` deliberately.** Run intelligence-sensitive money logic at `high` or `xhigh` with the full spec given up front; drop to `low` for cheap sub-tasks. The newer Task Budgets beta lets you hand the model a token countdown for a whole loop (minimum 20,000) so it self-moderates instead of running until your `max_tokens` ceiling cuts it off mid-thought.
- **Handle `refusal` as a first-class outcome.** Branch on `stop_reason` and wire a fallback to another model rather than crashing, so a tripped safety classifier does not silently strand a payment run.
- **Scope the wallet, not the prompt.** Per-run spend caps, allowlisted payees, and revocable, short-lived credentials matter more than prompt instructions. The model can run for hours; the credential should not let it run off a cliff.

The horizon got longer, which is genuinely useful. What did not change is that an autonomous agent moving money needs a verification gate, idempotency, and a reconciliation step, exactly the unglamorous plumbing that decides whether the demo survives contact with production. If you are weighing where a longer-horizon agent belongs in your payment stack, [tell us what you are trying to automate](/#contact) and we will help you draw the line between what to hand the agent and what to keep behind a human.


# Agentic payments: what x402, AP2 and MPP mean for builders

URL: https://codedecoders.io/blog/agentic-payments-x402-ap2-mpp
Published: 2026-06-05 (updated 2026-07-07) | Category: AI/ML | Tags: agentic payments, x402, stablecoins, ai agents, fintech, acp

Four agentic payment standards launched in a year. What x402, Google AP2, Stripe MPP, and OpenAI's ACP each do, how they differ, and what you need to build.

Four competing standards for agentic payments shipped within little more than a year. [x402](https://www.x402.org/) from Coinbase, [AP2](https://ap2-protocol.org/) from Google, and [MPP](https://docs.stripe.com/payments/machine/mpp) from Stripe and Tempo all use the dormant HTTP 402 status code and all settle in stablecoins. The [Agentic Commerce Protocol (ACP)](https://www.agenticcommerce.dev/) from OpenAI and Stripe targets a different layer entirely: structured retail checkout between agents and merchants, settled over the merchant's existing payment rails. If you are building anything that involves an AI agent spending money, you need to decide which one, or which combination, to support.

This is the current state of each protocol, where they actually differ, and the four things you must build correctly regardless of which you choose.

## Why this converged on HTTP 402

HTTP 402 ("Payment Required") has been reserved since 1991. For thirty years no one used it. The specification simply said "reserved for future use." It turns out the future is autonomous agents that need to pay per API call without a human's credit card in the loop.

Three of the four protocols share the same basic flow (ACP is the exception, more on that below):

```cd-steps
{ "title": "The shared 402 payment handshake", "steps": [
  { "title": "Request", "body": "The agent sends a request to a paid endpoint." },
  { "title": "402 Payment Required", "body": "The server responds with 402 and payment instructions: token, amount, and destination." },
  { "title": "Authorize & retry", "body": "The agent authorizes the payment and retries the request with proof attached." },
  { "title": "Deliver", "body": "The server returns the resource along with a receipt." }
] }
```

No accounts. No OAuth flows. No subscription billing. The payment substitutes for the API key.

By April 2026, x402 alone had processed 165 million transactions across approximately 69,000 active agents, at an average transaction size under $0.31. Roughly 98.6% of machine payments settled in USDC, according to [Keyrock's "Who Pays the Agent?" report](https://keyrock.com/who-pays-the-agent/). The micropayment thesis is proving out faster than most expected.

## The four protocols

### x402, Coinbase's open standard

Coinbase open-sourced x402 in May 2025, then donated it to the [x402 Foundation at the Linux Foundation](https://www.linuxfoundation.org/press/linux-foundation-is-launching-the-x402-foundation-and-welcoming-the-contribution-of-the-x402-protocol) in April 2026. The foundation's launch members include Adyen, AWS, American Express, Circle, Fiserv, Google, Mastercard, Microsoft, Polygon Labs, Shopify, Solana Foundation, Stripe, and Visa, a deliberately broad coalition.

The protocol is deliberately minimal. A server returns a `402` with a JSON body describing the payment requirements (token, amount, destination address). The client pays, attaches the payment proof in a header, and retries. No custody, no session state, no central intermediary.

x402 currently runs primarily on Base (Coinbase's Ethereum L2), settling in USDC. The Coinbase CDP wallet infrastructure handles the agent-side wallet management. Coinbase launched a Bazaar MCP server to make paid APIs discoverable by agents.

### AP2, Google's mandate-based protocol

Google's Agent Payments Protocol (AP2) takes a different approach. It extends the Agent2Agent (A2A) protocol and Model Context Protocol (MCP) with a **mandate** system: cryptographically-signed digital contracts that establish what an agent is authorized to buy and under what conditions.

AP2 distinguishes between two authorization modes:

**Human-present**: An Intent Mandate captures the user's request. A Cart Mandate is generated from the specific items and prices. The user approves the cart before payment. This is closer to a traditional checkout with an agent doing the browsing.

**Human-absent (delegated)**: The user pre-authorizes conditions upfront ("buy concert tickets under $150 when they go on sale"). The agent generates a Cart Mandate autonomously when conditions are met and executes the purchase. The mandate creates a non-repudiable audit trail.

AP2 is payment-agnostic, it supports cards, stablecoins, real-time bank transfers, and stored value. Google collaborated with Coinbase, the Ethereum Foundation, and MetaMask on an **A2A x402 extension** that handles stablecoin settlement within the AP2 framework. The protocol [launched with 60+ partner organizations](https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol) including Adyen, American Express, Mastercard, PayPal, Worldpay, Etsy, and Intuit.

### MPP, Stripe and Tempo's session-aware protocol

Stripe co-authored the Machine Payments Protocol with Tempo in March 2026. Tempo is a purpose-built Layer-1 blockchain, developed with Paradigm, designed specifically for high-frequency stablecoin transactions. MPP runs on Tempo mainnet, settling in USDC.

MPP uses the same HTTP 402 handshake but adds a dual-method architecture. A server can advertise both a crypto method (on-chain via Tempo) and a fiat method (via Stripe's Shared Payment Tokens). The client selects:

```
HTTP/1.1 402 Payment Required
WWW-Authenticate: Payment id="chal_abc123", method="tempo", intent="charge", ...
WWW-Authenticate: Payment id="chal_def456", method="stripe", intent="charge", ...
```

This means an agent with a funded USDC wallet uses the Tempo path; a traditional service consuming the API can use the SPT/card path. Same endpoint, two rails.

Stripe's PaymentIntents API handles both paths. Crypto payments auto-capture when funds settle on-chain. Visa, Stripe, and Lightspark have already extended MPP to cards, wallets, and Bitcoin Lightning respectively.

### ACP, OpenAI and Stripe's checkout standard

The Agentic Commerce Protocol is the odd one out. It does not use HTTP 402 and it is not built for micropayments. [Announced by OpenAI and Stripe in September 2025](https://stripe.com/blog/developing-an-open-standard-for-agentic-commerce), ACP standardizes how an AI agent completes a retail checkout against a merchant's existing commerce backend. It is the protocol behind Instant Checkout in ChatGPT, which launched with Etsy sellers and expanded to Shopify merchants including Glossier, Vuori, and SKIMS. The spec is Apache 2.0 licensed and maintained by OpenAI and Stripe, with the reference documentation in [OpenAI's agentic commerce docs](https://developers.openai.com/commerce/).

ACP has three building blocks. Product feeds give the agent a machine-readable catalog so it can surface items in conversation. Checkout sessions are a REST flow the merchant implements: the agent creates a session, updates the cart and fulfillment options, and completes it. Delegate payments move the money: the agent passes a Shared Payment Token instead of raw card credentials. If that name sounds familiar, it is the same Stripe primitive that MPP's fiat path uses.

The authorization model is human-present by design. The buyer confirms each purchase inside the chat, and the SPT the agent carries is scoped to a single merchant, capped at a maximum amount, and expires with the checkout session. There is no standing delegation and no autonomous spending. The merchant stays merchant of record, keeping control of pricing, fulfillment, returns, and fraud decisions.

Settlement is deliberately boring: the merchant's existing payment processor charges the token like any other payment, on card rails, with card economics. The spec is processor-agnostic even though Stripe shipped the first SPT implementation. No wallets, no stablecoins, no chains.

Choose ACP when you sell physical goods, digital products, or subscriptions and want them purchasable where agent traffic already is. It solves distribution, not machine-to-machine metering. An agent paying $0.003 per API call has no use for a checkout session; a merchant who wants ChatGPT users to buy without leaving the conversation has no use for a USDC wallet. The protocols barely overlap, which is why most stacks will end up with both.

## Protocol comparison

```cd-compare
{ "columns": ["x402", "AP2 (Google)", "MPP (Stripe + Tempo)", "ACP (OpenAI + Stripe)"],
  "rows": [
    { "label": "Origin", "cells": ["Coinbase → x402 Foundation (Linux Foundation)", "Google + 60 partners", "Stripe + Tempo (Paradigm-backed)", "OpenAI + Stripe"] },
    { "label": "Auth model", "cells": ["Stateless: payment = credential", "Cryptographic mandates (Intent + Cart)", "Signed challenges; SPTs for fiat", "Buyer confirms in chat; scoped SPTs"] },
    { "label": "Human-absent delegation", "cells": ["Yes, by default", "Yes, via Intent Mandate", "Yes, via SPT pre-authorization", "No, human-present checkout"] },
    { "label": "Settlement", "cells": ["USDC on Base (primarily)", "Payment-agnostic: cards, stablecoins, RTP", "USDC on Tempo; fiat via Stripe SPTs", "Merchant's existing rails (cards via PSP)"] },
    { "label": "Audit trail", "cells": ["On-chain receipts", "Non-repudiable mandate chain", "Receipts + Stripe Dashboard", "Checkout sessions + order webhooks"] },
    { "label": "Fiat support", "cells": ["No", "Yes", "Yes (cards via SPT)", "Yes (native)"] },
    { "label": "Governance", "cells": ["Linux Foundation", "Open standard; FIDO Alliance", "Open spec; Stripe-anchored", "Open spec (Apache 2.0); OpenAI + Stripe"] },
    { "label": "Traction (Apr 2026)", "cells": ["165M txns, 69k agents, ~$50M", "Launched; 60+ partners", "Launched Mar 2026; preview", "Live in ChatGPT Instant Checkout (Etsy, Shopify)"] }
  ] }
```

The key difference is authorization philosophy. x402 is minimal by design: if the agent can pay, it can access the resource. AP2 adds a formal consent and delegation layer that creates an auditable record of what the user authorized. MPP sits between them, structured enough for compliance, simple enough to implement in a few lines via the Stripe SDK. ACP sidesteps delegation entirely: a human confirms each purchase, and the token the agent carries is scoped to one merchant, one amount, one expiry window.

## What you actually have to build

Whatever protocol you support, four components matter. None of them are exciting. All of them break in production.

### 1. Authorization and mandate management

For human-absent flows, you need a way for users to define spending policies upfront. At minimum: per-category spending caps, a maximum per-transaction amount, and a hard daily/weekly ceiling. For AP2, those constraints are encoded in the Intent Mandate. For ACP, the Shared Payment Token enforces merchant, amount, and expiry scoping at the token layer. For x402 and MPP, you enforce them yourself in the agent runtime before allowing a payment to proceed.

Do not skip this. An agent with open-ended wallet access and a bug is an expensive bug.

### 2. Settlement and custody

The three 402-based protocols currently bias toward USDC on Ethereum-family chains (Base for x402, Tempo for MPP, chain-agnostic for AP2's crypto path). Your agent needs a funded wallet. The questions are: who holds the keys, how is the wallet topped up, and what is the float strategy.

Self-custody gives you control but puts key management in your stack. MPC wallets (Coinbase CDP, Privy via Stripe) delegate that complexity to a custodian. The same trade-offs we covered when [building stablecoin payment rails](/blog/shipping-stablecoin-rails-without-the-pain) apply here, custody is the decision you will keep revisiting as volume grows.

For fiat flows (MPP's SPT path, ACP checkouts, AP2's card rail), you are back in traditional payment processing territory. Stripe's dashboard handles reconciliation. The complexity is lower but fees are higher.

### 3. Idempotency

Agents retry. Networks drop packets. The 402→pay→retry loop is inherently at-risk of double payments if your server or client is not idempotent.

On the server side: deduplicate payment proofs. An x402 payment receipt is an on-chain transaction hash, treat it as the idempotency key. Cache it with a TTL that exceeds your maximum retry window (5 minutes is the recommendation in the MPP docs; longer is safer). On the client side: do not retry a payment unless you have confirmed the previous payment either failed or was not submitted.

A double payment in a micropayment context (sub-$1 per call) is manageable. A double payment on a $500 software license purchase via an autonomous agent is a support ticket you cannot easily resolve.

### 4. Reconciliation

Agent payments break the assumption that a human initiated each transaction. Reconciliation tooling needs to answer: which agent, for which task, authorized by which user, at what time, for what amount. AP2's mandate chain gives you that audit trail by design. x402 gives you on-chain transaction data. MPP gives you Stripe Dashboard events plus on-chain receipts. ACP gives you checkout session records and order webhooks. The matching problems are the same ones we cover in [building a payment reconciliation engine](/blog/payment-reconciliation-fintech), except the initiating party is now software.

Plan for this before you have 10,000 transactions a day and someone asks you to reconstruct the spending for a specific user's agent session last Tuesday.

## The competitive picture

[Juniper Research projects](https://www.juniperresearch.com/press/agentic-commerce-set-to-generate-15-trillion-globally-by-2030-as-payments-infrastructure-leaders-revealed/) $8 billion in agentic payments spend in 2026, growing to $1.5 trillion by 2030. [Visa invested in Replit](https://techcrunch.com/2026/05/28/visa-invests-in-replit-to-power-agentic-payments-for-developers/) in May 2026 specifically to build agentic payment tooling for developers. The card networks are not standing still.

```cd-chart
{ "type": "bar", "title": "Projected agentic payment spend (Juniper Research)", "xKey": "year",
  "series": [{ "key": "spend", "label": "Spend ($B)", "color": "#5BBDF9" }],
  "data": [{ "year": "2026", "spend": 8 }, { "year": "2030 (proj.)", "spend": 1500 }],
  "unit": "B USD" }
```

The interesting observation from the x402 traction numbers is that the average transaction value is under $0.31. That is firmly micropayment territory, API calls, data access, compute. At that price point, card rails are economically unworkable (Visa/Mastercard minimum fees exceed the transaction value). Stablecoins win by default at the bottom of the value distribution.

AP2's card support and MPP's SPT path matter for higher-value delegated purchases: software licenses, SaaS subscriptions, B2B procurement. ACP's bet is different again, distribution rather than rails: Instant Checkout puts merchant catalogs directly in front of ChatGPT's traffic, on ordinary card economics. That is a different problem from paying $0.003 for a vector search API call.

## Which one to build for

If you are building a paid API or data service that agents will call: x402 is the fastest path to production. The protocol is minimal, the SDK is small, and the ecosystem is the largest.

If you are building an agent that makes purchasing decisions on behalf of users, and you need defensible consent capture for compliance or fraud reasons, AP2's mandate model gives you an audit trail that a payment receipt alone does not.

If you are already using Stripe for payment processing, MPP is the obvious path. You get both crypto and fiat rails from the same API, and it integrates with your existing Stripe Dashboard reconciliation.

If you sell products and want them purchasable inside ChatGPT and whatever agent surfaces adopt the spec next, ACP is the only one of the four with a live consumer channel today. Implement the product feed and checkout spec, keep your merchant-of-record status, and your existing payment processor handles the money.

The honest answer is that these protocols are not mutually exclusive. x402 and AP2 already have an explicit integration (the A2A x402 extension). Stripe co-authored MPP and ACP, sits in the x402 Foundation, and supports MPP and x402 from the same endpoint stack. You will likely end up implementing multiple.

If you are scoping out the payment authorization, settlement, or reconciliation layers for an agent-native product, that is exactly the kind of architecture problem we work through with fintech and Web3 teams. [Get in touch](/#contact) if you want a second opinion.


# Shipping stablecoin rails without the pain

URL: https://codedecoders.io/blog/shipping-stablecoin-rails-without-the-pain
Published: 2026-04-20 (updated 2026-07-07) | Category: Fintech | Tags: stablecoins, payments, remittance, fintech

What we learned launching a USDC-powered remittance corridor, settlement timing, custody trade-offs, and the boring infrastructure that makes it actually work.

Stablecoins promise global, instant, low-cost transfers. The promise is mostly real: the rails themselves work. Almost all of the friction lives in the boring layers around them, and that is where the engineering time actually goes.

We recently helped a client, a regulated digital asset firm, launch a stablecoin-backed remittance corridor between the US and the Philippines. The first version went from concept to production in three weeks. Since then the platform has settled more than $100M across 100,000+ transfers, with peak days above 5,000 transactions, and a typical settlement time of about five seconds on a route where the old correspondent banking path took two to three banking days. This is the condensed retrospective: what settlement actually means to a customer, how we made the custody decision, and why the unglamorous layers of idempotency, reconciliation, and logging turned out to be the real product.

## Settlement is a UX choice, not a technical one

On a public chain, [USDC](https://www.circle.com/usdc) settles in seconds. The token mechanics are well documented in [Circle's developer docs](https://developers.circle.com/), and the on-chain leg of a transfer was never our problem. The problem is that "settled" means at least three different things to someone sending money home: is my money safely in the system, what exchange rate am I getting, and when can my family actually spend pesos.

We ended up surfacing three explicit settlement states in the UI, and that single decision removed roughly 70% of support tickets.

### The three states

**Funds received.** We have the sender's dollars and the transfer has been accepted into the system. Compliance screening has passed and the amount is locked in. The FX rate is not yet locked, and the recipient cannot spend anything. This state answers the sender's first question: did my money arrive safely.

**Converting.** The USDC leg has settled on-chain and the USD to PHP rate is locked. A payout instruction is on its way to the local payout partner. For the customer this is the moment the price becomes fixed: the pesos they were quoted are the pesos that will arrive. It is also the point of no cheap return, because unwinding from here means reversing an FX conversion at whatever the rate is now.

**Delivered.** The recipient's bank account or e-wallet has been credited and the money is spendable. Not "sent to the bank", credited. We only show this state after the payout partner confirms the credit, because the gap between "we sent it" and "they have it" is exactly where customer trust dies.

### Why one "pending" state fails

Collapse those three into a single "pending" and you have one answer for three different questions. A sender asking "did my transfer go through" and a recipient asking "can I withdraw this yet" both see the same word, and neither gets an answer. Refunds make it worse: canceling in funds received is a full refund, canceling in converting involves an FX reversal, and canceling in delivered is not a cancellation at all, it is a new transfer in the opposite direction. If the interface cannot distinguish these states, your support team becomes the state machine, resolving it one ticket at a time.

## Custody is the hardest decision you'll keep reopening

Every custody option costs something different, and the costs are not on the same axis. We weighed four:

- **Self-custody.** You hold the keys. Maximum control and minimal counterparty risk, but you now own hardware security modules, key ceremonies, disaster recovery drills, and the hardest conversation with your regulator. Cheap in fees, expensive in engineering and audit.
- **MPC.** Key material is split into shares across parties or devices, so no single machine can sign alone. Strong key control with a much better operational story than raw self-custody, but you inherit a vendor dependency and platform costs, and your team still owns signing policy and quorum operations.
- **Qualified custodian.** A regulated third party holds the assets. Lowest operational burden and the cleanest regulatory posture, at the price of custody fees, withdrawal latency, and concentrated counterparty risk.
- **Banked custody.** Stablecoin balances held inside a banking partner's platform. The least crypto operations of any option, and the most constrained: you move at the bank's speed and inside the bank's product roadmap.

The axes that actually mattered in the decision: who controls the keys, who carries the operational burden, how much counterparty risk you concentrate in one name, what story you can tell a regulator with a straight face, and what it costs at your expected volume.

We started with a qualified custodian, for speed. With a three-week timeline, "a regulated custodian holds the assets" is a sentence that compliance teams, banking partners, and regulators all understand on the first pass. Nobody had to hire key management staff or rehearse a key ceremony before launch. We accepted the fees and the concentration risk as the price of shipping.

The reason the decision keeps reopening is that the right answer changes with volume. Custody fees that are noise at low volume become a line item worth engineering against later, and the operational maturity you lack at launch is exactly what you build by operating.

### An interface the settlement engine never sees through

What made this a decision instead of a trap: custody sits behind an interface. The settlement engine speaks a small vocabulary. Get a deposit address, initiate a withdrawal, fetch a balance, subscribe to confirmation events. It holds opaque account references and never imports a custodian SDK. Maker-checker approval for outbound signing lives behind the same boundary, so the approval workflow does not change when the implementation underneath it does.

That boundary is what let us commit, credibly, to swapping to MPC within a quarter once volume justified the operational investment. The swap plan was mechanical: implement the same interface against the MPC provider, run both implementations in parallel against the reconciliation checks, then cut over. The ledger, the transfer state machine, and the UI never learn that anything changed.

## The boring stuff is the actual product

Half the engineering hours went into the parts no one demos. All of them show up at 2am.

### Idempotent transfers, or double payouts

Every transfer request carries a client-supplied idempotency key, and every transfer is a state machine with explicit states and a short list of legal transitions. Retry a request and you get the same transfer back, not a second one. This sounds obvious until a mobile client on a weak connection retries a payout request four times, and each retry would have sent pesos.

Under the state machine sits an append-only double-entry ledger with per-asset zero-sum enforcement: every movement is a balanced pair of entries, and the books for each asset must sum to zero at all times. Transfers post two-phase entries, a hold placed on the debit leg when the transfer is accepted, then captured or released once the outcome is known. Replay protection falls out of this structure almost for free, because a duplicate posting against the same key is a no-op by construction. We cover the pattern in more depth in our post on [double-entry ledger design](/blog/double-entry-ledger-foundations).

### Reconciliation is three-way or it is theater

We reconcile three sources against each other: on-chain events, our internal ledger, and bank statements from the fiat legs. Two-way reconciliation misses entire classes of failure. Comparing chain to ledger will not catch a bank crediting the wrong amount, and comparing ledger to bank will not catch a deposit that landed on-chain but was never booked. With three sources, every discrepancy has a direction, and the direction tells you which system is lying.

The checks are fail-closed. If an asset's books stop balancing, payouts for that asset halt until a human explains the difference. The same fail-closed posture applies to compliance: party, transaction, and address screening gate every transfer, and a screening outage stops the pipeline instead of waving transfers through. How we structure the matching and the halts is the subject of our post on the [payment reconciliation engine](/blog/payment-reconciliation-fintech).

### Postmortem-grade logs

"Postmortem-grade" has a concrete meaning for us: months later, with access to nothing but the logs, you can reconstruct a single transfer's complete history. Every state transition records the idempotency key, the ledger entry ids, the transaction hash, the custodian reference, and the screening decision id. When a customer says a transfer from March arrived short, the answer is one log query and a ledger lookup, not an archaeology project across four dashboards. The test we apply is simple: could an engineer who joined last week write the incident timeline from logs alone.

## What we'd do differently

- **Build reconciliation on day one.** We built it after the first discrepancy, which means the first discrepancy was diagnosed by hand across three exports. The engine was a small build. The manual version of it was one long, unpleasant investigation that the engine would have finished before lunch.
- **Name the settlement states in the API from the start.** We added the three states to the UI first and retrofitted them into the partner API later. Partners had already built against a single "pending", so we carried a translation layer longer than we wanted to.
- **Treat the custodian sandbox as a fiction.** Sandbox withdrawal behavior differed from production in both timing and error shapes. Run small real-money transfers through the production path early, before launch traffic finds the differences for you.
- **Write the runbook with the feature.** Every alert we shipped without a runbook was triaged from scratch the first time it fired.

One forward-looking note. The same discipline, idempotency keys, explicit states, fail-closed gates, is exactly what machine-initiated payments will demand. If software agents start moving money over rails like these, and protocols like x402 suggest they will, the systems that hold up will be the ones already built to be retried and replayed safely. We wrote about that shift in our post on [agentic payments and x402](/blog/agentic-payments-x402-ap2-mpp).

If you're shipping financial rails of any kind, stablecoin or otherwise, and want a second pair of eyes on the architecture or the compliance trade-offs, that is exactly the kind of conversation we're happy to have.