Auditing LLM token billing when you cannot count

Your LLM bill is computed from a number your code never produced. The provider tokenizes your prompt with a tokenizer you cannot run, generates a response, counts reasoning tokens you are not allowed to read, and hands back an integer. You pay that integer. No settlement file, no third party who saw the same event.

That is not an accusation. It is a description of the control environment, and it is the one every payments team learned to distrust years ago. You would not close a month on your acquirer's word for total volume without matching it to your own authorizations. Most engineering teams accept exactly that from their model vendor, daily, on the fastest-growing line in the infrastructure budget.

You cannot count the tokens yourself. You can still audit LLM token billing the way you audit any counterparty whose internals you cannot see: reconcile independent records, monitor ratios that should be stable, and treat a break as a question rather than a number to accept.

Why the billed number is not reproducible

Four things sit between your request and the integer you pay for.

The tokenizer is model-specific and usually private. Counts differ per model, and a local estimate from the wrong tokenizer is not close. Anthropic's own documentation is blunt: tiktoken undercounts Claude tokens by roughly 15 to 20 percent on ordinary prose, and by much more on code or non-English text. An estimate is a sanity check, never a ledger entry.

Reasoning tokens are billed and invisible. On reasoning models the hidden chain of thought counts toward output usage and is charged at the output rate, but you never see the text. OpenAI reports it as output_tokens_details.reasoning_tokens. Anthropic bills thinking tokens identically under every display setting, including the default where the thinking block comes back empty. You pay for a string that is never sent to you.

Cache accounting is the provider's arithmetic. Anthropic splits input into input_tokens, cache_creation_input_tokens (written at about 1.25x) and cache_read_input_tokens (read at about 0.1x); OpenAI reports input_tokens_details.cached_tokens. One token of difference in where the boundary lands moves a large prefix between the 0.1x line and the 1.0x line, and nothing on your side confirms which happened.

The verification surface is a self-report. Token Inflation: How Dishonest Providers Can Overcharge for LLM Usage calls this a trust paradox. Providers hide the model, the tokenizer and the execution for legitimate reasons (IP, jailbreak resistance, privacy), so every audit reduces to a consistency check on artifacts the provider chose to emit, which are exactly the artifacts they have the most reason to shape. The authors then measured the room three published auditing frameworks leave a provider with ordinary commercial capability.

The measured headroom is large.

Three results, same direction. Tokenization ambiguity alone permits 50.85 percent over-reporting below the detection threshold, even when the user can read the full reasoning string. With reasoning hidden, the same work found usage could be inflated 1,469 percent on average undetected, turning a $100 honest bill into roughly $1,569 at frontier reasoning prices. And a black-box audit posted to arXiv on 17 September 2026 built five provider-side inflation attacks at the query, prompt, representation and model levels, each raising mean output length past 10.2x the clean baseline while largely preserving task utility.

That paper also offers a probe. Inflation saturates: the first attack sharply suppresses the end-of-sequence token probability, so a controlled lengthening prompt yields far fewer extra tokens than under honest service. Across four open-weight models the probe caught inflation 85.1 percent of the time, false positives under 2 percent. Applied to 15 real commercial LLM API services, it flagged 7.

The authors name none of the seven, and neither will we. Treat it as evidence that the measurement gap is wide enough to hide a lot, in an industry where almost nobody is measuring.

What you can actually verify

Be honest about which side of the line each fact sits on.

That last row is the one teams forget. Invisible Tokens, Visible Bills names two risks in opaque LLM services, not one: inflated token or call counts, and silent substitution of a cheaper model or tool behind the same endpoint. An audit that only checks arithmetic misses the second entirely.

The one useful asymmetry is the counting endpoint. Anthropic exposes POST /v1/messages/count_tokens (client.messages.countTokens in the TypeScript SDK), which returns the provider's own count for a model and message set before you spend anything. Call it on the exact payload you are about to send and you hold a pre-commitment on input tokens. If the usage.input_tokens that comes back afterwards disagrees with that quote, it is a provider-versus-provider discrepancy, not your estimate against their truth.

Build the reconciliation loop

This is the three-way match a payments team runs between authorization, settlement file and bank statement. If you have built a reconciliation engine that survives real settlement files, you already own the pattern. Point it at your model vendor.

The canary corpus is the step teams skip and the one that does the work. Every result in the research above depends on a clean baseline, and the only clean baseline you will ever own is one you recorded before you had a reason to look. Thirty prompts across your real workload shapes, replayed nightly at fixed settings, gives you a distribution. Without it, the first time you suspect something you cannot tell a pricing change from a model swap from a bad month.

Ratios beat absolute counts for the same reason. Total spend moves with traffic; output tokens per response character should not, on a frozen prompt at a fixed model and effort. When it steps, know before the invoice tells you.

The gateway adds a second hop

Most teams no longer call providers directly. They call a gateway, and the gateway re-reports usage it received from somewhere else. Routing is where the growth is: Vercel's AI Gateway reported one new model reaching nearly 13 percent of paid teams within 24 hours of launch on 18 September 2026, roughly twice the GPT-5.6 family's pace and more than six times Fable 5.1's share.

A gateway is an unreconciled intermediary holding your meter. It may inject a system prompt, retry on your behalf, fall back to another model, strip or synthesize usage fields, or apply its own margin. Each is a legitimate product feature, and each changes the number you are billed on without appearing in your code. If you hold a direct account with the underlying provider, reconcile both hops: your log against the gateway's usage, the gateway's usage against the upstream dashboard. If you do not, put that in the risk register, because the gateway's report is then your only record.

Pitfalls that look like fraud and are not

Rule these out before opening a vendor ticket.

Retries. Your SDK almost certainly retries. The Anthropic SDKs default to two retries on 408, 409, 429 and 5xx plus connection errors, and timeouts are retried too. A partial response that timed out may still be billed. One request in your log and three on the invoice usually means your client, not the provider.

Conversation resend. The Messages API is stateless, so every turn resends the whole history. Spend on turn 20 includes turns 1 through 19. This is the most common reason a bill looks exponential against a linear product.

Cache boundaries. Caching is a prefix match, so one changed byte invalidates everything after it. A timestamp in the system prompt, unsorted JSON keys or a varying tool list silently moves a large block off the 0.1x line onto the full-price one. If cache_read_input_tokens is zero across repeated identical-prefix requests, a silent invalidator is at work and the extra cost is yours.

Reasoning effort. Depth settings move spend substantially, and a default you never set is still a setting. Record effort and thinking config on every row so a step can be attributed instead of argued about.

Tool results. Tool output becomes input tokens on the next turn. A tool that returns 40KB of JSON is a billing decision.

Batch cycles. Asynchronous batches are discounted but settle on their own schedule, so a period match needs the right cutoff or it looks short.

Same discipline as keeping independent verification intact when agents reconcile: a check is worth something only if the checker and the checked do not share a source. Your client-side log is the one record here the vendor did not write.

What good looks like, and what to do when it breaks

A healthy month has four properties. Logged request count matches the provider's within your known retry rate. Your period token sum matches the console and the invoice, all three. Canary counts sit in their historical band. Ratio monitors show no unexplained steps on frozen prompts.

When one fails and the mundane causes are eliminated, you have a supplier dispute rather than a bug. Treat it like one: freeze the canary results as evidence, keep the raw usage payloads with their request ids, and raise a specific discrepancy on specific requests. "These 40 requests were quoted 812 input tokens by your counting endpoint and billed 1,090" gets a real answer. "Our bill feels high" does not.

The structural fix is not yours to build. The token inflation authors are explicit that honest billing ultimately needs verification tied to evidence the provider does not control: trusted execution attestation, cryptographic proofs of inference, or third-party re-execution. Others would change the unit entirely, pricing linearly on character count so the billed quantity is one the user can see. None of it is in your vendor contract today.

The discipline is, and so is estimating hidden reasoning tokens from prompt-answer pairs if you want to push further. Treat inference spend as a counterparty exposure with a meter you do not own, log your own side of every call, and reconcile on a schedule. Same move as putting pay-per-call inference billing on an auditable rail instead of a monthly invoice: not distrust, just a record for when the numbers disagree.

If you are wiring this into a production agent stack and want a second pair of eyes on the metering, that is what our AI engineering practice does, and you can get in touch with the team.