The Spend Ledger

What your traffic actually cost, grouped by whoever is responsible for it, with a charge-back export.

Backstop can tell you how many tokens a request cost before it happens. This page is about the part that tells you, after the fact, what the last day of traffic actually cost, grouped by whoever is responsible for it — and says so in a form finance can use.

It is off by default. Turning it on adds one record per completed provider request, priced in Decimal from a bundled rate card, and it never puts a file open or a network call on the request path.

What it is

• A SpendEvent per completed provider request: provider, model, endpoint, priority, outcome, four token counts, latency, retries, your attribution, and a priced cost breakdown.
• An ambient attribution context, so a call site says who is spending this once, at the top of the function that spends it.
• A price catalog with three precedence layers, and Decimal money end to end.
• A charge-back aggregator and an RFC 4180 CSV export, with the honesty columns (unattributed, unpriced, estimated) sitting next to the money.
• A runaway-spend detector that reports four kinds of unusual spending and, by default, blocks nothing.
• Four CLI subcommands: backstop ledger demo, show, export and backstop reconcile.
• A reconciliation against a provider statement file, per model and in total, with the reason for every difference and a measured error budget. It calls no provider API and has never been run against a real invoice in this repository.

What it is not

• Not a metrics backend. It writes a file, or keeps a bounded in-memory ring. Nothing ships to a collector, nothing is queried over HTTP, nothing is retained for longer than the file you chose.
• Not a revenue system. It never infers, allocates or fetches revenue. It joins cost against a revenue map you supply, on group keys you choose, and shows a visible (none) wherever either side is missing.
• Not an enforcement primitive for spend. The detector reports; it cannot block, cancel or kill. Spend avoided is not in the ledger either — see Limitations.
• Not a cross-customer benchmark. Nothing in it aggregates across deployments, and there is no published figure of what “normal” cost is.
• Not exact. It prices the token counts the provider reported, from a rate card that is a dated snapshot, and where either is missing it says so in a column rather than filling the hole.
• Not a payment system and not a billing connection. It reconciles against a statement file you supply. It cannot fetch an invoice, cannot credit a difference, and cannot tell a committed-spend discount from a rate change without a rate card that already knows about it.

Quickstart

One flag, on a real client:

from openai import OpenAI
from backstop import Backstop, BackstopConfig, attribution

client = Backstop.wrap(
    OpenAI(),
    budget=500_000,
    config=BackstopConfig(ledger_enabled=True, ledger_path="ledger.jsonl"),
)

with attribution(team="payments", feature="checkout-v2"):
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Hello"}],
    )

Every completed request now appends one line to ledger.jsonl. Read it back:

backstop ledger show --path ledger.jsonl
backstop ledger export --path ledger.jsonl --out chargeback.csv --group-by team,feature

No key and no file needed for the pitch:

backstop ledger demo

Three config fields do the wiring; all default to off or to a usable value, so an existing BackstopConfig() is unchanged.

The SpendEvent schema

One record per completed provider request. Frozen, validated on construction, and round-trippable: SpendEvent.from_dict(SpendEvent(...).to_dict()) == SpendEvent(...). schema_version is the literal "1.0".

The four token counts are validated as int and >= 0 on construction — True is refused by name, because bool is an int subclass. A persisted line must carry exactly the 18 declared fields: an unknown key and a missing key are both ValueErrors naming every offender, so a typo or a truncation cannot reload as plausible data.

Attribution fields, all optional: team, agent, session, task, feature, surface, customer, tenant, environment, repo, cost_center, gl_code, currency.

Every amount on the wire is a string ("0.010000"), never a JSON number. A JSON number is read back as a binary float, and a finance export that loses cents is worse than one that is awkward to parse.

Attribution

Three forms, imported from the top-level package. The transport reads the active value when it builds each event, so nothing else in your code changes.

from backstop import attribution, with_attribution, current_attribution

# 1. a block
with attribution(team="payments", feature="checkout-v2", customer="acme"):
    client.chat.completions.create(...)

# 2. a function or method, sync or async
@with_attribution(agent="refund-bot")
def handle_refund(ticket):
    ...

# 3. read it back
current_attribution().team

current_attribution() never returns None; in a process that has declared nothing it returns an all-None Attribution, which is a valid record.

Nesting merges, and the outer value is restored on exit including when the body raises. Both halves of this are in the output below — the inner scope sets two fields, keeps the outer two, and the decorator on a separate function lands in its own row:

import httpx

from backstop import BackstopConfig, attribution, with_attribution
from backstop.ledger import build_chargeback
from backstop.ledger.export import render_chargeback_markdown
from backstop.state import BackstopState
from backstop.transports import BackstopTransport


def handle(request: httpx.Request) -> httpx.Response:
    return httpx.Response(
        200,
        json={
            "id": "chatcmpl_doc",
            "object": "chat.completion",
            "model": "gpt-4o-2024-08-06",
            "choices": [{"message": {"role": "assistant", "content": "ok"}}],
            "usage": {"prompt_tokens": 2_000, "completion_tokens": 500, "total_tokens": 2_500},
        },
    )


state = BackstopState.create(
    100_000,
    BackstopConfig(
        default_max_output_tokens=500,
        retry_max_attempts=1,
        ledger_enabled=True,
    ),
)
client = httpx.Client(
    transport=BackstopTransport(state, httpx.MockTransport(handle)),
    base_url="https://api.openai.com",
)


def call(text: str) -> None:
    client.post(
        "/v1/chat/completions",
        json={"model": "gpt-4o", "messages": [{"role": "user", "content": text}]},
    )


with attribution(team="payments", feature="checkout-v2", customer="acme"):
    call("authorize a card")
    with attribution(agent="refund-bot", environment="prod"):
        call("refund ticket 8821")
    call("capture a card")


@with_attribution(team="support", feature="triage")
def triage(ticket: int) -> None:
    call(f"triage ticket {ticket}")


triage(4711)

client.close()
report = state.close()
print(f"submitted={report.submitted} landed={report.landed} lost={report.lost}")
print(render_chargeback_markdown(build_chargeback(state.ledger.sink.events)))

submitted=4 landed=4 lost=0
| team | feature | request_count | input_tokens | output_tokens | total_usd | unpriced_requests | unpriced_components |
| --- | --- | ---: | ---: | ---: | ---: | ---: | --- |
| payments | checkout-v2 | 3 | 6000 | 1500 | 0.03 | 0 | (none) |
| support | triage | 1 | 2000 | 500 | 0.01 | 0 | (none) |

Three rules worth knowing before you rely on it:

• A blank is not a value. None and an empty or whitespace-only string both mean unset, so an environment variable that arrived empty cannot erase the value an outer scope set. A field passed as None means “do not override”.

• The value lives in a ContextVar, so it follows asyncio tasks and any thread-pool submission that copies context. A bare executor.submit or loop.run_in_executor does not copy context and would record an empty attribution — copy it explicitly:

  import contextvars
  
  ctx = contextvars.copy_context()
  loop.run_in_executor(executor, ctx.run, blocking_call, arg)

• with_attribution refuses a deferred body. A generator function, an async generator function, or a plain function returning a coroutine or generator is refused with TypeError rather than leaving the ledger silently unattributed. Scope the deferred body instead:

  def stream_refunds(handle, ticket):
      def events():
          with attribution(agent="refund-bot"):
              yield handle(ticket)
  
      return events()

  An asyncio.Future/Task is allowed through, because the work was scheduled inside the scope and the task copied the context there.

An unknown field name, or a non-string value, raises TypeError at the call rather than being dropped from the ledger.

The price catalog

A PriceEntry is one model’s rates in USD per million tokens, with its provenance. PriceCatalog.resolve(provider, model) returns one, or None.

Precedence, highest wins:

1. an explicit per-call price_overrides passed to compute_cost — price_source="override"
2. your catalog file, config.price_catalog_path — price_source="user"
3. the bundled table — price_source="bundled"

Each layer is searched in full, across every name tier, before the next is consulted. A user price for gpt-4o beats a bundled price for the exact string gpt-4o-2024-08-06, because your layer is the higher one.

Name resolution is three tiers: the exact string, then the name with a trailing snapshot marker stripped repeatedly (-2024-08-06 OpenAI, -20241022 Anthropic, -latest), then a fixed alias map. So a dated snapshot, an -latest alias and claude-3.5-sonnet all resolve.

An unknown model is None, not a guess. compute_cost returns None, the event is recorded with cost=None, and the export counts it in unpriced_requests. There is no default rate and no neighbouring-model fallback, because a charge-back that is quietly wrong is worse than one that is absent.

cache_write is separate. A component the rate card publishes no rate for is charged zero and marked in CostBreakdown.priced_components, so a row can say its cache-write line is a floor instead of presenting a zero as a measurement.

Your own rates, from a JSON file:

{
  "effective_from": "2026-10-01",
  "entries": [
    {
      "model": "gpt-4o",
      "provider": "openai",
      "input_per_mtok_usd": "2.10",
      "output_per_mtok_usd": "8.40",
      "cache_read_per_mtok_usd": "1.05"
    }
  ]
}

from backstop.ledger import PriceCatalog

catalog = PriceCatalog.from_file("prices.json")
for model in ("gpt-4o", "gpt-4o-2024-08-06", "no-such-model"):
    entry = catalog.resolve("openai", model)
    print(model, entry.input_per_mtok_usd if entry else "no price")

gpt-4o 2.10
gpt-4o-2024-08-06 2.10
no-such-model no price

Every entry in a file you supply is filed as source="user", confidence="negotiated": a rate a deployment wrote down is a rate it agreed to, whatever it was copied from. Write money as a string; a JSON number is accepted and read back through its shortest repr, so a hand-written 2.10 means exactly 2.10.

The arithmetic runs under the ledger’s own decimal context — 28 significant digits, ROUND_HALF_UP, every field named — never the host process’s. A decimal context is process-global and a library cannot ask the host to leave it alone, so an application that tightened prec would otherwise void every cost here silently: the request still succeeds, the ledger records nothing, and every total is wrong by a rounding step nobody asked for.

The charge-back export

build_chargeback(events, group_by=("team", "feature")) aggregates events into one row per group, largest total first. group_by is validated against the Attribution fields, so any of the thirteen dimensions works and an unknown one is an error rather than a silently dropped column.

Columns, in export order — group keys first, then:

An unset group key renders as (unattributed), never as a blank cell — an unattributed dollar is the thing finance most needs to see, and a blank cell is the one thing that hides it. There is no totals row in the CSV: a totals line inside a table somebody intends to SUM is a double count waiting to happen.

backstop ledger export --path ledger.jsonl --out chargeback.csv --group-by team,feature

wrote 6 row(s) to chargeback.csv from 1967 event(s) in ledger.jsonl — 34.56 USD total, 34 unpriced, 1.54 unattributed

revenue_join(rows, revenue_map) emits a second CSV keyed on the same group tuple: request_count, total_usd, revenue_usd, margin_usd, match, currency. A group missing on one side carries (none) and a match column naming the case — both, cost_only or revenue_only — because margin_usd is absent unless both sides are present. A float revenue figure is refused rather than read through its shortest repr.

Delivery, and the loss numbers

Two reports, and they do not answer the same question.

From a live writer (state.close(), or delivery_report(writer)) — the only place the process-local counters exist:

The accounting invariant is submitted == written + dropped_events + sink_errors, asserted only when the report says drained.

From a file (backstop ledger show --path ...) — a JSONL line records an event, never a counter. So a file read reports dropped_events: unknown and sink_errors: unknown with the reason, rather than as 0, because a zero is a claim about a process it cannot see. The one loss a file can prove is a torn final line:

• a final line that will not parse → reported as a torn write, the rest is read, exit 0;
• a final line that is unterminated but parses → reported, and kept, because dropping a complete event loses a dollar;
• an unparseable line anywhere other than the end → real corruption. The command names the line number, exits 1, and export writes nothing, because a charge-back that silently skips rows is worse than a failed command.

backstop ledger show --path ledger.jsonl      # exit 0
backstop ledger show --path missing.jsonl     # exit 1, message on stderr

BoundedWriter never blocks the caller. It is one short lock, one append to a bounded deque, return; the sink’s own work — including opening the JSONL file — happens on a single daemon thread. Overflow refuses the incoming event and counts it, rather than buffering without bound or silently evicting the oldest. close() joins that thread with a bounded timeout, so a stalled sink cannot hang shutdown; if the join expires the backlog is discarded and reported as undrained rather than written after close() returned.

Detection, and shadow mode

detection_enabled=True builds a RunawayDetector over a bounded window of the last detection_window_size events per attribution key, and evaluates four things on every submission:

Two of those choices are deliberate and worth stating. The window is split in half and the older half is the reference period, because over one shared window the median follows a sustained step change and a doubling stays silent forever. And an unpriced event contributes zero to the velocity numerator while still being observed for tokens and retries — padding it with a guessed rate would make a finance-facing figure look measured.

Each signal is a frozen record: kind, severity, key, observed, threshold, detail, occurred_at. It cannot block a request, cancel work, kill an agent, or raise. That is structural, not a promise: nothing exported returns a decision, and a test walks the public API for a verb that could act on a request. Auto-kill is absent on purpose — an agent run can be six hours of work, and a kill switch that cannot resume destroys it.

Shadow mode is the default (detection_shadow=True). In shadow, the same signals are produced and also filed in a bounded log with a per-kind counter, so a threshold can be tuned against evidence before it can interrupt anybody. BACKSTOP_DETECTION_SHADOW overrides the config in both directions, so a detector switched on by an over-eager edit can go back to shadow without a redeploy.

The detector can be driven directly, with an injected clock so the numbers are reproducible:

from dataclasses import replace

from backstop.detection import DetectionConfig, RunawayDetector
from backstop.ledger import Attribution, PriceCatalog, SpendEvent, compute_cost

catalog = PriceCatalog()
config = DetectionConfig(
    enabled=True,
    shadow=True,
    velocity_threshold_usd_per_min=1.0,
    min_samples=8,
    window_size=16,
)

clock = {"now": 0.0}
detector = RunawayDetector(config, time_fn=lambda: clock["now"])


def priced(at: float, input_tokens: int, output_tokens: int) -> SpendEvent:
    base = SpendEvent(
        provider="openai",
        model="gpt-4o",
        endpoint="/v1/chat/completions",
        priority="default",
        outcome="success",
        input_tokens=input_tokens,
        output_tokens=output_tokens,
        estimated=False,
        attribution=Attribution(team="payments", feature="checkout-v2"),
    )
    return replace(base, cost=compute_cost(base, catalog))


# Eight quiet minutes: one request a minute, 20k input tokens.
for minute in range(8):
    clock["now"] = minute * 60.0
    detector.observe(priced(clock["now"], 20_000, 500))
# Then an agent loop: 20 requests in 20 seconds, each with 200k input tokens.
for second in range(20):
    clock["now"] = 8 * 60.0 + second
    detector.observe(priced(clock["now"], 200_000, 2_000))

print("shadow:", detector.shadow, "keys:", detector.key_count(), "errors:", detector.errors)
print("counts:", {k: v for k, v in detector.counts().items() if v})
seen: set[str] = set()
for signal in detector.recorded():
    if signal.kind in seen:
        continue
    seen.add(signal.kind)
    print(signal.kind, signal.severity, round(signal.observed, 3), signal.threshold, signal.detail)

shadow: True keys: 1 errors: 0
counts: {'velocity': 10, 'drift': 10, 'context_growth': 8}
context_growth critical 10.0 2.0 input tokens 200000 against reference 20000 over 9 events
drift warning 93100.0 61500.0 tokens_per_request 93100.0 against reference 20500.0 (x1.514, multiplier 3) over 10 events
velocity info 1.16 1.0 $1.16/min over 16 events in 5.166667 min, 16/16 priced

Every threshold is reachable from BackstopConfig as a detection_* field, and a nonsense one is refused at wrap() rather than discovered on the first request: a negative or non-finite threshold, a baseline_multiplier below 1.0 (which inverts the comparison rather than alerting earlier), a non-positive window, and min_samples > window_size — a detector that can never speak, and looks exactly like a healthy one. detection_max_keys (default 1,024) bounds the number of keys; eviction is LRU and counted in evictions and reported to the signal sink, because a silently evicted key is a key whose baseline silently restarts.

What it costs

With the ledger off, which is the default: one attribute load, one truth test, one function call per request. No event is built, nothing is read from the response, no catalog is touched, nothing is offered to a queue. Verify that on your host with the two committed commands:

backstop verify      # overhead check passes below a 5 ms p99 threshold
backstop benchmark   # the same measurement the README table quotes

With the ledger on: it is material, and here is the number rather than a promise. Measured on this branch by calling BackstopTransport.handle_request directly over an httpx.MockTransport, one arm per subprocess, 7 alternating runs per arm, 3,000 iterations after 500 warm-up, Python 3.12.3 on Linux. Each cell is the median of the 7 runs:

The spread between runs is the honest uncertainty and it is wide: ledger-off p50 ranged 147.0-156.2 us, ledger-on p50 ranged 289.6-377.1 us. An independent measurement of the same shape on this build (.superpowers/sdd/_build-plan/task-6-report.md) put it at +130.0 us p50 (+84.6%) on a 153.6 us request. Two runs, one order of magnitude: turning the ledger on roughly doubles the cost of a request. The default path is unaffected, and the p95/p99 overhead class the library advertises still holds for it — but anyone who opts in is paying this, and it is larger than the library’s entire advertised overhead budget.

Where it goes, measured from inside the ledger call on the request path (task-6-report.md): SpendEvent construction 50.5 us, compute_cost 24.2 us, BoundedWriter.submit 3.7 us, plus roughly 35-40 us of GIL time lost to waking the drain thread. Reproduce it with:

import json, sys, time

import httpx

from backstop import BackstopConfig
from backstop.state import BackstopState
from backstop.transports import BackstopTransport

arm, iterations = sys.argv[1], int(sys.argv[2])
body = {
    "id": "chatcmpl-measure",
    "object": "chat.completion",
    "model": "gpt-4o-2024-08-06",
    "choices": [{"message": {"role": "assistant", "content": "ok"}}],
    "usage": {"prompt_tokens": 1200, "completion_tokens": 300, "total_tokens": 1500},
}
config = BackstopConfig(
    initial_concurrency=64,
    max_concurrency=64,
    ledger_enabled=(arm == "on"),
    ledger_memory_events=20_000,
)
state = BackstopState.create(10**9, config)
transport = BackstopTransport(state, httpx.MockTransport(lambda r: httpx.Response(200, json=body)))
request = httpx.Request(
    "POST",
    "https://api.openai.com/v1/chat/completions",
    json={"model": "gpt-4o", "messages": [{"role": "user", "content": "measure me"}]},
)

for _ in range(500):
    try:
        transport.handle_request(request)
    except Exception:
        pass

samples = []
for _ in range(iterations):
    start = time.perf_counter_ns()
    try:
        transport.handle_request(request)
    except Exception:
        pass
    samples.append((time.perf_counter_ns() - start) / 1000.0)
samples.sort()
print(arm, f"p50 {samples[len(samples) // 2]:.1f} us", f"p95 {samples[int(len(samples) * .95)]:.1f} us")

for i in 1 2 3 4 5 6 7; do
  PYTHONPATH=src python3 measure.py off 3000
  PYTHONPATH=src python3 measure.py on 3000
done

These are wall-clock figures on one host, so they are a measurement rather than a guarantee — the same caveat the README attaches to its own committed benchmark snapshot applies here. What is not a measurement is the shape: no file is opened, no socket is touched, and no lock a sink could hold is taken on the request path, and tests pin that against a deliberately stalled sink, a sink that raises, and a state whose ledger is not a writer at all.

Known limitations

• The rate card is a dated snapshot. The bundled table carries BUNDLED_EFFECTIVE_FROM = "2026-09-26", transcribed from two published price pages. Nothing in this repository detects a provider rate change, and the bundled table is the lowest layer, so it is what every deployment uses until somebody notices it is wrong. effective_from exists on PriceEntry but is not a charge-back column today. If a total drifts from your invoice in one direction only, that is a rate, not a token-count bug.
• The Anthropic cache-write rate is the 5-minute tier. The schema has one cache_write_per_mtok_usd field and Anthropic publishes two (5-minute at 1.25× base input, 1-hour at 2×). A deployment on a 1-hour cache TTL is under-counted on its cache writes by ~37.5% of that component — and nothing in the record reveals it, because cache_write is in priced_components: the row looks fully priced. This is a silent under-count, which is the worst failure class here, and it is the top open item in docs/planning/07-risks-and-killshots.md.
• Three of the six outcomes never reach a ledger. circuit_open, budget_denied and queue_timeout describe requests that never reached a provider: they have no usage report and no dollars, and a row for one would carry a pre-flight floor for spend that never happened — and push that floor into the detector’s window, contaminating the very ratios the detectors exist to measure. They stay in the schema so a durable line written by a future build still validates. Spend avoided is on the enforcement surface instead: budget_exceeded / rate_limited counters, the requests{outcome=circuit_open|exception} breakdown, and the audit log’s deny records.
• A streaming request is a floor, not a measurement. There is no usage report at stream setup, so the event is recorded at dispatch with estimated=True, input_tokens set to the pre-flight estimate and output_tokens=0. That is a visible, filterable understatement rather than a confident wrong split.
• A request host that is neither provider is unpriced. The provider is matched from the request URL, and a host that names neither api.openai.com nor api.anthropic.com is recorded as provider="unknown" and left unpriced — an invented provider produces a row that looks authoritative and is about the wrong vendor. Point BackstopConfig at a gateway and the price is yours to supply via price_catalog_path, keyed on the model rather than the host.
• The three loss counters are process-local. A JSONL line records an event, never a counter, so a file read reports dropped_events and sink_errors as unknown. If you need them, keep the writer in the process and read state.close().
• A burst can lose events, by design. The writer’s buffer is bounded, and overflow refuses the event rather than growing without limit. That is visible in dropped_events; the mitigation is ledger_memory_events and a longer queue at your own call site, not an unbounded queue.
• The ledger has no tenancy, no retention and no query layer. One file per process, whatever the process chose, until somebody deletes it. Rotation, archival and querying are yours.
• The demo’s teardown sentence is hardcoded. backstop ledger demo with a --group-by other than team,feature still says requests “declared no team and no feature” in its prose. The numbers and the table are correct; that one sentence is not. backstop ledger show and export word it generically.

Reconciling against a provider statement

build_chargeback produces a number. This section is about the check that makes it a number somebody can act on: comparing it against what the provider actually charged.

What it reconciles against is a statement FILE — a CSV or a JSON document that OpenAI’s or Anthropic’s console or admin surface produced, which you supply. backstop reconcile calls no provider API, holds no credentials, and has never been run against a real invoice in this repository. Every provider field name it reads is centralised in OPENAI_STATEMENT_FIELDS and ANTHROPIC_STATEMENT_FIELDS, marked as transcribed from those providers’ published reporting surfaces, and marked as needing verification against a real statement before you rely on them. One constant per provider is the whole cost of correcting a spelling.

backstop reconcile --demo                                                    # fixed, offline, synthetic
backstop reconcile --ledger ledger.jsonl --invoice statement.csv [--json]    # the real thing

from backstop.ledger import (
    Invoice, measure_error_budget, parse_invoice_csv, reconcile_invoice, reconciliation_csv,
)

variance = reconcile_invoice(events, parse_invoice_csv("statement.csv"))
budget = measure_error_budget(events, [parse_invoice_csv("statement.csv")])
print(variance.to_markdown())
print(reconciliation_csv(variance))

The five statuses, and why a variance is reported rather than fixed

The sign convention is ledger - invoice, so positive means Backstop claims more than it was charged. A negative variance is not the safe direction: it is what a streaming floor, an unpriced component and a stale rate card all produce, so both signs are reported and never an absolute.

Three rules make the number worth auditing:

• Nothing is netted away. A model on one side only gets its own row and is never offset against the other side, and the total says how many rows it had to exclude. A missing dollar cannot hide inside a matched one.
• Absent is not zero. If either side published no figure, the row’s variance_usd is absent, not zero, because a difference cannot be taken against a number that does not exist. This is the ledger’s own rule — an unpriced request is recorded with cost=None, not with 0.00.
• One unpriced request takes its model’s total with it. A sum of the events that did price is a figure nobody can defend, so the model’s ledger_usd is absent and unpriced_events says how many requests are responsible.

An unknown model is never guessed

A model the catalog does not carry is a row whose ledger side is absent, with a reason saying so — the visible form of the rule above. There is no default rate, no neighbouring-model fallback, and no dropping the row. Supply a catalog entry through price_catalog_path and the same three inputs reconcile; that is a test, not a claim.

Rows are keyed on the exact model string, not on a resolved price-card name. A statement that spells a model the way the ledger does not therefore produces a visible pair of rows rather than one merged row, because two names that happen to share a rate card are two names and merging them is a guess.

A variance can be attributed, not just reported

Every row carries the token counts on both sides and the delta beside them, so the difference can be attributed to a rate card or to a measurement:

The reason string on every non-matched row says which of the four it is, in words. within_tolerance is a fourth state on purpose: a difference inside a tolerance is still a difference, and the exact signed figure is on the row regardless.

The token-count asymmetry, and where it bites here

The two providers count cached tokens on opposite sides of the number they call the input, and the rule keys on the provider recorded on the line, never on the column name:

Read one as the other and the error is the size of the cache, not a rounding difference. A row that publishes an explicitly exclusive count (input_uncached_tokens, uncached_input_tokens) is read as the fresh input by either provider. This is the same asymmetry above records for the transports, and a test pins both directions.

The tolerance, and why those numbers

DEFAULT_TOLERANCE_USD is $0.01 and DEFAULT_TOLERANCE_PCT is 0.5%; the looser of the two wins on each row, and both are parameters (--tolerance-usd, --tolerance-pct).

1. A cent is the provider’s own stated precision — both published statements are denominated in cents, so a cent is the smallest difference one can express.
2. It absorbs the ledger’s own six-decimal quantisation: $0.0000005 per request, so a cent covers the worst case for 20,000 requests.
3. It cannot hide a rate error: 1% of a $100 row is $1.00, a hundred times the tolerance. This is the direction that matters, because a stale rate card is the first gap in the planning document’s own list.
4. It is loose in token terms on a large row — a cent is 4,000 input tokens at gpt-4o’s $2.50/Mtok — because a tolerance is a claim about dollars and a rate card is a dollar claim. A deployment that wants a tighter answer on one model passes a smaller tolerance, and every row prints its exact signed variance either way.

One number from someone else’s tolerance is not a number to adopt. Measure yours.

The error budget, measured rather than asserted

measure_error_budget(ledger_events, invoices, catalog) reports what was observed: the absolute variance summed over every model row, that sum as a percentage of what the statement billed on the rows that could be compared, the count of lines, and the worst single row by name.

The absolute figure is the headline on purpose. A netted figure lets a model that is over-counted hide one that is under-counted — which is the netting this module refuses everywhere else — so the netted number is reported beside it and never as the answer.

ErrorBudget.note states the limits in the function’s own output, so the caveat travels with the figure instead of living in a docstring somebody has to have read:

This is an empirical figure over the inputs it was given, not a guarantee. An error budget is only an error budget over a population that resembles the inputs it was measured on: a figure measured over a handful of statements says nothing about a deployment that streams, that sends traffic through an unwrapped client, that runs a model the catalog does not price, or that is on a rate card the providers have since changed.

An unmeasured budget — no statement, or a window where every row is missing a side — reports measured: false and within_tolerance: false. A budget over nothing has not been shown to hold.

The export

reconciliation_csv(variance) returns RFC 4180 CSV text, CRLF, stable column order (INVOICE_COLUMNS), largest variance first, and no totals row. Money is the two-decimal strings the charge-back export uses, so no cell in the file is a number a spreadsheet can re-round; an absent figure is (none), never a blank and never a zero. The exact six-decimal variance for every row that is not matched is inside that row’s reason cell, so the two-decimal money columns lose nothing a reader needs.

What the demo shows, and what it is worth

backstop reconcile --demo runs offline with no key and no file, and its report is byte-identical across runs. Its statement is synthetic and every disagreement is declared in advance, because a demo where everything reconciles teaches a reader nothing about what the tool is for:

One statement is built per provider and each is reconciled against that provider’s events only, because M1 is defined per provider per month and reconciling one vendor’s statement against the other’s traffic would report models as unbilled that are billed on the other statement.

What the demo’s error budget is worth: nothing beyond the inputs it ran on. It is one synthetic statement over one synthetic day, on a rate card dated 2026-09-26, with no streaming, no unwrapped client and one deliberately unpriced model. It is not a promise about any other account, and not a promise about this one. backstop reconcile --demo prints that caveat because the alternative — a number on a slide with no population attached to it — is the thing this whole section exists to prevent.

Reference

A reconcile variance is a finding, not a failure: the command exits 0 on a complete report whatever that report says, because a command that exited non-zero on a variance would train an operator to ignore it. Exit 1 means there is no report to read, and nothing is reconciled silently past one of those cases.

Both --group-by flags take a comma-separated list of Attribution fields and reject an unknown one as an argparse usage error (exit 2) listing the legal dimensions, rather than failing later inside the aggregator.

backstop ledger demo is synthetic traffic: the request counts, token counts and timestamps are fixed, and the dollars are exact for that volume under the bundled rate card. One model, vendor-preview-2027, is deliberately absent from the rate card, so its 34 requests are counted and their dollars are absent. That row is the pitch: a tool that quietly filled that hole would be showing a number nobody could defend.