The Counting House

Most of Yegge's recent essays are about letting agents roam. This one is about the shape that works when you can't let them roam: huge volume, regulators watching, money moving. The answer is classic data science and boring engineering, with cheap models as the payoff.

The thesis: the opposite of a software factory

Yegge's own factory, the Wheelhouse (successor to Gas Town), treats agents as long-lived colleagues with names, memories and broad access. That suits exploratory software work. It is also expensive and risky, and locking it down kills what makes it work. The Counting House inverts every one of those choices. Each model call handles exactly one case, starts from a fresh context, sees data only through a schema, and exits. The payoff is that you can run millions of cases a year on comparatively cheap, even open-weight, models, and you don't need your software-development AI story sorted out first.

Why this works on messy real-world operations: the volume is enormous but the variety is small. A bank or an airline has maybe a few dozen to a few hundred intents, which are finer-grained than support queues (a goodwill refund and a waiver refund follow different procedures, so they are different intents). You can solve them one at a time. Yegge ties this to his years running Amazon's customer-service tools: the weekly contact-reason histogram reviewed with Bezos, with each top reason routed to a business owner to fix.

Five tiers of rope

The policy table has one row per intent: which model, what tools, which humans approve, and how much autonomy. Every intent starts with none and moves up only on evidence.

• off: humans handle it. This is also the fallback when the whole system trips a rate limit or budget cap.
• shadow: the model answers silently alongside the human, purely so it can be scored. No savings yet.
• assist: the model drafts suggestions before the rep sees the case. This is the first tier that saves money, and the first that needs guarding, because reps start deferring to the suggestion.
• contain: the model answers the customer directly. Reputational risk starts here.
• transact: the model's decisions change the real world (refunds, rebookings), always through the orchestrator.

Score the humans first

You can't grade a model without knowing what "right" is, so the first artifact is a gold set: for each intent, roughly 200 to 1,000 real human-handled cases, more for money and regulated intents. Each case is broken into yes/no rubric questions (followed procedure? compliant tone? answered the question?). Two people score each case blind, a third adjudicates disagreements, and a slice is held out as the final exam that is never used for tuning. An LLM judge then grades candidate models against that set. The judge is trusted only on intents where it agrees with humans about as often as humans agree with each other.

That agreement rate is the hidden ceiling. Yegge puts human-to-human agreement at around 90%, and past that line your evals can't tell whether a model beats your people. He also sizes the work honestly. A whole gold set runs to tens of thousands of cases and months of QA time, but the first intent takes a few hundred cases and a couple of days, and everything after that is incremental.

The confusion matrix: a price tag on every mistake

The second eval asks a narrower question: did the model even pick the right intent? Yegge's example uses four airline intents that sound alike and mean very different things: flight status (is my flight still on?), rebooking (get me on the next one), goodwill credit (that trip was awful, make it right) and special assistance (will someone meet me at the gate?). Rows are what the customer actually needed; columns are what the model picked. The diagonal is correct, and every other cell is a specific kind of error with a specific cost.

Numbers and per-cell costs from Yegge's example figure in the essay; table layout ours. Error cost by row: $782 · $1,780 · $6,000 · $150,000 = $158,562.

"This is basically a fancy screw-up cost table, one beloved by data scientists everywhere, and you’ll soon see why."— The Counting House

The overall accuracy is 906 of 1,000, or 90.6%, and Yegge dismisses it as a boast: it counts the hits and ignores what the misses cost. The table shows why. The six special-assistance cases the model sent elsewhere (a stranded child, a missing wheelchair) cost $150,000 of the $158,562 in errors. The other 88 mistakes together cost under $9,000. The errors fall into three bands: cheap misroutes that the orchestrator or a human catches, money given away out of policy (tens to hundreds of dollars each), and harm to the customer (tens of thousands, sometimes a letter to a regulator).

Precision, recall, and the duck hunt

Yegge admits these two terms never stuck for him until he pictured a carnival shooting gallery. Precision reads down a column: of everything the model labelled flight status, how much really was? (470 of 510, or 92%.) Recall reads across a row: of all the real special-assistance requests, how many did the model catch? (44 of 50, or 88%.) You can max out either one by cheating at the other. Fire once and you have perfect precision. Spray the whole gallery and you have perfect recall.

"In short, high recall means you hit all the ducks, whereas high precision means you only hit ducks."— The Counting House

Which one matters depends on the intent. When an intent hands out money, you want precision: credits should go only to people who qualify. When missing a case hurts someone, as with special assistance, you want recall near 100%, even if that means some people get a wheelchair they didn't need. The test for a model is not a threshold on either number. It is a comparison: the model's error cost plus its inference spend against your humans' error cost plus their labor, computed on the same gold set. If your reps miss 2 of 50 special-assistance cases and the model misses 6, the model fails that intent.

The orchestrator: models propose, machinery disposes

Clerks never write. A clerk emits a write intent, a ticket naming the outcome it wants, such as moving a customer to a cheaper flight and refunding the difference. A purely mechanical orchestrator turns it into the half-dozen real transactions it implies. Those steps have to succeed or unwind together, so Yegge reaches for the Saga pattern, with compensating actions to undo partial work, and recommends running it on Temporal rather than rebuilding durable execution yourself. He lists six organs:

• Intake: schema-check the write intent, and bounce anything malformed.
• Eligibility: re-check every claim against the live system of record, not the cache the model read. A customer who sweet-talks the clerk still has to actually qualify.
• Idempotency: derive a key from the case, booking, action and amount, so retries can never issue the same refund twice.
• Approval: small amounts pass automatically, larger ones go to a human, and both record who approved and under which rule.
• Workflow catalog: one saga per write-intent type, many lasting days, with read-backs after each write.
• Exception queue: refusals carry reason codes that route them to the customer, a supervisor or IT. The queue also doubles as a map of where your procedures are broken.

It's an org change, and the clerks deserve a decent shift

Yegge is blunt that no single AI pilot team can build this. Ops owns the procedures, QA and data science own the gold set and judge, finance prices each cell of the matrix, legal decides which intents may ever reach contain or transact, and IT owns the plumbing. Each promotion needs sign-off from all of them. He closes by translating his model-welfare principles to clerks that live for one case: tell the clerk truthfully what it is and how it's graded, use no threats or fake urgency, give it a clean path to decline and escalate, score the intent rather than the clerk, and give it occasional "play" sessions to suggest changes to its own job. His argument is that skipping this builds a Severance-style anxiety machine, and in his experience anxious models burn more tokens and make more mistakes.

The line worth keeping

"Whoever owns the evals decides which models you can switch to, so make sure it’s your own data science team."— The Counting House, on why the gold set must stay in-house

That is the strategic core. Switching to a cheaper model is the cost lever the whole system exists to pull, and only the team that owns the intent taxonomy, gold set, policy table and scoring harness can pull it. Rent vendors' engineers if you like, but don't rent your evals.

The one honest objection

The matrix is an illustration, not field data. The per-cell costs are round numbers chosen to make the point, and the essay reports early success at one unnamed shop rather than results you can check. The architecture is also heavier than the "postcard" framing suggests: a PII vault, an MCP read gateway, a judge, an orchestrator team and a standing QA operation. The welfare section rests on Yegge's own experience more than on evidence he presents. None of that weakens the method, though. Scoring humans first, pricing each kind of error, and promoting one intent at a time works whether or not his dollar figures match yours.