Telling Technology · Learning in public
All episodes
EP.13 — COST ROUTER
A cost-first AI router, built live

I made my AI shop for the cheapest brain that can do the job

Running AI at home burns real money — the API meter never stops, and half my tasks don't need the smartest model. So I had Claude build a two-layer router: one flag picks the cheapest model that's still good enough, then the cheapest host to serve it. Same scripts, same prompts, a fraction of the bill.

AI requests fanning out toward a row of model price tags; a cost-first router lights the cheapest capable model and its cheapest host in teal, house aurora-teal style
⌁ one env flag · same prompts · fraction of the bill
First the problem: the meter that never stops
01 The problem · the meter never stops

Running AI at home costs real money

Every script I run, every nightly job, every little classify-this or summarise-that call hits an API that charges by the token. It adds up fast — and the dumb part is that most of those calls don't need the smartest model. I was paying premium prices to rename files and tidy captions. The fix isn't writing worse prompts; it's making something else decide, per call, how much brain the job actually deserves.

claude code — desktop
> half my AI calls are trivial but they all hit the top-tier model. can you make my code auto-pick a cheaper model when the task is simple — without me rewriting any prompts?
Yes. We add a router in front of the API: a single low_cost switch that sends easy work to a vetted pool of cheaper models, and leaves the hard work on the premium one. Your prompts don't change — only what they're routed to.
A rising AI API cost meter with most calls flagged as trivial — small tasks billed at premium model prices, house aurora-teal style
before
trivial calls, premium prices — the meter that never stops
02 The build · two levers, one wrapper

One file, two ways to save

Claude built or_client.py — a thin wrapper around OpenRouter with two cost levers. Lever one: low_cost=True routes the call to a quality-vetted pool of capable-but-cheap models (Gemma-4-31b, DeepSeek-v4-flash and Nemotron-3-Super all clear the bar), with live prices refreshed every ten minutes. Lever two: inside that pool, provider.sort=price tells OpenRouter to pick the cheapest host serving that model. You win twice — on the model and on the infrastructure under it.

or_client.py — the two levers
> chat(messages, low_cost=True)
  # lever 1 → cheapest vetted model in the pool
  # lever 2 → provider.sort="price" → cheapest host
Routed to deepseek-v4-flash via the lowest-price host · prices were refreshed 4 min ago · same messages, same response shape — only the destination changed.
The or_client.py two-layer selection logic highlighted: a low_cost model-pool filter feeding a provider.sort=price host picker, house aurora-teal style
the code
or_client.py — model pool, then cheapest host
03 The honest part · cheap needs a gate

"Cheapest" only works because of the quality bar

Routing every call to the literal cheapest model on the market would be a disaster — you'd save money and get garbage. The router only works because the cheap pool is curated: a model earns its place by passing a quality bar first, and only the vetted ones are eligible for the price sort. And the hard jobs — the ones that actually need it — still go to the premium model. So the honest claim isn't "AI for free." It's: spend premium money only where premium matters, and let everything else fall to the cheapest option that still passes.

claude code — desktop
> won't "always cheapest" just pick a bad model and wreck my output?
It would — so cheapest is chosen within a vetted pool only. A model has to pass the quality bar to be eligible, and anything genuinely hard stays on the premium model. Cost-first, not quality-blind.
A quality gate in front of a pool of cheap models — some pass and become eligible for the cheapest-host sort, some are rejected, house aurora-teal style
the gate
pass the quality bar first → then sort by price
⚠️

"Same results" means within the quality bar — not identical bytes

A cheaper model won't return the exact same words as the top-tier one. The promise is that for the tasks I route low-cost — classify, summarise, tidy, extract — the output clears the same bar I'd accept from the expensive model. Anything where the wording really matters, I leave low_cost off and pay full price on purpose. The savings are real; the honest framing is "good enough, far cheaper," not "free and identical."

04 The receipts

The code, the compare tool, and the bill

Four moments from the build: the two-layer selection logic, the live price-vs-latency compare tool, the route diagram, and a before/after cost estimate. Tap any image to enlarge it and read the exact prompt that drew it.

The or_client.py wrapper with the low_cost model-pool branch and the provider.sort=price host selection highlighted
two levers in one wrapper
Terminal output of llm_route_compare.py: a price-vs-latency table across models with the cheapest passing row highlighted
price vs latency, live
A horizontal route diagram: prompt to router to cheapest vetted model to cheapest host to response, with hard tasks branching to the premium model
the route, on one line
A before/after cost bar chart, clearly labelled as an illustrative estimate: a tall all-premium bar versus a short low-cost bar
before / after · illustrative
ONE FLAG · SAME PROMPTS · LOWER BILL

The whole switch is one line in the env

That's the payoff. I don't touch a single prompt or script — I set AIOS_LOWCOST=1 in the environment, and every eligible call quietly reroutes to the cheapest model that still passes, on the cheapest host that serves it. The kicker writes itself: Claude Opus built this router in one session — the AI optimised its own cost footprint, then handed me the bill it had just shrunk.

A single glowing env line AIOS_LOWCOST=1 above a route collapsing to the cheapest capable model and host, with a shrinking cost meter, house aurora-teal style
AIOS_LOWCOST=1 · prices refreshed every 10 min · hard tasks still go premium
06 Steal this

Steal the router

The or_client.py OpenRouter wrapper with both cost levers, the llm_route_compare.py price-vs-latency tool, and the vetted-pool config. Everything in this episode is free and open — clone it, set your own key, flip the flag.

wrapper or_client.py tool llm_route_compare.py config vetted model pool guide AIOS_LOWCOST=1
run it gh repo clone jacobskogstrom/or-cost-router ~/or-cost-router && cd ~/or-cost-router && cat README.md
↧ Get it on GitHub ⎘ Run the compare tool ▶ Try the demo (placeholder key)

No GitHub? Comment ROUTER on the post and the bot DMs you the link.

Next episode

I planned a whole AI feature before writing a single line of code

A voice conversation tier for my home AI — spec'd and planned end to end with Claude before any code. The episode is the planning, not the typing: how to design a complex AI feature so the build is the easy part.

Darkened teaser — a voice waveform turning into a spec document and an implementation plan before any code appears
drops next · follow @bitesizeai so you don't miss it