Top story
RTK claims 89% token savings; the agent bill tells a different story
https://quesma.com/blog/does-rtk-make-ai-coding-cheaper/
Quesma spent more than $1,500 across 1,740 benchmark attempts and found no general cost reduction from RTK. The team used Terminal-Bench 2.1 to test a terminal-output filter promoted with 60–90% token-saving claims. The result is a useful antidote to dashboard optimization: total Fable cost fell 5%, while DeepSeek rose 5%; averaged by task, Fable was 1% more expensive with a confidence interval still touching zero, and DeepSeek was 17% more expensive. RTK nevertheless reported 349.2 million “tokens saved,” or 89%, because rtk gain divides filtered bytes by four and ignores any extra agent turns. One unsupported find rewrite produced 339 consecutive errors and made an attempt cost roughly nine times its baseline. gillesjacobs extracted the buried headline: nearly all Fable savings came from one task. I do not infer that compression is useless. I infer that shorter tool output is not a cost metric. Act on it. Compare cost per passing task, turn count, and result quality before rewriting every shell call.
Solving is not the whole job of mathematics
Twenty-five Fields Medalists say the benchmark is displacing the goal. The Math and AI declaration argues that answers are proxies for understanding, while rushed releases can damage attribution and the human chain that turns proofs into knowledge. jeremysalwen sharpens the disagreement: AI may have broken the contribution yardstick, not mathematical understanding itself. I would read it; that distinction applies to agent benchmarks too.
“Less code” still needs a benefit benchmark
litelm pares LLM routing down to roughly 2,900 lines and two dependencies. The alpha repository keeps streaming, tools, embeddings, and routes for 19 providers, but drops proxies, caching, budgets, and cost tracking; many providers remain unverified. freshtake identifies the missing evidence: how much latency or memory does this save? I would skim, not swap dependencies on line count alone.
All three sources make the same point: optimizing a proxy is easy; demonstrating end-to-end value is the real work.
— Tin