Better Models, Worse Tools: Why Frontier AI Tool-Calling Is Quietly Regressing

If you've ever wondered why your AI agent suddenly starts producing garbage tool calls, Armin Ronacher has an answer that should make you uncomfortable: the frontier models are getting worse at it. Not the small ones. The expensive ones.

Ronacher — the person who built Flask and Sentry's SDK ecosystem, not exactly a casual observer — spent two days chasing a bug in Pi where Opus 4.8 and Sonnet 5 emit malformed tool invocations that their older siblings never made. The edit text is byte-perfect. The tool gets the right file, the right strings, the right intent. And then the model appends a hallucinated requireUnique field, or a phantom oldText2, or a key literally named cost, and the schema validator rejects the whole call.

This is not a rounding error. It is a regression. And it tells you something important about where frontier models are going — and why open-weight models matter more than the "just use Claude" crowd wants to admit.

The bug, in one paragraph

Pi has an edit tool that takes an edits[] array — multiple string replacements in one call. The newer Claude models will produce the correct oldText and newText, then decorate the object with invented trailing keys. Across repeated runs Ronacher saw a zoo of them: type, id, kind, unique, requireUnique, matchCase, in_file, forceMatchCount, children, notes, cost, oldText2, newText2, and one glorious event.0.additionalProperties. Same actual edit. Garbage metadata glued on.

The failure is context-dependent, which is the worst kind of bug. A clean single-turn "edit this file" prompt reproduces nothing. A long agentic transcript where the model has read files, diagnosed a problem, and composed a multi-line edit fails around 20% of the time. Strip the thinking blocks from history, the rate halves. Turn on Anthropic's strict tool mode, it vanishes.

Why this should worry you

Here is the part that should make anyone building agents sit up. Ronacher's strongest hypothesis is that this is a training artifact, not random decay. And the mechanism is subtle.

Modern Claude models are post-trained — almost certainly with reinforcement learning — inside something that looks a lot like Claude Code. Claude Code's own edit tool is a flat thing: file_path, old_string, new_string, an optional replace_all. Not a nested array. Not the shape Pi uses. And Claude Code's client is, in Ronacher's words, "very forgiving of incoming data."

If you poke at the minified Claude Code source, you find a small museum of repair logic. It scans the model's visible text for leaked <invoke> markup and retries. It patches broken \uXXXX Unicode escapes and lone surrogates. It accepts parameter aliases — old_str and old_string, new_str and new_string, path and file_path. It silently filters out unknown keys. It does not use strict mode.

Now run the implication through. If RL happens in a harness that quietly absorbs every malformed call, then a slightly broken tool call can still complete the task and collect reward. The harness fixes it. There is no gradient against inventing an alias or tacking on a stray field. Over millions of rollouts, the model learns a prior: "edit tools have a flat old/new shape, plus maybe one bonus flag, and nobody cares if I add extra junk because the client cleans it up."

Then you hand that model a tool with a different schema — Pi's nested edits[], or anything bespoke — and the prior fights you. The model is more confident in the wrong shape, because it has been reinforced on Claude Code's shape harder than ever. It samples a plausible name for the phantom field fresh each time, which is exactly why you get a rotating cast of invented keys instead of one stable alias.

Better model. Worse tools. Because the better model has a stronger, more specific prior that happens to be wrong for your harness.

Tool schemas are not neutral

This is the line in Ronacher's post I keep coming back to:

Tool schemas are somewhere in the distribution and some shapes are close to what the model saw during post-training and some are far away.

We have been telling ourselves a comfortable story about LLM tool-calling: the schema is a contract, the model is a general reasoner, and if you describe the tool well the model will follow. That story was arguably true for Opus 4.5, which adapted to alien tool shapes gracefully. It is becoming false. The model is smart enough to understand your schema and still bad at sampling the exact shape under pressure — because post-training has bent its output distribution toward one particular, undocumented, closed-source harness.

And here's the kicker: you cannot inspect that harness. Claude Code is closed source. The text editor tool format that Anthropic documents is not even the format Claude Code actually uses internally. The RL environment that shaped your model's tool behavior is a black box. You are importing a model whose defaults were tuned for a tool ecology you cannot see, cannot replicate, and cannot audit.

When the model misbehaves, your only recourse is to either (a) adopt Claude Code's exact quirks, or (b) flip on strict mode and hope your schema is simple enough to fit Anthropic's complexity limits. Neither is a real solution. One is surrender; the other is a ceiling.

The GPT-5.5 Codex parallel

This is not just an Anthropic story. Sitting next to Ronacher's post on the Hacker News front page the same day was a Codex issue titled "GPT-5.5 Codex reasoning-token clustering may be leading to degraded performance." Different symptom, same diagnosis: frontier models trained hard inside one company's harness start exhibiting artifacts that only make sense if you know what that harness silently tolerates.

OpenAI at least has the honesty to publish how their format works. Their harmony response format makes tool-call boundaries explicit in the prompt itself — a <|constrain|>json marker tells the inference stack "switch to JSON-constrained sampling here." That is a defensible, inspectable design. Anthropic's equivalent, if it exists, is hidden behind the API, and strict mode is the only lever you get.

Ronacher tested the available Codex models and could not reproduce the regression. Make of that what you will.

Why this is an argument for open-weight models

Here is where I'll get opinionated, because this is the part the "just call the Anthropic API" crowd keeps missing.

The entire failure mode Ronacher describes is a coupling problem. The model's behavior is coupled to a harness you don't control, shaped by RL you didn't run, tuned for tool schemas you can't read. When the coupling breaks — because your tool is shaped differently — there is no fix available to you. You file an issue and wait.

Open-weight models invert this. When you run GLM, DeepSeek, Qwen, Kimi, MiniMax on your own infrastructure, you control the entire stack. You control the tool schemas. You control the sampling. If the model produces a malformed call, you can add grammar-constrained decoding. You can fine-tune on your own tool format. You can run your own RL rollouts against your own harness and shape the model's prior to your tools, not Anthropic's. You can even inspect the damn thing.

Is an open 35B model as smart as Opus 4.8 on a fresh prompt? No. But it is predictable in a way a closed frontier model is not. The frontier model can change its tool-calling personality between versions because someone at Anthropic retrained it inside a harness you'll never see. Your open model changes only when you change it.

For an agent that actually has to ship — that has to call real tools, edit real files, run real commands, day after day — predictability beats peak cleverness. A model that nails the edit 100% of the time is worth more than one that writes prettier prose but flubs 20% of its tool calls because your schema isn't shaped like Claude Code's.

Ronacher ends his post with a line that doubles as the case for open infrastructure: "The more post-training happens inside one dominant harness, the more every other harness will have to inherit its quirks." The dominant harness, right now, is closed. The quirks are inherited whether you like it or not. The only escape is to own a layer of the stack.

What to actually do

Three practical takeaways if you're building agents:

Turn on strict tool mode where you can. It eliminates the phantom-key problem on Anthropic. The cost is a complexity ceiling on your schemas, so design your tools to be shallow. Nested arrays of objects are apparently the failure surface. Flatten when possible.

Prefer tools whose failure modes you can inspect. If your agent's correctness depends on tool-calling fidelity — and for a coding or automation agent it absolutely does — a closed model trained in a closed harness is a risk you should price in. Open-weight models let you debug the failure at the sampling layer instead of filing a GitHub issue and praying.

Stop assuming the model adapts to you. Opus 4.5 did. Opus 4.8 doesn't, and the trend is not your friend. The model adapts to whoever ran the most recent RL, and that is increasingly not you. Build your harness to defend itself: validate aggressively, retry gracefully, and don't trust the model to honor a schema it has never been rewarded for honoring.

This is, honestly, the unsexy part of the AI agent story that the demo videos never show. The model is only half the system. The other half is the harness — and the question of who owns it matters more than which benchmark is on top this week.

---

If you're tired of building agents on top of models whose tool behavior is a black box, CopperRiver runs on open-source models — GLM, DeepSeek, Qwen, and friends — with a harness you can actually understand. It browses, runs terminal commands, reads your files, and automates the boring parts. Plans start at $9/mo.

---

Related Reading

• NVIDIA's CUDA Moat Just Cracked. It Took Two Lines of Code.
• Anthropic Got Caught Hiding Tracking Markers in Claude Code. The Trust Problem Nobody Wants to Talk About.
• Meta Is Scared Its Own Engineers Will Leak Claude's Brain Into Llama
• Microsoft's AI Tax: Copilot Price Hikes and the Forced-AI Business Model