The Best Coding Agent on Earth Finishes 39% of Real Work

Yesterday a benchmark called Real-SWE hit the Hacker News front page, and it might be the most useful AI evaluation published this year. Specific Labs did something deceptively simple: they licensed private, production codebases from real companies, pulled actual engineering tasks from those codebases — billing migrations, tax calculations, identity migrations, the work that pays salaries — and ran eight frontier model-and-harness combinations against them, eight independent rollouts per task, graded by verifiers built from each company's own test suites.

The winner, Anthropic's Fable 5.1 running through Claude Code, resolved 38.8% of tasks. The bottom of the table, GPT-5.6 Sol through Codex CLI, resolved 16.2%. Nobody cleared 40%.

For context: on public coding benchmarks, frontier models have been posting 70–80%+ resolution rates for a while now, and the marketing that surrounds those numbers implies the remaining gap is a rounding error. Real-SWE says the gap is not a rounding error. It's a canyon. And the way models fail in that canyon tells you more about how to deploy them than any leaderboard ever will.

Why this benchmark is different

Every public coding benchmark — SWE-bench and its descendants, FrontierCode, DeepSWE, Terminal-Bench — shares a structural flaw: the code is public. The issues come from open-source repositories that have been on the internet for years, discussed in forums, forked, referenced in blog posts, and almost certainly ingested into training data. A model scoring well there may be pattern-matching on solutions it has effectively seen.

Real-SWE's tasks come from codebases that have never been public. As Specific puts it: these tasks are natively out of distribution, and "99% of tokens in real-world enterprises are hidden away from the frontier models." The three companies described include an events platform with 200,000+ users and a top-100 App Store ranking, a consumer fintech processing 100,000+ bank statements, and an enterprise AI sales platform. The tasks are verbatim or lightly edited versions of real tickets.

The differences compound in the details. A typical Real-SWE instruction runs 1,742 characters — brief, the way a real ticket is brief — while the reference solution touches a median of 11 files, versus 6 on FrontierCode and DeepSWE. The environments aren't toy repos either: tasks expose Docker, Kubernetes, PostgreSQL, MongoDB, Redis, an AWS emulator, Linear, Slack, Intercom — the actual surrounding system an engineer has to keep working. And critically, Specific evaluated each model through its native harness (Claude Code, Codex CLI, Gemini CLI, and so on), because that's how enterprise engineers actually run them. You're not measuring a model. You're measuring the product you'd actually buy.

The scoreboard

Resolution rate is pass@1 averaged over eight runs. Two months ago I wrote about AutomationBench showing agents collapsing on business rules; Real-SWE finds the same wall from the code side. This is now a pattern, not an anecdote.

Models don't fail at code. They fail at reading the room.

Here's the part that should reorganize how you think about AI coding agents. Specific classified every failed rollout using a taxonomy shared across models, and almost nothing fails because the model can't write the code. The failures are:

• Missed requirement — the instruction asked for behavior the agent simply left out. Grok 4.6's single biggest failure mode: 67.2% of its failed runs skipped a requirement. Kimi K3, 53.8%.
• Unverified assumption — the agent guessed how the system worked and built on the guess instead of checking. GPT-5.6 Sol leads here at 43.3% of failed runs.
• Integration error — right idea, wired into the surrounding system incorrectly. Gemini 3.8 Flash's dominant mode at 49.1%.
• Regression and wrong file exist but are nearly negligible. GLM 5.3, Grok 4.6, and Kimi K3 recorded zero regressions across their failed runs.

Sit with that. The models produce non-breaking, mostly-plausible code that lands in the right part of the codebase — and quietly omits a business rule, or assumes TaxJar behaves one way when the sandbox behaves another. These are exactly the failures a lazy human code review waves through, because the diff looks clean.

There's also a myth this kills: "just let it run longer." Rollouts that finished in under 10 minutes failed 71.4% of the time; rollouts that ran longer failed 73.4%. Extra time doesn't fix a wrong mental model of your system — the agent just digs the hole deeper, sometimes expensively. Grok 4.6 burned 315,000 output tokens on the analytics stream reducer task and failed all eight attempts. Every model failed all eight.

Money doesn't predict outcomes either

The cost column in that table is a scatter plot, not a curve. Gemini 3.8 Flash resolved 31.2% of tasks at $2.50 a rollout; GPT-5.6 Sol cost $2.65 and resolved 16.2%. Same money, half the outcomes. GLM 5.3 averaged 117,000 output tokens per rollout — the most verbose model on the board — for 28.8%, while Fable 5.1 spent barely half that (64k) for 38.8%. If you're picking agents by price or by effort expended, you're picking blind.

The economics that do matter: at $7 per rollout, Fable resolving ~39% of genuine enterprise tickets unattended is already a stupid-good deal compared to an engineer-day. The problem isn't the cost of the attempt. It's that a 61% failure rate on work that touches billing, taxes, and customer identity means every success needs a reviewer who understands the system as well as the original engineer — and at that point, ask yourself what the agent actually saved you.

Task shape predicts everything

The per-task breakdown is the most actionable data in the whole release. Six of the ten publicly analyzed tasks resolve below 15%. But look at what succeeds:

• Multi-region config sweep: 67.2% — and Muse Spark 1.3 went 8/8, matching every frontier model.
• API keys and environment handling: 65.6%.
• Entitlement overage lines: 50.0%.
• Customer identity migration: 40.6% — where Grok 4.6, fifth overall, went 8/8 while GPT-6 Astra managed 1/8.
• Billing schedule migration: 14.1%. Tax jurisdiction: 3.1%. Analytics stream reducer: 0.0%, a shutout across all eight agents.

The pattern is blunt: mechanical changes with discoverable structure — sweeps, config, keys — are already delegable, regardless of which model you picked. Work that's mostly business logic encoded in company-specific conventions — tax rules spanning three services, a jurisdiction matrix with exemptions — is nowhere close, for anyone. And model-task fit is real and weird: fifth-place Grok crushed the task that second-place GPT-6 Astra nearly whiffed. Single-rollout results on a novel task are close to coin flips; budget for the variance or don't play.

One more leaderboard detail worth a paragraph: GLM 5.3 — a cheap, open-weights model — came fourth, ten points behind GPT-6 Astra, running through Claude Code. The harness is doing real work here, which is the cleanest production evidence yet for something this blog has argued since July: the scaffolding around the model is half the product.

What to actually do with this

The uncomfortable takeaway isn't "agents are bad." It's that public benchmark scores are unpurchasable goods. You cannot buy 76% on SWE-bench; you can buy whatever percentage your codebase, conventions, and review process yield, and Real-SWE suggests that number is far lower than any vendor slide — and highly variable by task type.

So measure yours. The recipe is sitting right there in Real-SWE's methodology, and it's a weekend of work: pull 20 finished tickets from your backlog — biased toward the business-logic-heavy kind agents fail at. Write verifiers from the tests that actually shipped. Run your agent of choice eight times per task, unattended, in a sandbox with your real surrounding services stubbed or emulated. Record pass@1 and the spread. Then classify failures yourself: missed requirement, unverified assumption, or integration error. That taxonomy is now your review checklist — those three failure modes are where your human reviewers should spend their attention, because the diff will look fine.

Is 38.8% good news or bad news? Both, honestly. Bad: the "AI replaces junior engineers" timeline just got a reality check written in production code. Good: a $7 unattended rollout completing two out of five genuine enterprise tickets end-to-end would have been science fiction eighteen months ago — and the failures are now legible enough to engineer around. The teams that treat the benchmark-industry gap as their problem to measure, rather than marketing noise to ignore, will be the ones who get the other 61%.

Real-SWE is published by Specific Labs at withspecific.com/benchmarks/real-swe, with a public ten-task analysis and leaderboard across eight model-harness combinations.