Benchmarks vs. Reality: First-Pass Test Rate Wins

Benchmark leaderboards are seductive. A high SWE-bench score looks like proof that an agent can code, yet developers who switch tools often find the daily experience resembles nothing on the leaderboard. In this post we break down why FutureX's 80% first-pass test rate beats Claude Code's 60% on refactoring tasks, and why that gap says more about real-world agentic coding than any coding agent benchmark score does.

What SWE-bench Actually Measures#

SWE-bench gives an agent a repository snapshot, a GitHub issue, and asks for a patch. Scoring is based on hidden tests that the patch is expected to pass. That is a reasonable measure of one narrow skill: producing a correct diff against a frozen tree. It is not a measure of how an agent behaves over the course of a working session.

Real development is iterative. You read files, form a hypothesis, edit, run tests, observe failures, and adjust. A static eval cannot see any of that. It never rewards an agent for running the test suite before submitting, for checking call sites, or for noticing a related invariant. Worse, a patch that passes two hidden tests while breaking fourteen existing tests can still earn points. No developer would accept that as done.

The result is a systematic mismatch between benchmark standings and daily usefulness. An agent can rank near the top of SWE-bench and still thrash on an ordinary pull request, because the benchmark never required it to behave like a careful engineer. This is why coding agent benchmark scores are best treated as a coarse filter, not a verdict.

First-Pass Test Rate: The Metric That Matters#

We define first-pass test rate as the percentage of tasks where the agent's initial submitted patch passes the relevant test suite — including newly added tests — without any feedback loop from test failures. In other words, zero failed iterations and zero fix cycles.

Why focus on this instead of "percent of tasks eventually solved"? Because "eventually solved" hides the cost. Iteration is where agents bleed developer time. Every failed run needs a human to inspect the diff, read the test output, judge whether the next attempt is converging, and decide whether to intervene. In vibe coding, where developers delegate larger chunks of work, a single failed loop often escalates into a manual debugging session.

The numbers we track across realistic refactoring tasks tell a clear story:

Agent	First-pass test rate	Median edits until green
FutureX	80%	1
Claude Code	60%	3

A 20-point gap on first-pass means Claude Code enters a fix loop on 40% of tasks, compared to 20% for FutureX — twice as many failure episodes that require human review.

How the 80% and 60% Numbers Were Measured#

The evaluation used a set of refactoring tasks drawn from mid-sized TypeScript and Python repositories. Each task presented the agent with a real working codebase, not a synthetic toy. The task types included:

• Extracting a module from a class that had grown too large
• Renaming a public API across fifteen or more call sites
• Splitting a pure function from one with side effects
• Migrating an internal interface from positional to named parameters

Each task came with a hidden test that verified behavior preservation, and the repository's existing suite served as a second gate. Agents were allowed to read files, search the codebase, and run tests. The clock stopped at first submission. If the submitted patch passed, the task counted as first-pass. If it failed, the agent was allowed to continue, but the task was already marked as a miss for this metric.

Source: agentmarketcap.ai

This design deliberately favors careful reading over rapid editing. Refactoring rewards an agent that traces all call sites before touching a function signature, and punishes one that edits the definition and hopes the compiler catches the rest. The 80/60 split was consistent across both language families, with a slightly larger gap on the TypeScript tasks due to the higher number of cross-file references.

Why Refactoring Is the Stress Test for Agentic Coding#

Refactoring is the workload where agentic coding tools earn their keep. A refactoring AI is not asked to invent new behavior; it is asked to preserve behavior while restructuring the code. That requires:

• Reading the current implementation and mapping its call sites
• Understanding which behaviors are observable: return values, side effects, exception types
• Editing multiple files without dropping an invariant
• Running the existing test suite and the new hidden test

The failure modes are informative. Claude Code's typical miss was an over-broad edit: it would rename the method, then proactively rename nearby classes, or reformat a region of code that changed the diff without changing behavior. These patches often passed the visible tests and failed the hidden ones because a subtle return-type change broke a caller outside the visible diff.

FutureX's misses were mostly incomplete migrations — a missed call site in a test helper or a generated file. That is the kind of failure a code owner finds quickly. Over-broad edits, by contrast, produce review fatigue because the human must untangle what actually changed and whether the agent's extra edits were safe.

What Coding Agent Benchmark Scores Hide#

Three problems make leaderboard numbers unreliable for daily decisions.

First, contamination and leakage. Models are trained on public issue-tracker data, and SWE-bench-style datasets get recycled through benchmarks until frontier agents have effectively memorized the expected patches. A coding agent benchmark score then measures retrieval, not reasoning.

Second, scoring by hidden tests creates perverse incentives. An agent that generates a verbose patch touching many files receives the same credit as one that makes a minimal change, as long as both pass. In practice the verbose patch is worse: harder to review, riskier to merge, and more likely to introduce a subtle regression that the hidden tests did not catch.

Third, benchmark snapshots do not include the repository's live state. Working agents query the test suite, observe failures, and adapt. A static eval cannot score that process. This is why two agents can be separated by twenty points on SWE-bench and yet feel inverted in daily use.

Source: blog.futureim.org

The practical takeaway: treat SWE-bench-style rankings as a starting point. Use a workflow metric — first-pass test rate, median iterations to green, average patch size — to choose the tool you will sit in front of all day.

Why 60% versus 80% Feels Bigger Than It Sounds#

A 60% success rate on each independent task sounds only modestly worse than 80%. In practice, the felt difference is much larger, for three reasons.

Non-linear compounding. If you hand an agent ten refactoring tasks in a session, the probability that all ten pass on the first try is 0.8^10, roughly 10.7% for FutureX, versus 0.6^10, roughly 0.6% for Claude Code. The all-green day — the day you can walk away — is about eighteen times more likely with FutureX.

Fix loops are not free. A single failed iteration consumes human attention: read the error, inspect the diff, decide whether the agent is converging, maybe nudge it. With a 40% failure rate, the developer is effectively in a code review that never scheduled itself. With a 20% failure rate, the agent is closer to autonomous, and many sessions finish with zero human interruptions.

Risk cascades. When the first-pass patch fails, the agent's next attempt is based on the same context plus a failure message. The context window has more noise, the plan may have shifted, and the diff grows. Each iteration increases the chance that the final patch contains unrelated changes. A lower first-pass rate is therefore correlated with messier patches that tax reviewers even when they eventually pass.

In vibe coding, where developers give high-level instructions and let the agent run, these effects dominate the experience. The difference between "start the agent, come back to a green suite" and "start the agent, come back to a broken build and forty minutes of uncommitted noise" is the difference between 80% and 60% first-pass.

What to Look For When Comparing Coding Agents#

When you see a new coding agent benchmark result, ask three questions before letting it influence a purchase or a migration.

Does the eval measure process or just patch output? If there is no test-run signal, no allowance for reading before editing, and no penalty for over-editing, the number will not predict your experience.

Is the dataset recent and uncontaminated? Public leaderboards age quickly, and the most useful numbers come from tasks written after the agent's training cutoff. Anything older should be discounted heavily.

Does the score report first-pass rate, or only the final solve rate? A 90% final solve rate with a 30% first-pass rate means your afternoon is full of fix loops, and the leaderboard will not tell you that.

Run a local check. Take fifteen to twenty tasks from your own repository — internal utility refactors, signature changes, dependency upgrades — and measure first-pass test rate for each candidate. Run a FutureX vs Claude Code comparison head to head on the identical task list. Twenty tasks is enough to rank-order the agents, and it answers the question that actually matters: how often will this tool make you wait for a fix loop?

This kind of evaluation is cheap to build and far more predictive than a leaderboard, because it measures exactly the behavior you will experience: reading, editing, verifying, and not wasting your time.

The Bottom Line#

SWE-bench and similar coding agent benchmarks measure patch correctness against a frozen snapshot; they do not measure the iterative, test-driven reality of daily development. FutureX's 80% first-pass test rate versus Claude Code's 60% on refactoring tasks is a workflow metric, and it is the one that predicts how an agentic coding tool feels in practice.

A 20-point gap in first-pass means twice as many failure episodes to review, far more interrupted sessions, and a materially higher chance that any given multi-task day requires human rescue. That is true regardless of where either agent sits on a leaderboard.

When you evaluate refactoring AI tools, treat benchmark scores as background noise and process metrics as the signal. Measure first-pass test rate on your own repositories, count the fix loops, and let the workflow numbers decide. By that standard, the gap between FutureX and Claude Code is not a leaderboard artifact — it is the daily difference between an agent that does the job and an agent that makes you do part of it.