A benchmark score is not a model review

A two-point lead on a coding benchmark does not mean two models will perform almost identically in a real repository. The score covers one set of tasks, one agent setup, and one grading rule. It does not show how much supervision each model needs, how costly its failures are, or whether I would accept its pull request.

DeepSWE provides a useful example. Luna Max scores 67.2%, while Sol High scores 69.4%, even though Luna’s displayed cost is about one-sixth as high [2]. That summary makes Luna look like the obvious choice. In practice, however, similar pass rates can hide large differences in the time I spend steering the model, checking its assumptions, and repairing its work.

Developers sometimes call that difference the “big model smell”: the sense that a stronger model understands the job with less explanation and makes fewer decisions that need to be undone. The phrase is informal, but the experience behind it can be tested. I can count how often I redirect the model, repair its work, or reject a patch that passed the tests.

What does 67% versus 69% actually measure?

Datacurve’s DeepSWE v1.1 data records 301 successful attempts out of 448 for Luna Max and 313 out of 451 for Sol High [1] [2]. Those totals produce the 67.2% and 69.4% scores. They show a 2.2 percentage-point gap on this test, but they do not show that the models perform alike in day-to-day work.

DeepSWE controls several important variables. Its 113 tasks come from 91 open-source repositories and cover TypeScript, Go, Python, JavaScript, and Rust [1]. The tasks were written for this evaluation instead of being copied from old GitHub issues. Every model also used mini-swe-agent with the same bash tool and base prompt [5], and a fresh container checked each patch for the required behavior and regressions.

Those controls make the comparison useful, but the result still has uncertainty. Datacurve gives Luna a 95% run-to-run interval of 63.2% to 71.2%, while Sol’s interval is 68.0% to 70.8% [2]. The ranges overlap, so the 2.2-point difference is smaller than the observed variation between runs. Datacurve does not publish a statistical comparison of the two models on the same tasks, so the data does not establish that Sol’s 2.2-point lead is repeatable.

The two configurations also use different effort settings: max for Luna and high for Sol. The ranking changes when I use Datacurve’s empirical Pass@4 measure, which asks whether at least one of the recorded attempts solved each task. On that measure, Luna covers 90.3% of the 113 tasks and Sol covers 86.7% [2]. The per-attempt score favors Sol, while task coverage favors Luna, because the two measurements answer different questions.

Measurement	Luna Max	Sol High
Attempt success, %	67.2	69.4
95% run-to-run interval	63.2–71.2%	68.0–70.8%
Attempts scored	448	451
Tasks with ≥1 success, %	90.3 (Best value in this row)	86.7
Average cost per attempt, $	0.61 (Best value in this row)	3.47
Output tokens per attempt, K	73.4	28.5 (Best value in this row)
Agent steps per attempt	101.7	36.9 (Best value in this row)

The full comparison gives me a reason to test both models, but not to treat them as equivalent. Every measurement in Figure 1 comes from DeepSWE’s tasks and agent setup. My own repository may reward different strengths and expose different failures.

Why can two close scores feel so different?

Two models can pass the same number of tasks while requiring very different amounts of work from the developer. I notice whether a model finds the right files, preserves the existing design, asks before making a risky assumption, recovers after a failed command, and stops when the requested change is complete. DeepSWE reduces the final outcome to pass or fail, so it does not grade those behaviors separately.

DeepSWE’s raw data does provide one clue. Luna Max averaged 101.7 agent steps and 73,400 output tokens per attempt, compared with 36.9 steps and 28,450 tokens for Sol High [2]. Luna therefore used about 2.8 times as many steps and 2.6 times as many output tokens to reach a similar success rate. The data does not tell me whether those extra steps were useful, but it clearly shows that the models did not work in the same way.

The displayed cost needs context too. Luna’s trials ran on July 7, when the recorded token usage cost about $3.03 per attempt [2] [3]. OpenAI cut Luna’s token prices by 80% on July 30 [4], and DeepSWE recalculated the old usage at the new rate. That is how $3.03 became the displayed $0.61. The lower number is a valid estimate of what the same usage costs now, but it is not what Datacurve paid when it ran the test.

API cost per attempt is also different from the cost of getting a change accepted. The leaderboard price leaves out retries, review time, manual edits, waiting, and any second model used to inspect or repair the first model’s work. A model can cost one-sixth as much per attempt and still cost more overall if an engineer has to supervise every decision.

These working differences are what the informal “big model smell” is trying to describe. The useful questions are concrete: Did the model keep track of every constraint? Did it make sensible choices for this repository? Did it avoid unrelated changes, state uncertainty clearly, and fail in a way that was easy to notice and reverse? A single pass rate cannot answer any of them.

How much can the benchmark setup change the score?

A coding benchmark tests a complete system, not the model alone. The prompt, tools, context policy, runtime limits, number of attempts, and grader all affect the result. Changing one of them can move the score even when the model itself stays exactly the same.

The NeurIPS SWE-agent paper provides a clean example. With the same GPT-4 Turbo model on SWE-bench Lite, a shell-only interface solved 11% of tasks, while the full SWE-agent interface solved 18% [6]. When the researchers replaced repeated search output with a tool that summarized the results, the score rose from 12% to 18%. Limiting the file viewer to 100 lines instead of returning a whole file moved it from 12.7% to 18%. These interface changes added five to seven percentage points without changing the model.

Attempt count changes the headline even more. Six runs of the same GPT-4 and SWE-agent setup averaged 17.94% on the first attempt. Counting a task as solved when any of six attempts passed raised coverage to 32.67% [6]. That 14.73-point increase can be useful if a production system really runs six candidates and can identify the correct one. It is misleading when placed beside a single-attempt result without a clear label.

Infrastructure can add noise too. Anthropic held the Claude model, harness, and Terminal-Bench 2.0 task set constant, then changed CPU, memory, and runtime limits. The uncapped setup scored about six percentage points higher, while infrastructure errors fell from 5.8% to 0.5% [7]. This is why a leaderboard entry should be read as a result for a tested system, not a permanent number attached to a model name.

System change	Before	After	Change
Shell only → SWE-agent interface	11.0%	18.0%	+7.0 pp
Whole-file → 100-line viewer	12.7%	18.0%	+5.3 pp
One → six attempts	17.94%	32.67%	+14.73 pp
Strict → uncapped runtime	Not published	Not published	about +6 pp

All four changes in Figure 2 are larger than the 2.2-point gap between Luna and Sol. That does not make benchmarks useless, but it does make the setup part of the result. Before I compare two scores, I check that they use the same task version, agent setup, resources, attempt count, and grading rule.

Why can a passing patch still be rejected?

A benchmark pass means that a patch passed the benchmark’s checks. That is useful evidence that the code works, but it is not the same as a code review. The checks may miss an incomplete fix, and they may not consider maintainability, fit with the existing architecture, unrelated changes, or the work required before the patch can be merged.

EvalPlus showed how strongly the selected tests can affect the result. The researchers expanded HumanEval’s test suites by roughly 80 times and evaluated 26 models. Under the stronger tests, pass rates for the most affected models fell by 19.3 to 28.9 percentage points, and some model rankings reversed [8]. The answers stayed the same; the larger test suites simply caught more errors.

Repository benchmarks have the same problem. In February 2026, OpenAI audited 138 SWE-bench Verified tasks that o3 failed inconsistently across 64 runs. It found material issues in 59.4% of that selected group, including tests that rejected valid solutions and tests that required behavior not mentioned in the issue [9]. The audit deliberately focused on suspicious tasks, so 59.4% does not describe all 500 tasks. It does show that errors in the tasks and tests can distort differences between highly capable models.

The opposite error matters just as much: a patch can pass the automated tests and still be unfit to merge. METR asked maintainers to review 296 AI-generated pull requests across three repositories and 95 tasks. Their acceptance rate was 24.2 percentage points lower than the automated SWE-bench score [10]. They rejected patches for failures in core functionality, unrelated breakage, poor code quality, and other integration problems. A binary benchmark score hides all of those reasons behind the same “fail” or “pass” label.

more tests in EvalPlus

80×

some rankings reversed

issues in targeted SWE-bench audit

59.4%

not representative of all 500 tasks

automated-to-maintainer gap

24.2 pp

METR review of 296 pull requests

Together, these results show that test quality and human review answer different questions. “The patch passed” is evidence worth keeping, but it does not tell me whether I want the model making changes in my branch every day.

What does an average hide about reliability?

An average pass rate combines every success and failure into one number. It does not show whether the model succeeds consistently, whether reliability falls on longer tasks, or whether a failed run produces a small bad diff or damages the working state. Those differences matter when I use the model repeatedly rather than testing it once.

The tau-bench paper makes the repeatability problem visible with pass^k, which asks whether an agent succeeds on the same task every time across multiple trials. GPT-4o achieved 61.2% on retail tasks in a single-trial view, but its retail pass^8 fell below 25% [11]. A model that works three times and fails the fourth may still have a respectable average. As a daily collaborator, it feels unpredictable.

Task length creates a similar split. METR evaluated agents on 170 tasks with about eight runs per model-task pair. The task length at which a model succeeded 80% of the time was four to six times shorter than its 50% success horizon [12]. A model may therefore earn credit on impressive two-hour tasks at a 50% threshold while remaining dependable only on much shorter work.

This helps explain why hands-on impressions can disagree with a leaderboard. Daily use includes repeated tasks, repository context that is easy to miss, broken commands, ambiguous requirements, review cycles, and the consequences of the worst failed runs. Personal impressions can still be wrong because one polished answer or one disastrous failure may dominate my memory. Instead of choosing between benchmarks and intuition, I need to measure the parts of my workflow that created that impression.

How should I compare models for my own work?

I use public benchmarks as a filter. They tell me which models are worth spending time and money on, and a controlled benchmark such as DeepSWE is much better evidence than a launch demo. The final choice still comes from a small evaluation built from my own work.

Anthropic recommends starting an agent evaluation with 20 to 50 tasks drawn from the manual checks, common failures, bug reports, and user requests already seen in development [13]. That is enough to reveal large mismatches without pretending to produce a universal leaderboard. For a close comparison, I would run the same tasks more than once and keep the environment, tools, prompts, effort level, and budget fixed.

Before looking at the outputs, I would define what acceptance means. Tests come first, but they are not the whole rubric. I would also record:

• whether the result was accepted on the first attempt;
• total time to an accepted result, including review and repair;
• active review minutes rather than model latency alone;
• clarifications, redirections, restarts, and manual edits;
• unnecessary files changed and violations of repository conventions;
• severe failures such as security regressions, data loss risk, or unrelated breakage;
• tokens and API cost for the complete accepted result.

For code quality, I would remove model names and review competing patches in random order when practical. I would report results by task type instead of combining bug fixes, reviews, refactors, and long-running feature work into one total. If two models remain close across repeated tasks, I would choose the cheaper one. If its higher review cost consumes the API saving, however, it is not the cheaper model for my workflow.

The DeepSWE result is strong evidence that Luna Max deserves a proper trial. Its success rate falls inside the uncertainty around Sol High, and its current API price is much lower. That still does not make the models interchangeable. To decide which one is cheaper and better for my work, I have to count the steering, review, repair, and failed runs that the leaderboard leaves out.