Can a coding-agent benchmark become wrong after release? 28 of 89 Terminal-Bench tasks needed repair
An audit of Terminal-Bench 2.0 found instruction mismatches, dependency drift and resource failures. The repaired 2.1 release kept the broad ranking but shifted individual scores by as much as 12 points and positions by three places.
The short version
- What happened
- Terminal-Bench 2.1 revised 28 of 89 tasks: 57% of repaired tasks had instruction-verifier mismatch, 43% task drift and 29% environment issues, with categories overlapping.
- Who it affects
- Engineers and employers can make poor tool choices if a few leaderboard points reflect timeouts or broken tests rather than useful coding capability.
- What remains unproven
- The work is an unreviewed preprint, and its authors participated in producing the repair set used to evaluate workflow precision and recall.
Keep in mind
The work is an unreviewed preprint, and its authors participated in producing the repair set used to evaluate workflow precision and recall.
Nineteen agent configurations do not represent every coding task, model, harness or workplace.
The extended-budget analysis covers six agent-model pairs and deliberately departs from the default protocol.
Rank correlation can remain high while individual scores and close comparisons change materially.
The external Harbor sample included 34 of 80 tasks and still contained defects missed by the workflow.
Source analysis · 1 linked source. Primary analysed source date: 5 November 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The direct answer: executable benchmarks accumulate maintenance debt like software
A Stanford and Cornell team audited Terminal-Bench 2.0, a suite of 89 command-line tasks used to compare coding agents. They found that 28 tasks—about 31%—needed repair before release as Terminal-Bench 2.1. Some instructions and tests disagreed; some external packages or services had changed; and some memory, disk or time limits were so tight that a correct reference solution or verifier could fail. These are measurement errors: an agent can lose credit for reasons unrelated to the capability the task is supposed to test.
The repairs did not overturn the broad leaderboard. Across 19 agent configurations, old and new scores had a Spearman rank correlation of 0.958, with a task-bootstrap 95% interval from 0.881 to 0.979. Yet individual accuracy shifted by as much as 12 percentage points and rank by three places. That combination is the central finding: a benchmark can preserve the overall story while materially changing close comparisons, absolute capability claims and purchasing decisions based on a few points.[1]
What failed across the 28 repaired tasks
Instruction-verification mismatch appeared in 57% of repaired tasks. In one database task, the verifier demanded byte-for-byte preservation even though the instruction never said so. Task drift affected 43%: package versions, public APIs and web content changed after release. Environment issues affected 29%: the oracle, verifier or agent reached resource limits, or execution backends isolated files and networks differently. Percentages overlap because one task can fail in several ways.
Fourteen of the final 28 were newly identified by this audit. Eleven had been flagged by an earlier audit, five more appeared in public issues or pull requests, and two tasks overlapped across sources. Repairs followed conventions: accept every solution satisfying the written instruction while rejecting invalid ones; pin or remove mutable dependencies; and give the known-correct oracle and verifier at least 1.5 times their observed requirement on binding resources. Maintainers documented difficulty changes instead of tuning pass rates back to old values.[1]
The distinction that matters
Why a benchmark score can change without the model changing
Task software, verifiers and resource budgets are all part of the measuring instrument.
Specify
Write the instruction and intended outcome
Can diverge from the verifier
Execute
Run in a changing environment
Dependencies and limits can fail
Verify
Grade the final state
Tests can reject valid work or accept shortcuts
Version
Repair and rerun agents
Scores and close ranks can move
The automated workflow proposes evidence, but humans keep the merge gate
The workflow collects task files, prior agent traces and community reports. An auditor applies a rubric covering specification fidelity, tests, resources and risk, then proposes a cited label and repair. A verifier challenges the diagnosis, runs the oracle and checks whether the patch addresses the underlying defect. Runs are capped at five repair iterations and emit a reviewable diff; agents cannot merge changes directly. That propose-only design treats automation as triage and engineering assistance rather than an autonomous benchmark authority.
Claude Code with Opus 4.6 and Codex with GPT-5.4 agreed on the primary category, including no defect, with Cohen's kappa 0.80 across all 89 tasks. Lower-cost auditors agreed on binary flagging at kappa 0.55 to 0.77, but agreement on task drift was only 0.21 to 0.41. The deterministic oracle check therefore carries much of the drift burden. Against the final 2.1 repair set, precision was 57.1% and recall 71.4%; because the workflow helped construct that set, those are retrospective consistency measures, not an independent test.[1]
Repair moved scores unevenly because agents solve different kinds of tasks
The audit re-ran 19 agents on versions 2.0 and 2.1. Claude Code with Claude Opus 4.6 rose more than seven points above the fitted old-to-new relationship and climbed three places, while Codex with GPT-5.4 fell below the fit. The two weakest agents lost accuracy. Nineteen repaired tasks were classified as hard, seven medium and two easy, so making hard tasks feasible mainly benefits systems already capable of reaching plausible solutions.
Five instruction-verification repairs moved pass rates from no more than 15% to more than 50%, accounting for about 62.7% of the absolute pass-rate movement across modified tasks. Other changes moved in both directions: a broader verifier can recognise valid work, while a stricter test can remove false-positive credit. Maintaining a benchmark should not mean maximising scores. The goal is fidelity to the intended task, accompanied by versioned results and a public explanation whenever the measured object changes.[1]
Time limits can make one harness appear far smarter than another
More than half of unsuccessful 2.1 trials ended in timeout at the default budget, ranging from 68% under Claude Code and 67% under Terminus-2 to 25% under Codex and 18% under Gemini CLI. The researchers increased wall-clock limits up to 32 times their defaults for six agent-model pairs. Agents could stop early, so the larger ceiling did not require consuming the full allocation. Most timeouts disappeared at the largest budget, although several smaller models still plateaued below full coverage.
At the standard budget, changing between Codex and Terminus-2 moved GPT-5.4 accuracy by about 20 points. With extended time, Codex scored 74.9% and Terminus-2 75.3%, effectively closing the gap. For GPT-5.4 mini, the gap shrank from roughly 30 points—67.0% versus 37.1%—to about five points, 68.2% versus 62.9%. A fixed timeout therefore mixes reasoning capability with software speed and harness overhead. Buyers should ask which budget, sandbox and interface produced a headline number.[1]
External checks help, but continuous validation still misses defects
To test transfer, the workflow audited 34 randomly selected tasks from an 80-task Harbor-Index snapshot spanning 29 source benchmarks. It flagged 14; human review confirmed 12, an 86% precision among flags. Reviewers also found missed defects, including a wrong reference answer, nondeterministic grading and an instruction allowing two plausible answers. Two shared-code bugs affected 13 tasks. Automated triage helps, but domain review, repeated execution and shared-infrastructure tests remain necessary.
Confidence would rise with prospective audits on untouched benchmarks, blinded human adjudication and published estimates of reviewer agreement and cost. Reports should name the exact task version, harness, model snapshot, sample count, wall-clock and output-token limits, then test sensitivity around those choices. The paper received extensive public and industry support, including funders and firms connected to AI infrastructure and evaluation; open artefacts and outside replication therefore matter. A benchmark score should be treated as a versioned measurement, never a permanent fact about a worker or model.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring coding agents: compare the finding and its evidence limits.
Which coding agent works best when a bug report unfolds over several turns?
In an unreviewed 72-task benchmark, the one-turn leader was not the eight-turn leader. The result suggests that coding-agent rankings depend on how long users can clarify a problem—but the users were simulated and the benchmark was built by Tencent researchers.
Read the next reportExplore the full source and evidence record
What this means for people
- Engineers and employers can make poor tool choices if a few leaderboard points reflect timeouts or broken tests rather than useful coding capability.
- Benchmark maintainers need an ongoing review budget, version history and human authority over automated repair proposals.
- Workers assessed with agent benchmarks deserve clarity about which tasks, resources and software versions underlie claims about automation or substitution.
Global context
Coding-agent benchmarks influence product marketing, research funding and forecasts about software work. Executable suites are more realistic than static questions, but they inherit software maintenance burdens: package drift, infrastructure differences, ambiguous specifications and resource ceilings. Versioned continuous validation can improve trust when artefacts are public. It cannot remove judgment about what counts as a valid solution or which resource budget represents real work.
What the evidence does not yet show
- The work is an unreviewed preprint, and its authors participated in producing the repair set used to evaluate workflow precision and recall.
- Nineteen agent configurations do not represent every coding task, model, harness or workplace.
- The extended-budget analysis covers six agent-model pairs and deliberately departs from the default protocol.
- Rank correlation can remain high while individual scores and close comparisons change materially.
- The external Harbor sample included 34 of 80 tasks and still contained defects missed by the workflow.
What to watch next
- Prospective audits of benchmarks whose repair sets were not shaped by the workflow.
- Standard reporting of benchmark version, harness, timeout, token budget and infrastructure.
- Independent replications of Terminal-Bench 2.1 score and rank changes.
- Public issue-to-repair histories and regression tests for shared benchmark infrastructure.
- Human workplace studies connecting benchmark performance with safe, maintainable software outcomes.
Living evidence record
Impact record IAI-1G24UYK
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
How post-release defects and resource limits alter coding-agent benchmark scores, rankings and claims about harness performance
Topics: Coding agents · Benchmarks · Software maintenance · Evaluation · Preprint
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.