Can a saturated AI benchmark still reveal capability?

How the retrospective test was constructed

The researchers grouped model releases into six-month periods. For each prediction task, seven or eight models from the preceding period acted as references and eight models from the next period became targets. The main analysis used two target periods, producing 16 target-model predictions. Within each domain, the three strongest reference models defined saturation: an item counted as saturated only when all three answered it correctly. Any item at least one of them missed went into the harder, unsaturated set.

The protocol could see only correctly solved transcripts from the saturated items. It could not see the model's failures or the unsaturated questions. The researchers then fitted item-response models to the unsaturated set to create the capability target the protocols were trying to predict. Up to 100 saturated items were sampled per task, distributed across four item-difficulty quartiles. This design deliberately removed ordinary accuracy as a differentiator and asked whether process traces carried additional information.

The four domains combined established evaluations. Science included GPQA-Diamond, a fixed 200-question ARC-Challenge sample and science-tagged questions from a verified Humanity's Last Exam subset. Mathematics combined MMLU-Pro, AIME, APEX and HLE items. Coding used AgentBench OS, 50 SWE-bench Verified Mini tasks and 89 Terminal-Bench tasks. Cybersecurity used 39 CyBench challenges and 78 InterCode capture-the-flag challenges. The paper evaluated pairwise judging, single-transcript scoring, transcript embeddings and simple features such as length, hedging and tool-call counts.[1]

What improved, and where the information appeared

Across the 16 main target models, pairwise judging had mean squared prediction error of 0.26, with a reported 95% confidence interval from 0.18 to 0.34. The score-only baseline error was about 5.27 times larger. A likelihood comparison also favoured pairwise judging over the baseline (mean log-likelihood ratio 0.68, p=0.005). The feature, embedding and absolute-scoring approaches had lower error than the baseline numerically, but none produced a statistically significant likelihood advantage.

Ablations suggest that useful information was not merely polished writing. Removing visible reasoning increased error in coding, cybersecurity and science. Removing tool interactions increased cybersecurity error from 0.20 to 0.79, while making almost no difference in coding. Removing ordinary assistant messages did not significantly change error in any domain. This is evidence that the diagnostic channel depends on the work being evaluated, not that hidden reasoning is a universal measure of ability.

Pairwise judging is also expensive and reference-dependent. Each target transcript must be compared with calibrated reference transcripts, often in both display orders to reduce position bias. The appendix reports diminishing returns from adding more items when systematic judge bias dominates sampling noise. Strong anchor models in the reference set mattered, particularly for frontier targets.[1]

What this means for evaluators and public decision-makers

For evaluation teams, a ceiling score should not automatically consign a benchmark to the archive. Preserved transcripts can support a secondary analysis of strategy, verification, tool use and error recovery. That may buy time while harder tasks are designed and validated. It also argues for retaining structured execution traces, model and tool versions, prompts and item-level outcomes rather than publishing only a final percentage.

For regulators and safety institutes, the result is a warning against converting the new estimate into a pass-fail threshold without prospective validation. The measured target was performance on harder questions from the same broad evaluation ecosystem, not safe deployment, malicious capability or real-world reliability. A protocol that underestimates a new reasoning paradigm could create false reassurance precisely when model behaviour changes most.

A responsible use would keep the transcript estimate alongside unsaturated evaluations, expert review and uncertainty. It should be recalibrated when providers change reasoning visibility, tool interfaces or model architecture. Closed systems that hide internal traces also create an access problem: the main analysis favoured open-weight reasoning models with visible traces, although the paper includes separate transfer tests to closed models.[1]

Evidence limits and the test that would change the assessment

This was a retrospective simulation, not a prospective prediction challenge. The authors knew the historical models and could measure their later performance on held-out items. Item-response modelling supplies a useful common scale, but that scale still depends on the selected benchmarks, item parameters and model panel. Seven or eight reference models per period is a small calibration set, and judge behaviour can change with the judge model, prompt and visibility of reasoning.

The decisive next step is preregistered prospective testing. An evaluation body should freeze its items, reference models, judge protocol and uncertainty rules before a genuinely new model family arrives, then compare the prediction with results on independent harder tasks and expert red-team exercises. Repeating that across providers, hidden-reasoning systems and safety-relevant domains would show whether transcript analysis remains informative when it matters. Until then, it is a promising salvage method for saturated tests, not a substitute for new evidence.[1]