Can better research handoffs make AI experiments reproducible? An unreviewed benchmark found large gains—but at a substantial construction cost
Across 1,029 tasks from 36 machine-learning papers, four coding agents reproduced 76.4% of tasks from released code and 92.6% from refined artifacts on average. The intervention bundled code repair, environments, launchers and guides, so the study cannot isolate a single cause or certify scientific validity.
The short version
- What happened
- The authors began with 742 ICML 2026 Spotlight papers, retained 36 test papers after code, theory, compute and access screening, and created 1,463 candidate tasks. Five independently executed reference observations supported 1,029 downstream tasks from 387 experiments.
- Who it affects
- Researchers may spend less time rediscovering dependency fixes and score-extraction steps when the recovered procedure is packaged with the code.
- What remains unproven
- Unreviewed preprint using 36 screened ICML Spotlight papers; it is not a conference-wide or field-wide reproducibility estimate.
Keep in mind
Unreviewed preprint using 36 screened ICML Spotlight papers; it is not a conference-wide or field-wide reproducibility estimate.
Five reference observations may not characterize every task’s uncertainty, and numerical compatibility cannot certify scientific validity.
The intervention bundles repair, environment preparation, launchers, documentation, review and packaging, limiting causal attribution to any one component.
The primary automated auditor missed eight of 39 invalid outcomes in the human-verified subset.
Reported downstream API savings exclude construction, GPU execution, auditing, maintenance, human review and human comprehension.
Four authors list Google affiliations, the study evaluates commercial models, and the accessible manuscript lacks a separate competing-interest statement.
Source analysis · 1 linked source. Primary analysed source date: 5 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The answer: preserving execution knowledge helped the tested agents, but the result is not a general reproducibility rate
An unreviewed preprint reports that four coding agents reproduced machine-learning results more often when they received a deliberately refined research artifact rather than the authors’ released code alone. Averaged equally across the four models, task success rose from 76.4% to 92.6%. The refined handoff combined four things: a repaired codebase, a prepared environment, experiment launchers and a guide explaining settings, paper–code discrepancies and measurement procedures. Every tested model improved, although the size of the gain ranged from 5.5 percentage points for the strongest raw-code reproducer to 23.2 points for the weakest.
That is evidence that operational knowledge can matter as much as model capability in this benchmark. It is not evidence that 92.6% of published machine-learning research is reproducible. The evaluation included a screened subset of 36 ICML 2026 Spotlight papers whose artifacts, compute and access requirements fit the protocol. Success meant producing a numerically compatible result without a confirmed audit violation. It did not certify that the original scientific claim was correct, that every experimental choice was faithful or that a human researcher would find the artifact understandable. The authors explicitly warn that refined artifacts can preserve a mistaken assumption.[1]
How 742 papers became 36—and how the 1,029-task denominator was built
The test-pool construction is central to interpreting the headline. The researchers began with 742 ICML 2026 Spotlight papers. They identified 241 with released code and content that was not purely theoretical, then randomly sampled 50. Six were excluded for artifact or code availability, four for compute requirements and four because the experiments depended on closed-model access. The final test set contained 36 papers across eight topic families. A separate ten-paper set from NeurIPS and ICLR was used during development and reported separately rather than mixed into the main estimate.
From the 36 test papers, the team curated 1,463 own-method measurements from 502 experiments. Five human reproducers, each working on a separate server from the same pinned raw-code commit, attempted reference executions. Coding-agent assistance was allowed, but people inspected configurations, scripts, edits and score extraction. The five observations were used to estimate a prediction interval for each task. A total of 1,029 tasks—701 classified as stable and 328 as more variable—had reference results compatible with the published value and entered the downstream benchmark. Another 183 had numerical mismatches, 186 were blocked by artifacts or code, and 65 hit resource or access limits.[1]
The distinction that matters
What entered the benchmark
The headline percentage applies to a screened, reference-verified subset—not all machine-learning papers.
Starting pool
742 Spotlight papers
241 had code and non-theoretical content
Test papers
36 retained
After availability, compute and access exclusions
Verified tasks
1,029 of 1,463
Compatible five-run reference results
Agent conditions
Raw versus refined
76.4% versus 92.6% mean success
What the intervention changed and how fresh sessions were tested
ReproBridge turns the work of making an experiment run into a reusable package. Agents first map the intended measurement from the paper, prepare dependencies and data, execute the experiment, record justified repairs and then package the result. A reviewer operating in a separate context challenges unsupported choices and the finished entry points are tested from a clean shell. The final artifact contains executable code and launchers, but also a change record explaining why a dependency was pinned, a loader repaired or a reporting statistic aggregated in a particular way. Hidden human reference scores are withheld during construction.
For evaluation, fresh sessions of Sonnet 4.6, Sonnet 5, Opus 5 and Gemini 3.8 Flash received the paper, the selected tasks and either raw code or the refined artifact. Each paper–model–condition session had a 24-hour wall-clock limit. GPU experiments used eight H100 80GB accelerators. Opus 5 constructed the refined artifacts for the main experiment, and each condition was evaluated three times. The primary automated auditor traced submitted values back to scripts, timestamps and output files, while a finder–refuter process tested suspected violations. Missing results and results outside the fixed compatibility interval failed.[1]
The gains were consistent, while audit quality and cost accounting qualify the result
Sonnet 4.6 improved from 72.3% to 91.6% mean task success; Sonnet 5 from 76.5% to 93.3%; Opus 5 from 89.2% to 94.7%; and Gemini 3.8 from 67.6% to 90.8%. The gap between the best and worst models narrowed from 21.6 to 3.9 percentage points. Downstream agent token use fell between 6.3% and 19.8%, and the reported API bill for those reproducer sessions fell between 16.8% and 23.7%. A separate 45-capsule CORE-Bench Hard test also favoured refined artifacts, with success rising from 62.2% to 73.3% and median session duration falling from 33 to 14 minutes.
The audit was not perfect. Two people cross-verified 818 outcomes from eight randomly selected papers. Against those labels, the primary Codex auditor had 88.6% precision and 79.5% recall, missing eight of 39 invalid outcomes. Uncertain findings were not counted as invalidations. The construction cost was also material: $5,026 in reported API use, or $139.61 per paper, before GPU use, auditing, maintenance and human review. The paper’s break-even calculations require repeated reuse and fixed model prices, and they exclude human time. Lower downstream API spend should therefore not be confused with lower total research cost.[1]
What this could change for researchers, maintainers and institutions
For authors, the strongest practical idea is to treat reproduction instructions as part of the research object rather than an afterthought. A pinned environment, one-command launchers, explicit seeds, output locations and a record of paper–code differences make it easier for another person—or another agent—to determine what was actually run. Running this process before submission could expose broken dependencies or ambiguous reporting while the original team can still correct them. For maintainers, a structured guide may reduce repeated support requests and make later model or hardware migrations more auditable.
Institutions should resist turning the benchmark into a requirement that every project buy an expensive agent workflow. The selected experiments were computational machine-learning studies with released code; qualitative research, clinical trials, proprietary data and field studies have different reproducibility constraints. A refined artifact can also make an erroneous procedure easier to repeat. Human judgment is still needed to decide whether the implementation matches the research question, whether a repair changes the estimand and whether restricted data or security rules permit reuse. The result supports better handoffs, not the idea that an agent’s green check is scientific peer review.[1]
What would change the assessment
Peer review and independent replication are the first tests. A stronger evaluation would preregister the paper selection, repeat the study with other conferences and less curated repositories, and include experiments that fail the initial availability screen. Human researchers should be randomly assigned raw and refined artifacts so the study can measure time, comprehension, error detection and confidence rather than infer human benefit from agent behaviour. A factorial design could separate the contribution of repaired code, environments, launchers and documentation instead of evaluating a bundled workflow. Reviewers should also examine whether fixes silently alter the original method.
For now, the defensible conclusion is that executable context mattered greatly for four current coding agents on a carefully verified benchmark. The result is especially useful because the paper reports selection attrition, reference variability, audit errors and excluded costs. It remains an unreviewed study produced largely by researchers affiliated with Google and evaluates commercial models, including a Google model. The assessment would strengthen if independent groups reproduced the gains, if human users benefited, and if refined artifacts improved transparency without preserving incorrect assumptions or making expensive infrastructure a condition of scientific participation.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring coding agents: compare the finding and its evidence limits.
Can an LLM selector make coding agents more reliable? Gains came with up to 8.7× the tokens
On 89 terminal tasks, generating four or eight candidate actions at every turn and letting a stronger model choose improved reported accuracy by 5–10 percentage points. Executed trajectories shortened, but total token use rose sharply and the preprint has not been peer reviewed.
Read the next reportExplore the full source and evidence record
What this means for people
- Researchers may spend less time rediscovering dependency fixes and score-extraction steps when the recovered procedure is packaged with the code.
- Lab leaders and funders need to count construction, GPU, audit, maintenance and human-review costs rather than quote downstream API savings alone.
- Readers should not treat a reproduced number as proof that a scientific conclusion is valid; reproduction is one layer of scrutiny, not a substitute for it.
Global context
Machine-learning reproducibility problems affect labs well beyond the North American institutions behind this work. Yet access to H100 GPUs, paid model sessions and dedicated artifact engineering is highly unequal. Lower-cost, open tooling and tests on modest hardware will matter if standardized handoffs are to reduce rather than deepen participation gaps.
What the evidence does not yet show
- Unreviewed preprint using 36 screened ICML Spotlight papers; it is not a conference-wide or field-wide reproducibility estimate.
- Five reference observations may not characterize every task’s uncertainty, and numerical compatibility cannot certify scientific validity.
- The intervention bundles repair, environment preparation, launchers, documentation, review and packaging, limiting causal attribution to any one component.
- The primary automated auditor missed eight of 39 invalid outcomes in the human-verified subset.
- Reported downstream API savings exclude construction, GPU execution, auditing, maintenance, human review and human comprehension.
- Four authors list Google affiliations, the study evaluates commercial models, and the accessible manuscript lacks a separate competing-interest statement.
What to watch next
- Peer review and independent reproduction of the benchmark results.
- Randomized studies measuring human researcher time, comprehension and error detection.
- Tests on unscreened repositories, smaller labs and research outside machine learning.
- Factorial comparisons separating code repair, environments, launchers and guides.
- Open, lower-cost implementations that retain audit evidence without requiring frontier hardware.
Living evidence record
Impact record IAI-0350X77
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
Whether packaging recovered execution knowledge into a standardized research artifact improves independent agent reproduction of machine-learning experiments
Topics: Research reproducibility · Coding agents · Machine learning · Scientific infrastructure · Benchmarks · Research costs
Regions: Canada · United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.