AI agents reproduced 13 of 93 scientific tasks—what failed?
PaperBenchX replayed agent submissions across 12 research areas and scored only regenerated evidence. The best configuration fully completed 13 tasks, while validation remained the weakest stage; 81 tasks and most reference workflows stay private.
The short version
- What happened
- PaperBenchX contains 93 expert-constructed reproduction tasks from 93 papers in 44 venues, spanning 12 research areas and 3,168 scorable criteria.
- Who it affects
- Scientists could use agents to draft models, operate tools or diagnose runs, but weak validation can turn a plausible workflow into unsupported evidence. Researchers and reviewers still need inspectable files, domain judgment and responsibility for the final claim.
- What remains unproven
- Unreviewed preprint submitted 29 October 2026; no independent benchmark replication was identified.
Keep in mind
Unreviewed preprint submitted 29 October 2026; no independent benchmark replication was identified.
Only 12 of 93 tasks are public; 81 tasks and private reference workflows remain controlled by the benchmark team.
A GPT-5.6-sol evaluator applied the expert-written rubrics, so scores include a model-judge layer.
The paper reports one outcome per model–task configuration rather than repeated stochastic trials.
Three public tasks require separately licensed commercial software, limiting equal access.
The project includes industry contributors, and no separate funding or competing-interest statement was identified.
Multi-source analysis · 2 linked sources. Primary analysed source date: 29 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The benchmark requires executable evidence
An unreviewed preprint tests whether AI agents can reproduce published computational experiments outside the familiar software stacks of machine-learning research. PaperBenchX contains 93 tasks drawn from 93 papers in 44 venues, across electromagnetics, photonics, computational chemistry, materials science, bioinformatics, robotics and embodied AI. Experts converted each target experiment into a runnable environment and 3,168 scorable rubric criteria—a median of 38 per task.
An agent receives the paper, a public task specification and access to domain-native software. It must infer the intended model, run the computation and produce a workflow that supports the target claim. Evaluators delete mutable outputs, disable outbound network access and replay each submission from scratch. Only regenerated artefacts count. That design is stronger than checking whether an agent writes a plausible report or matches a reference chart visually.[1][2]
Partial progress rarely became a complete reproduction
Across ten model–scaffold configurations, the best—GPT-6-Astra with Codex—earned 59.08% of available rubric credit on average. It passed the benchmark's operational full-completion threshold of more than 90% on 13 of 93 tasks, or 13.98%. Across all 930 model–task outcomes, only 80 exceeded that threshold. Seventy-one tasks were never fully reproduced by any configuration, and another 11 were completed by no more than two.
The failure pattern is revealing. Macro-average scores were 62.73% for modelling, 61.83% for execution and 42.91% for validation. Agents often reconstructed part of the setup or ran scientific software, but struggled to establish that outputs actually supported the paper's claim. Tasks had time limits from four to 72 hours, with a seven-hour median, so this is a demanding autonomy test rather than a short question-answer benchmark.[1]
The distinction that matters
Where reproduction broke down
The agents made more progress constructing and running experiments than validating what the regenerated evidence meant.
Modelling
Recover assumptions, equations and experiment setup
62.73% macro-average score
Execution
Operate domain software and complete computation
61.83% macro-average score
Validation
Show regenerated evidence supports the target claim
42.91% macro-average score
The result is a warning, not a universal ceiling
The benchmark does not show that AI cannot assist scientists. It tests one demanding form of end-to-end autonomous reproduction, and partial credit may still represent useful work under expert supervision. Nor does a score above 90% prove a scientific claim true; it means the submission satisfied this benchmark's expert-written criteria. The same GPT-5.6-sol evaluator applied every rubric, introducing a model-judge layer even though the underlying criteria were expert validated.
Independent inspection is also limited. Twelve representative tasks are public, while 81 and their private reference workflows remain held out to reduce contamination. Three public tasks depend on commercial software that users must license. Because task selection, reference workflows and scoring choices shape the result, outside teams cannot yet audit most task-level failures or test whether the private set favours particular tools. The project lists contributors from UniPat AI, ByteDance Seed and Peking University, and the paper gives no separate funding or competing-interest statement. The assessment would change with repeated runs, blinded external expert scoring, broader public task access and independent replications on new papers. For now, the evidence says autonomous research agents need especially strong validation gates before their scientific outputs are trusted or published.[1][2]
Watch the background, then check the evidence
Illustrative photo · not a video frame
Google DeepMind · 1 min 52 sec · Background explainer
Protein folding explained
YouTube loads only when you choose; playback starts when you press play.
A short introduction to why a protein’s shape matters. It provides background for understanding what AI structure prediction contributes to biological research.
Evidence check: This is the developer’s explanation. Predicting a structure does not by itself establish a treatment’s safety or benefit for patients.
Published 2020-11-30. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current researchContinue the story
Read next
Continue exploring ai for science: compare the finding and its evidence limits.
Can AI identify an unknown molecule from 1D NMR alone?
An unreviewed system ranked the correct planar structure first for 82.3% of 2,031 benchmark molecules and helped resolve four new natural products. The four-compound demonstration is promising, not prospective proof of routine laboratory performance.
Read the next reportExplore the full source and evidence record
What this means for people
- Scientists could use agents to draft models, operate tools or diagnose runs, but weak validation can turn a plausible workflow into unsupported evidence. Researchers and reviewers still need inspectable files, domain judgment and responsibility for the final claim.
Global context
Scientific automation spans very different software, compute budgets and evidence standards. PaperBenchX broadens evaluation beyond machine learning, but it remains a computational benchmark; it does not cover wet laboratories, human-subject research or the institutional safeguards around real publication decisions.
What the evidence does not yet show
- Unreviewed preprint submitted 29 October 2026; no independent benchmark replication was identified.
- Only 12 of 93 tasks are public; 81 tasks and private reference workflows remain controlled by the benchmark team.
- A GPT-5.6-sol evaluator applied the expert-written rubrics, so scores include a model-judge layer.
- The paper reports one outcome per model–task configuration rather than repeated stochastic trials.
- Three public tasks require separately licensed commercial software, limiting equal access.
- The project includes industry contributors, and no separate funding or competing-interest statement was identified.
What to watch next
- Independent replay of the 12 public tasks
- Repeated runs and uncertainty around configuration scores
- Blinded scoring by external domain experts
- More public tasks without commercial-software barriers
- Validation tools that detect plausible but unsupported outputs
- Performance on new papers not selected by the benchmark team
Living evidence record
Impact record IAI-1QAXOF4
Evidence stage
Studied
Confidence
Supported
Reporting basis
Multi-source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
1 November 2026
Source trail
2 direct sources across 2 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Evidence trail
Sources used for this report
Links checked 1 November 2026
Research question, topics and regions
Cross-disciplinary evaluation of AI agents reproducing published computational experiments
Topics: AI for science · Research agents · Reproducibility · Scientific software · Benchmarking · Preprint
Regions: Asia · China
This report is labelled multi-source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.