AI papers share far more code and data—but are they really more reproducible?
A peer-reviewed analysis of 56,800 conference papers found large gains in open artefacts and documentation from 2014 to 2024. Its 64% reproducibility estimate is modelled, not the result of rerunning thousands of studies.
The short version
- What happened
- Researchers analysed 56,800 papers from five major AI conferences between 2014 and 2024; 52,328 were classified as empirical.
- Who it affects
- Researchers, reviewers and funders get more material with which to inspect and reuse AI results. But a repository link is not a guarantee that another team can reproduce a claim within realistic time and computing budgets.
- What remains unproven
- The study covers five leading conferences, not journals, workshops or the full global AI literature.
Keep in mind
The study covers five leading conferences, not journals, workshops or the full global AI literature.
A language model classified documentation indicators; manual evaluation found lower accuracy for several nuanced categories.
Estimated reproducibility relies on a small earlier reproduction sample and is not a direct rerun of the corpus.
The observational checklist analysis cannot isolate policy effects from existing trends or venue selection.
Funding came from the US National Science Foundation, the Research Council of Norway and an Alexander von Humboldt Professorship; the paper reports no commercial sponsor for the analysis.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
Open artefacts and basic reporting improved sharply
The authors collected 56,800 papers from AAAI, ICLR, ICML, IJCAI and NeurIPS over 2014–2024, then identified 52,328 as empirical. They checked seven features: open code, open data, dataset-split details, pseudocode, hardware, software dependencies and experiment setup. Across the decade, the share reporting both open code and open data rose from 11% to 64%. Papers documenting at least five of the seven indicators increased from 8% to 43%. Hardware and setup reporting also improved, though unevenly across venues and topics.[1]
A language model did the large-scale classification
Gemini 2.5 Flash classified the PDFs after prompt development on 400 manually annotated papers. The researchers then evaluated the final classifier on 160 randomly sampled papers. F1 scores exceeded 90% for most indicators, but were lower for dataset splits (81.48%), dependencies (81.25%) and experiment setup (86.21%). That is adequate for broad trend analysis, not perfect measurement of every individual paper. The authors publish their paper-level indicators, code and analysis data on Zenodo, making the trend calculation open to reanalysis.[1]
The distinction that matters
What changed—and what was inferred
The study separates observable reporting from an estimated chance of reproduction.
Open code + data
Compare conference papers across a decade
11% in 2014 → 64% in 2024
Five or more indicators
Count seven documentation features
8% in 2014 → 43% in 2024
Estimated reproducibility
Map documentation to prior reproduction results
28% → 64%, not directly retested
The 64% figure is an estimate, not a reproduction rate
The study estimates that reproducibility rose from 28% in 2014 to 64% in 2024. It derives that estimate by linking documentation patterns to a much smaller prior reproduction exercise, where six of seven papers with open code and data were reproduced, compared with five of 15 that shared datasets alone. The new study did not rerun 56,800 experiments, verify that repositories still execute or test whether released artefacts reproduce claimed results. Documentation is a necessary signal, but it can overstate practical reproducibility when versions, licences, compute or hidden preprocessing are missing.[1]
Checklists did not show a clear additional acceleration
The paper compared trends around conference reproducibility checklists and found no statistical evidence of an overall acceleration after adoption: 15 of 35 indicator trends were higher after a checklist, with p=0.84 and a small negative aggregate effect size. That cannot establish that checklists do nothing. Adoption years differed, reporting was already improving and authors select which venues to enter. Stronger evidence would directly rerun representative samples, test whether artefacts remain usable years later and compare matched venues or policy changes. For now, the robust result is narrower: AI research became much more open and better documented, while actual repeatability remains less certain.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
Google DeepMind · 1 min 52 sec · Background explainer
Protein folding explained
YouTube loads only when you choose; playback starts when you press play.
A short introduction to why a protein’s shape matters. It provides background for understanding what AI structure prediction contributes to biological research.
Evidence check: This is the developer’s explanation. Predicting a structure does not by itself establish a treatment’s safety or benefit for patients.
Published 2020-11-30. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current researchContinue the story
Read next
Continue exploring open science: compare the finding and its evidence limits.
Can researchers rerun clinical AI?
A peer-reviewed scoping review found accessible analytical code in 12.2% of 3,967 prediction-model papers that cited TRIPOD or TRIPOD+AI. Even among shared repositories, dependency versions, tests and reusable structure were often missing, so code availability alone did not establish reproducibility.
Read the next reportExplore the full source and evidence record
What this means for people
- Researchers, reviewers and funders get more material with which to inspect and reuse AI results. But a repository link is not a guarantee that another team can reproduce a claim within realistic time and computing budgets.
Global context
The conferences are international but disproportionately represent well-resourced research communities. Open artefacts can widen access, while large compute requirements, licences and unavailable training data can still prevent meaningful replication outside leading institutions.
What the evidence does not yet show
- The study covers five leading conferences, not journals, workshops or the full global AI literature.
- A language model classified documentation indicators; manual evaluation found lower accuracy for several nuanced categories.
- Estimated reproducibility relies on a small earlier reproduction sample and is not a direct rerun of the corpus.
- The observational checklist analysis cannot isolate policy effects from existing trends or venue selection.
- Funding came from the US National Science Foundation, the Research Council of Norway and an Alexander von Humboldt Professorship; the paper reports no commercial sponsor for the analysis.
What to watch next
- Direct reruns of representative paper samples
- Long-term checks that code, data and dependencies remain usable
- Coverage beyond five major conferences
- Policies that require versioned environments and compute disclosures
- Independent reanalysis of the published Zenodo dataset
Living evidence record
Impact record IAI-00CM4E5
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
22 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Communications AI & Computing published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 22 October 2026
Research question, topics and regions
Documentation and reproducibility trends in AI research
Topics: Research reproducibility · Open science · AI conferences · Research methods · Code and data sharing
Regions: Global · Europe · North America
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.