Can an AI agent audit cancer studies? A 20-paper test found useful triage—and many false alarms
An unreviewed benchmark reports that an agent reproduced 77.4% of 2,190 claims from US cancer-registry papers. But expert review rejected nine of ten sampled robustness warnings as consequential changes.
The short version
3 key points
- What changed
- The benchmark covers 20 published SEER cancer-registry papers, 901 findings and 2,190 atomic claims; the agent reproduced 1,694 claims, or 77.4%, under the authors’ grading rules.
- Why it matters
- More efficient study audits could help clinicians and patients by exposing fragile evidence sooner. The present evidence supports expert triage, not replacing epidemiologists, statisticians, peer reviewers or clinical judgement.
- What to watch
- Peer review and independent reproduction of the 20-paper benchmark
The agent recovered most claims—but the denominator matters
An AI agent reproduced 1,694 of 2,190 numerical and qualitative claims drawn from 20 papers using the US National Cancer Institute’s SEER registry, according to a new unreviewed preprint. Another 76 claims were classed as close and 420 were not reproduced. The test is substantial: it covers 901 parent findings and a curated environment of 39 data tables, 198 million rows and 40 documentation resources. It does not show that 77.4% of cancer research is correct, or that the remaining claims are wrong.[1]
Who it affects
More efficient study audits could help clinicians and patients by exposing fragile evidence sooner. The present evidence supports expert triage, not replacing epidemiologists, statisticians, peer reviewers or clinical judgement.[1]
Keep in mind
This is an unreviewed preprint from the team that created both the benchmark and the evaluation pipeline.
The benchmark contains 20 SEER-based observational papers; transfer to trials, laboratory research, other registries or non-US data is unknown.
Experts reviewed ten of 55 automated warnings, and nine did not establish a consequential change; unflagged findings were not sampled for missed problems.
The paper reports no end-to-end comparison of expert time, financial cost, downstream corrections or patient outcomes.
Source analysis · 1 linked source. Original source date: 16 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
The agent recovered most claims—but the denominator matters
An AI agent reproduced 1,694 of 2,190 numerical and qualitative claims drawn from 20 papers using the US National Cancer Institute’s SEER registry, according to a new unreviewed preprint. Another 76 claims were classed as close and 420 were not reproduced. The test is substantial: it covers 901 parent findings and a curated environment of 39 data tables, 198 million rows and 40 documentation resources. It does not show that 77.4% of cancer research is correct, or that the remaining claims are wrong.
The team treated a numerical claim as reproduced when the recreated value was within 5% of the paper’s value; qualitative claims received a yes-or-no judgement. Among the 420 misses, 251 were attributed to missing or changed data, 47 to under-specified analysis, 28 to unavailable-software approximations and 48 to agent errors. Historical registry coverage had also changed. Those categories show why a failed reproduction is a diagnostic lead, not a correction to the medical literature.[1]
Robustness warnings need human adjudication
The authors then asked several model-based analysts to alter defensible choices—such as cohort definitions or statistical specifications—and rerun analyses. At least one automated analyst linked a potentially major change to 55 of 579 fully reproduced findings, or 9.5%; two or more analysts agreed on 14. Yet 41 of the 55 were found by only one analyst, suggesting that model choice strongly shaped which warnings appeared.
That shortlist looked much less decisive after expert review. The paper reports examination of ten flagged cases: nine did not establish a consequential change under a defensible alternative, while one required further investigation. This is the study’s most important practical limit. The system may help concentrate scarce expert attention, but the sampled warnings had a high false-alarm burden and the experts did not review all 55 flags or a random sample of unflagged findings.[1]
The distinction that matters
What the audit actually measured
The stages answer different questions and should not be collapsed into one accuracy score.
20 papers
Split 901 findings into 2,190 claims
A claim-level benchmark
1,694
Recreated within the grading rule
77.4% reproduced
55 of 579
Flagged after alternative analyses
Candidates for expert review
10 checked
Experts examined sampled flags
Nine not consequential; one unresolved
Useful infrastructure is not an autonomous referee
For researchers and journal editors, the immediate value is an executable audit trail: code, outputs and alternative analyses attached to individual claims. That could make it easier to locate missing data, ambiguous methods and results worth checking. It should not become an automated pass-or-fail score. The benchmark uses one US registry family, the reproduction agent and grader share model infrastructure, and the work has not been peer reviewed or independently replicated.
Most authors are affiliated with Databricks AI Research or Stanford; the acknowledgements report support or resources from the Verdant Foundation, Hasso Plattner Institute, Itaú Unibanco, BMO Financial Group, Stanford HAI, the US NAIRR pilot and Microsoft Azure. No separate competing-interests statement appears in the version reviewed. Confidence would rise if an independent team reran the benchmark, experts blindly assessed all flags plus matched non-flags, and studies measured review time, cost and important errors avoided—not only claims reproduced. That evidence would show whether triage saves expert effort without hiding missed problems.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
Google DeepMind · 1 min 52 sec · Background explainer
Protein folding explained
YouTube loads only when you choose; playback starts when you press play.
A short introduction to why a protein’s shape matters. It provides background for understanding what AI structure prediction contributes to biological research.
Evidence check: This is the developer’s explanation. Predicting a structure does not by itself establish a treatment’s safety or benefit for patients.
Published 2020-11-30. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current researchEvidence check
This is an unreviewed preprint from the team that created both the benchmark and the evaluation pipeline.
The benchmark contains 20 SEER-based observational papers; transfer to trials, laboratory research, other registries or non-US data is unknown.
Experts reviewed ten of 55 automated warnings, and nine did not establish a consequential change; unflagged findings were not sampled for missed problems.
The paper reports no end-to-end comparison of expert time, financial cost, downstream corrections or patient outcomes.
Reporting basis: Source analysis. Original source date: 16 October 2026.
Continue the story
Read next
Also about ai agents
Can an AI choose your next experiment before you spend the compute?
A new preprint tests training intuition. We compare its narrow task with engineering, replication and architecture-search benchmarks to explain what research teams should measure next.
Read the 5 min reportAlso about preprint
AI research agents generate thousands of ideas—but may converge on the same directions
A preprint comparing research-agent frameworks generated 37,802 ideas from shared literature and found evidence that automated systems can narrow exploration even while increasing output.
Read the 2 min reportExplore the full source and evidence record
What this means for people
- More efficient study audits could help clinicians and patients by exposing fragile evidence sooner. The present evidence supports expert triage, not replacing epidemiologists, statisticians, peer reviewers or clinical judgement.
Global context
Cancer registries and access rules differ across countries. A workflow tuned to SEER’s coding, manuals and historical releases may not transfer to NHS, European, Asian or lower-resource registries without local data access, domain expertise and independent validation.
What the evidence does not yet show
- This is an unreviewed preprint from the team that created both the benchmark and the evaluation pipeline.
- The benchmark contains 20 SEER-based observational papers; transfer to trials, laboratory research, other registries or non-US data is unknown.
- Experts reviewed ten of 55 automated warnings, and nine did not establish a consequential change; unflagged findings were not sampled for missed problems.
- The paper reports no end-to-end comparison of expert time, financial cost, downstream corrections or patient outcomes.
What to watch next
- Peer review and independent reproduction of the 20-paper benchmark
- Blinded expert assessment of every flag plus matched unflagged findings
- Prospective comparisons of review time, cost, false alarms and important errors found
- Testing across trials, laboratory studies and non-US registries
Living evidence record
Impact record IAI-10SF889
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
19 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv / Stanford and Databricks AI Research published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 19 October 2026
Research question, topics and regions
Whether LLM agents can reproduce and stress-test findings from published cancer-registry studies
Topics: Research reproducibility · Cancer registries · AI agents · Scientific review · Preprint
Regions: United States · Global research
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.