A biomedical AI cited real papers—but did they support its claims?
In a 100-claim test, closed-book models produced 327 valid PubMed identifiers but an automated cross-family judge found only three relevant. Letting agents write their own searches improved retrieval, though the small accuracy gains were not significant.
The short version
- What happened
- The author sampled 100 of 300 labelled SciFact development claims and compared closed-book, one-shot PubMed retrieval, an agent-written multi-query loop and an oracle with gold abstracts across two model backbones.
- Who it affects
- Researchers and clinicians should open the cited paper and check the exact claim, not treat a resolving DOI or PMID as verification. Tool builders should bind citations to retrieved text and expose abstention.
- What remains unproven
- The unreviewed workshop paper evaluates 100 claims from one benchmark and two backbones.
Keep in mind
The unreviewed workshop paper evaluates 100 claims from one benchmark and two backbones.
Accuracy differences between realistic harnesses were not statistically significant at this sample size.
A cross-family LLM judge, not a complete human review, supplied the main citation-relevance audit.
The author reports no specific funding and no competing interests.
Source analysis · 1 linked source. Original source date: 21 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
The harness changed what evidence the model saw
The study used a fixed random sample of 100 claims from the 300-claim SciFact development split because the benchmark's test labels are hidden. Each claim was labelled supported, contradicted or not enough information. Two efficient model backbones ran four conditions at temperature zero: closed book, one PubMed query built directly from the claim, a ReAct-style agent that wrote and refined queries, and an oracle supplied with the gold abstracts.
The single-shot query found the gold paper for only 6% of claims and mostly converted overconfident errors into abstentions. Agent-written queries tripled gold recall and produced the best realistic verdicts, at 62% accuracy against 49% to 59% elsewhere. Those differences were not statistically significant on 100 claims. When given the gold abstracts, the same backbones reached 79% to 86%, leaving a 17-to-24-point retrieval gap.[1]
A valid identifier did not make the citation valid
Closed-book models cited 335 PubMed identifiers; 327 resolved to real records. A cross-family model judge rated only three of those 327 as relevant to the claim. Search agents produced fewer citations but about 67% were judged relevant, and they almost never cited outside their retrieved set. The result warns that identifier-existence checks catch fabrication but not papers attached to the wrong proposition.
The denominator is small, all claims came from one benchmark and abstracts were truncated to 1,500 characters in retrieval conditions. Citation relevance relied on another language model rather than comprehensive human adjudication, so the striking three-of-327 figure needs independent review. The released cache, prompts and outputs support reproduction. Larger human-audited studies across databases, specialties and full text would change the assessment.[1]
The distinction that matters
Four ways to answer the same claim
The backbone and output format stay fixed while the evidence path changes.
Closed book
Answer from model memory
Can produce real but irrelevant identifiers
Search agent
Write and revise PubMed queries
Finds the gold paper more often
Oracle
Supply the benchmark's gold abstracts
Estimates remaining reasoning error
Watch the background, then check the evidence
Illustrative photo · not a video frame
Google DeepMind · 1 min 52 sec · Background explainer
Protein folding explained
YouTube loads only when you choose; playback starts when you press play.
A short introduction to why a protein’s shape matters. It provides background for understanding what AI structure prediction contributes to biological research.
Evidence check: This is the developer’s explanation. Predicting a structure does not by itself establish a treatment’s safety or benefit for patients.
Published 2020-11-30. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current researchContinue the story
Read next
Continue exploring research agents: compare the finding and its evidence limits.
AI research agents generate thousands of ideas—but may converge on the same directions
A preprint comparing research-agent frameworks generated 37,802 ideas from shared literature and found evidence that automated systems can narrow exploration even while increasing output.
Read the next reportExplore the full source and evidence record
What this means for people
- Researchers and clinicians should open the cited paper and check the exact claim, not treat a resolving DOI or PMID as verification. Tool builders should bind citations to retrieved text and expose abstention.
Global context
PubMed is global but English-language coverage, indexing and access vary. Research agents also use Crossref, regional databases and full text, so the harness result should be retested beyond this single public API.
What the evidence does not yet show
- The unreviewed workshop paper evaluates 100 claims from one benchmark and two backbones.
- Accuracy differences between realistic harnesses were not statistically significant at this sample size.
- A cross-family LLM judge, not a complete human review, supplied the main citation-relevance audit.
- The author reports no specific funding and no competing interests.
What to watch next
- Human-adjudicated replication of citation relevance
- Larger samples with full-text retrieval
- Comparisons across biomedical databases and languages
- Prospective measurement of missed evidence and researcher correction time
Living evidence record
Impact record IAI-1RHB84F
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
22 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 22 October 2026
Research question, topics and regions
Which retrieval harnesses help language models verify biomedical claims and whether valid PubMed identifiers are actually relevant
Topics: Biomedical research · PubMed · Citation verification · Research agents · Retrieval
Regions: India · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.