Medical AI can notice a new guideline—and still give the old answer
A 1,709-revision preprint found that models often recognised changed clinical guidance without carrying every consequence into their recommendation. It is a benchmark result, not evidence of patient harm.
The short version
- What happened
- Researchers built ChronoEvo from 1,709 recommendation-level changes across 275 guideline lineages and tested 14 models with old, updated or no guideline evidence.
- Who it affects
- A clinician may see that an assistant has cited the newest guidance and reasonably assume the recommendation reflects it. This study shows why the final patient-specific advice still needs a line-by-line clinical check.
- What remains unproven
- The work is a preprint and has not been peer reviewed.
Keep in mind
The work is a preprint and has not been peer reviewed.
Most large-scale labels came from model evaluators; two clinicians adjudicated a 300-response subset rather than all 1,709 revisions.
The benchmark tests generated recommendations against guideline changes, not real clinical decisions, workflow use or patient outcomes.
Guideline sources and revision categories may not represent every specialty, country or form of evidence update.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Seeing the update was not the same as using it
Clinical guidance changes in ways that matter to an individual patient: a treatment can be withdrawn, a threshold moved or eligibility narrowed. University of Minnesota researchers paired old and revised recommendations across 1,709 changes, then asked models to respond with the revised evidence available. The primary model recognised 90.8% of changes but fully incorporated only 58.5%; two clinicians reviewing 300 responses found a 40.3-point recognition-to-incorporation gap.[1]
A stronger reminder fixed only part of the problem
An instruction to prioritise new evidence raised clinical correctness from 80.2% to 82.6% and reduced retention of outdated content from 5.9% to 4.3%. Full incorporation barely changed, from 58.5% to 59.3%, and that difference was not statistically detectable. Across all 14 tested models, recognition stayed ahead of full incorporation; even the strongest model retained a 20.1-point gap.[1]
The distinction that matters
Where the update can be lost
The benchmark separates three steps that are often collapsed into one accuracy score.
Recognise
Identify that guidance changed
90.8% in the primary test
Recommend
Produce a clinically correct answer
80.2% in the primary test
Incorporate
Carry every material consequence into the answer
58.5% in the primary test
The safe lesson is about verification, not diagnosis
Clinicians and health services should not treat retrieval of a current guideline as proof that an AI recommendation follows it faithfully. A useful check must compare the patient-specific output with every material change in the source. Patients should not use this preprint to choose or stop treatment: the benchmark generated recommendations from guideline text and did not test care pathways, outcomes or a regulated clinical product.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
Can a cheaper AI safely judge clinical AI answers?
A proprietary decision model was fast, stable and often competitive across four evaluation sets, but it missed most physician-labelled safety failures in one hospital sample. Hybrid review helped; patient safety remains unproven.
Read the next reportExplore the full source and evidence record
What this means for people
- A clinician may see that an assistant has cited the newest guidance and reasonably assume the recommendation reflects it. This study shows why the final patient-specific advice still needs a line-by-line clinical check.
Global context
Guidelines, approved indications and care pathways vary by country. The reliability question travels globally, but each deployment must be evaluated against the exact local guidance and patient population it will serve.
What the evidence does not yet show
- The work is a preprint and has not been peer reviewed.
- Most large-scale labels came from model evaluators; two clinicians adjudicated a 300-response subset rather than all 1,709 revisions.
- The benchmark tests generated recommendations against guideline changes, not real clinical decisions, workflow use or patient outcomes.
- Guideline sources and revision categories may not represent every specialty, country or form of evidence update.
What to watch next
- Peer review and external replication
- Prospective tests inside clinical workflows
- Independent evaluation on local guidelines and patient populations
- Whether structured checking of every changed qualifier closes the gap
Living evidence record
Impact record IAI-1GKBX1J
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
23 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 23 October 2026
Research question, topics and regions
Whether language models fully propagate explicitly supplied revisions to clinical guidance into downstream recommendations
Topics: Clinical AI · Medical guidelines · Knowledge updating · Patient safety · Evaluation
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.