The same cancer case changed medical AI staging accuracy by over 50% when its format changed
Across 3,035 physician-validated perturbations, three medical language models gave markedly different clinical answers when equivalent cases were reformatted. The peer-reviewed study supports model-specific safety testing, not a claim about patient outcomes.
The short version
- What happened
- Researchers tested GPT-5, MedGemma-27B-Text-IT and OpenBioLLM-Llama3-70B with 3,035 systematic changes to physician-validated oncology and hospital cases.
- Who it affects
- Clinicians could receive different recommendations from equivalent records, making local format testing and visible uncertainty important before use in care.
- What remains unproven
- No prospective clinical workflow, clinician comparison or patient outcome was measured.
Keep in mind
No prospective clinical workflow, clinician comparison or patient outcome was measured.
Only three models and selected oncology and hospital-case formats were tested; results may not transfer to other specialties, languages or model versions.
The mechanistic analysis covered the two open models; it should not be read as an explanation of GPT-5's internal representations.
The work was funded by the US National Institutes of Health and the Pelotonia Institute for Immuno-Oncology. The authors declared no competing interests.
Source analysis · 1 linked source. Primary analysed source date: 31 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Equivalent cases did not produce equivalent reasoning
A medical language model's answer can depend heavily on how a case is written, even when the clinical facts stay the same. Ohio State researchers evaluated GPT-5, MedGemma-27B-Text-IT and OpenBioLLM-Llama3-70B using 3,035 physician-validated perturbations of oncology SOAP notes, short vignettes and MIMIC hospital cases. They changed presentation rather than the underlying diagnosis question.
The paper reports staging-accuracy swings of more than 50 percentage points from prompt format alone. Models also sometimes gave a definite stage when the supplied case was clinically insufficient. For a clinician, that matters because copied notes, terse summaries and structured records can express the same evidence differently; a benchmark average can hide that sensitivity.[1]
Safety fixes may need to match the model
To examine why, the team trained sparse autoencoders on one billion tokens from 50,000 MIMIC-IV notes and analysed the two open models' internal representations. MedGemma showed a hierarchy in which stronger features tracked lexical identity and weaker ones carried contextual meaning, while OpenBioLLM distributed information more evenly. Retrieval and reasoning interventions therefore behaved differently across architectures.
This does not establish how often a deployed system would harm a patient. The cases were controlled tests, the models were a small selection, and the analysis did not compare clinicians using the tools with clinicians working alone. The practical result is a validation requirement: hospitals should test their exact model, prompt, record format and failure-handling process before relying on a general medical benchmark.[1]
The distinction that matters
Why formatting can become a clinical safety problem
The study kept the clinical substance stable while varying how the case reached the model.
Case format
Rewrite the same evidence as notes or vignettes
Staging accuracy moved by more than 50 percentage points
Missing evidence
Present a clinically insufficient scenario
Some outputs still supplied a definitive stage
Model structure
Inspect two open models' internal features
The same intervention did not transfer safely across architectures
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
Can chatbots match urologists on cancer decisions?
In 100 synthetic cancer cases, three leading chatbots were less often fully successful than two early-career urologists and made safety-critical errors far more often. Repeated answers also changed substantially.
Read the next reportExplore the full source and evidence record
What this means for people
- Clinicians could receive different recommendations from equivalent records, making local format testing and visible uncertainty important before use in care.
- Patients should not have safety inferred from a single benchmark score; the deployed workflow, model version and documentation style all need evaluation.
Global context
Hospitals in many countries are piloting language models against records written in different templates and languages. This US study gives a repeatable reason to test local documentation formats rather than importing a headline benchmark score.
What the evidence does not yet show
- No prospective clinical workflow, clinician comparison or patient outcome was measured.
- Only three models and selected oncology and hospital-case formats were tested; results may not transfer to other specialties, languages or model versions.
- The mechanistic analysis covered the two open models; it should not be read as an explanation of GPT-5's internal representations.
- The work was funded by the US National Institutes of Health and the Pelotonia Institute for Immuno-Oncology. The authors declared no competing interests.
What to watch next
- Prospective studies comparing clinicians with and without model support
- Replication across specialties, languages, record systems and model versions
- Calibration that allows a model to abstain when evidence is insufficient
- Architecture-specific interventions tested for unintended performance losses
- Public reporting of format robustness in procurement and regulatory evaluations
Living evidence record
Impact record IAI-01R77IY
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
31 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what npj Digital Medicine published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 31 October 2026
Research question, topics and regions
Whether clinically equivalent case presentations produce stable reasoning across medical language models
Topics: Clinical AI · Medical language models · Oncology · Model evaluation · Patient safety
Regions: North America · United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.