When reliable clinical AI is wrong, who catches the error?
In a 14-reader experiment, radiology residents accepted 52% of the reliable model's incorrect suggestions. Three neonatologists overrode 91.7% of deliberately flawed advice, but that specialist result is too small to generalise.
The short version
- What happened
- Fourteen clinicians interpreted 125 neonatal radiographs in a crossover experiment, producing 1,750 decisions across unaided, reliable-AI and error-injected-AI conditions.
- Who it affects
- In neonatal emergencies, a confidently missed finding or false alarm can delay surgery or trigger unnecessary escalation. Trainees need protected opportunities to practise independent verification before AI enters routine workflow.
- What remains unproven
- Retrospective web-based experiment with 14 readers and only three neonatologists.
Wrong advice exposed a trainee vulnerability
Researchers recruited six pediatric radiologists, three neonatologists and five radiology residents. Each interpreted cases in counterbalanced sessions separated by six weeks. The external set contained 125 radiographs from 11 hospitals, including 40 positive cases; 41 cases were unaided, 40 used the reliable model and 44 used a deliberately error-injected model. Readers were not told that two reliability levels existed.[1]
Who it affects
In neonatal emergencies, a confidently missed finding or false alarm can delay surgery or trigger unnecessary escalation. Trainees need protected opportunities to practise independent verification before AI enters routine workflow.[1]
Keep in mind
Retrospective web-based experiment with 14 readers and only three neonatologists.
The 125-case external set targeted one neonatal radiographic emergency; results may not transfer to other tasks or clinical environments.
Deliberate model errors and voluntary saliency-map use do not reproduce the full distribution of failures or attention in live care.
Source analysis · 1 linked source. Original source date: 20 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Wrong advice exposed a trainee vulnerability
Researchers recruited six pediatric radiologists, three neonatologists and five radiology residents. Each interpreted cases in counterbalanced sessions separated by six weeks. The external set contained 125 radiographs from 11 hospitals, including 40 positive cases; 41 cases were unaided, 40 used the reliable model and 44 used a deliberately error-injected model. Readers were not told that two reliability levels existed.
The reliable model's external AUC was 0.948, although it was 0.861 on the 40 cases assigned to that study arm. It did not significantly improve any group's accuracy over unaided reading. When it was wrong, residents accepted 52% of its suggestions, compared with 20% for pediatric radiologists. That difference remained significant after the authors' multiple-comparison correction.[1]
Expert vigilance is not a safety guarantee
Against deliberately flawed advice, neonatologists overrode 66 of 72 errors, or 91.7% (95% confidence interval 83.0%–96.0%). Pediatric radiologists overrode 85.4% and residents 81.7%. Disagreeing with AI also took longer across every group. The study therefore shows why hospitals should test the human-plus-system workflow under plausible failure, not only report a model's standalone AUC.
But the experiment was retrospective and web-based, used one narrow emergency finding and included only three neonatologists. Its error-injected model was engineered to fail in specific ways, while saliency-map use was voluntary and confounded by case difficulty. No patient outcome or live clinical workflow was tested. Larger prospective studies should vary failure type and frequency, measure overrides and delays, and test whether training reduces automation bias without creating blanket distrust.[1]
The distinction that matters
The safety test the study adds
Standalone model accuracy leaves the most important failure question unanswered.
Model test
Measure AI accuracy alone
Finds technical errors
Team test
Give wrong advice to clinicians
Measures acceptance and override
Clinical test
Run prospectively in care
Still needed for patient benefit and harm
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Evidence check
Retrospective web-based experiment with 14 readers and only three neonatologists.
The 125-case external set targeted one neonatal radiographic emergency; results may not transfer to other tasks or clinical environments.
Deliberate model errors and voluntary saliency-map use do not reproduce the full distribution of failures or attention in live care.
Reporting basis: Source analysis. Original source date: 20 October 2026.
Continue the story
Read next
Continue exploring automation bias: compare the finding and its evidence limits.
Can paediatricians resist wrong AI advice?
A Greek study of 98 paediatricians found that doctors usually rejected deliberately incorrect advice labelled as AI-generated, but residents and private-practice clinicians were more likely than hospital attendings to change towards a wrong answer. The experiment used fabricated advice and fixed-order vignettes, not a live clinical system.
Read the next reportExplore the full source and evidence record
What this means for people
- In neonatal emergencies, a confidently missed finding or false alarm can delay surgery or trigger unnecessary escalation. Trainees need protected opportunities to practise independent verification before AI enters routine workflow.
- Hospitals should measure which staff catch errors, how long verification takes and whether explanations help, rather than assuming a human reviewer automatically provides a safety net.
Global context
Regulators and health systems increasingly require clinical AI evaluation, but conventional validation often stops at model performance. This study offers a concrete human-factors stress test while showing that expertise, case difficulty and verification time belong in the safety evidence.
What the evidence does not yet show
- Retrospective web-based experiment with 14 readers and only three neonatologists.
- The 125-case external set targeted one neonatal radiographic emergency; results may not transfer to other tasks or clinical environments.
- Deliberate model errors and voluntary saliency-map use do not reproduce the full distribution of failures or attention in live care.
What to watch next
- Prospective, multi-centre replication with larger specialist and trainee groups
- Randomised tests of explanations, independent first reads and adversarial training
- Patient-safety, delay, workload and override outcomes during real clinical deployment
Living evidence record
Impact record IAI-09WFJ6Y
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
20 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what npj Digital Medicine published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 20 October 2026
Research question, topics and regions
How clinicians with different expertise respond when AI advice is reliable or deliberately wrong
Topics: Clinical AI safety · Automation bias · Human factors · Neonatal care · Radiology
Regions: South Korea · East Asia
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.