Did AI help radiologists read parotid ultrasound?
In a prospective reader study, three Korean radiologists improved pooled accuracy from 76.3% to 81.6% with an editable AI tool. But the system shifted errors between two tumour types, used only 107 surgical cases and has not shown better patient outcomes.
The short version
3 key points
- 1.Three board-certified radiologists made 321 reads of 107 external cases before and after seeing an AI system; pooled accuracy rose by 5.3 percentage points, from 76.3% to 81.6%.
- 2.The average gain concealed opposite class effects: sensitivity rose 18.1 points for Warthin's tumour but fell 9.3 points for pleomorphic adenoma, following the model's own prediction profile.
- 3.The study tested selected still images from two Korean tertiary centres, not live scanning or the full clinical pathway, and did not measure treatment decisions, harms or patient outcomes.
The answer: accuracy improved, but the errors moved
Yes, in this controlled reader study the AI-assisted radiologists were more accurate overall. Three board-certified radiologists each interpreted 107 ultrasound cases without assistance and then, after a washout of at least two weeks, interpreted the same cases with an editable AI interface. That produced 321 paired reads in each phase. Pooled accuracy increased from 76.3% to 81.6%. A generalized linear mixed model that accounted for both reader and case estimated an odds ratio of 1.54 for a correct reading with AI assistance, with a 95% confidence interval from 1.12 to 2.13 and p = 0.008.
The headline result is not a licence to deploy. Sensitivity for Warthin's tumour rose by 18.1 percentage points, while sensitivity for pleomorphic adenoma fell by 9.3 points. Readers were influenced by the system's class-specific bias rather than receiving a uniform improvement. Inter-reader agreement did improve, with Fleiss' kappa increasing from 0.475 to 0.611, but agreement can rise around a shared error as well as around the correct answer. The study therefore supports further testing of decision support, not autonomous diagnosis or a claim of safer care.[1]
What clinicians and patients are trying to distinguish
Pleomorphic adenoma and Warthin's tumour account for most benign parotid neoplasms, but they can look similar on ultrasound and can lead to different management. Pleomorphic adenoma is usually excised because it can extend beyond the apparent capsule and carries a risk of malignant transformation. Warthin's tumour is very rarely malignant and observation may be considered for an asymptomatic patient. Ultrasound is only one part of the work-up, which can also include clinical history, CT and fine-needle aspiration; postoperative histopathology supplied the reference diagnosis in this study.
For a patient, the useful question is not whether a model can classify a selected image. It is whether the whole diagnostic process becomes more reliable without adding harmful false reassurance or unnecessary intervention. The study isolates a narrower question: can an image-reading aid change radiologists' choices between two already identified benign masses? It excludes other benign lesions, malignant tumours and scans with no tumour. That makes the experiment interpretable, but much smaller in scope than the decisions faced in a head-and-neck clinic.[1]
How the model and external test were built
Model development used 949 preoperative ultrasound images from 316 consecutive adults who underwent parotid surgery at Korea University Ansan Hospital between 2002 and 2024. The authors split patients, not individual images, into 252 patients with 766 images for training and 64 patients with 183 images for internal testing, then used patient-level five-fold cross-validation. An independent tertiary hospital in Seoul supplied 427 images from 107 surgical patients collected from 2021 to 2024. The two hospitals had different patient catchments, staff and ultrasound equipment, although both were part of the Korea University system.
The system combined a whole-image YOLO detector with a separate classifier. When the detector's confidence met a fixed threshold of 0.80, its result was retained; below that threshold, a ConvNeXt classifier examined an expanded region around the proposed tumour. A clinician had to inspect, accept or edit the region before the final prediction. The authors explored 35 detector–classifier combinations, but prospectively pre-specified one YOLOv12m plus ConvNeXt configuration using only internal development results. That separation reduces, but does not eliminate, the risk of tailoring the tool to the external cohort.[1]
What the observer study actually compared
One image was randomly chosen for each of the 107 external patients—50 with pleomorphic adenoma and 57 with Warthin's tumour—so that no patient contributed several correlated reader-study images. Each radiologist had at least five years' experience. In the first phase, readers selected pleomorphic adenoma, Warthin's tumour or uncertain and rated confidence. In the second phase, they saw the same 107 images through the AI interface and could inspect or change its proposed region before recording a diagnosis. Readers were blinded to histopathology and to each other's answers, and clinical information such as age, sex and history was withheld.
This was prospective in the sense that the human–AI interaction and analysis configuration were planned before the assisted reading phase. It was not a prospective recruitment trial of newly presenting patients: the images came from a retrospectively assembled surgical cohort. Readers followed a correct AI answer in 76.2% of disagreements and followed an incorrect AI answer in 51.1%, a statistically significant difference. That suggests some selective resistance to bad advice, while also showing that wrong AI suggestions still changed many decisions.[1]
Performance improved less than the abstract can make it sound
The standalone detector and classifier achieved external areas under the receiver operating characteristic curve of 0.811 and 0.919. Across the exploratory grid, the hybrid exceeded both standalone models in 22 of 35 combinations and reached a highest external macro-F1 of 0.838. Those results describe discrimination and class-balanced performance on selected images; they are not probabilities that a patient has a tumour, and they do not establish calibrated risk in another hospital.
Crucially, the assisted readers did not outperform the standalone AI. The authors state that a synergistic benefit beyond the AI was not established. The practical advantage of keeping a clinician in the loop may lie in accountability, contextual judgement and the ability to reject a bad region or prediction, but this experiment did not measure those benefits across the full workflow. It also did not compare reading time, follow-up tests, surgical choices or adverse outcomes. A five-point improvement in image accuracy is encouraging, but its clinical value depends on which errors are prevented and which new ones are introduced.[1]
Limits that matter before a hospital trial
Both cohorts came from Korean tertiary referral centres and contained only surgically confirmed cases. That creates a selected disease spectrum: people observed without surgery, patients with malignant or rarer lesions, and negative examinations were absent. Only three radiologists participated, all with similar experience, and they re-read identical images after a two-week interval. Residual memory cannot be ruled out. The custom viewer also differs from live ultrasound, where the clinician moves the probe, chooses views and integrates symptoms, Doppler information and other tests.
The model's asymmetric effect is especially important. It improved recognition of Warthin's tumour while reducing sensitivity for pleomorphic adenoma, and the investigators could not identify which image characteristics or training choices caused that skew. The exhaustive threshold search across 35 combinations was exploratory. Robustness under broader domain shift remains unproven. The article is an accepted, peer-reviewed version shared before final copy-editing; the DOI is permanent, but presentation details may still change.[1]
What would change the assessment
A stronger test would recruit consecutive patients prospectively across unrelated hospitals, scanners and countries, include malignant and tumour-negative cases, and measure the complete diagnostic pathway. Readers should span different experience levels and use the system while performing live examinations. A randomized design could compare usual care with AI-assisted care and report reading time, additional imaging or biopsy, surgery, missed disease, unnecessary intervention and patient-reported outcomes. Calibration and failure rates should be published for each clinically important subgroup rather than only a pooled metric.
Hospitals considering similar tools should therefore ask for local validation and inspect class-specific errors before procurement. Clinicians need an obvious way to reject or amend the model's region and prediction, plus training that emphasizes automation bias. Patients should be told that the tool is decision support rather than a definitive test. The Korean study advances the evidence because it observes real clinicians interacting with the interface, but the next threshold is patient benefit in routine care—not another retrospective accuracy table.[1]
Funding and independence
The research was supported by the Korean Ministry of Health and Welfare through Korea Health Industry Development Institute programmes and by Korea University. The authors declared no competing interests. Institutional review boards at Korea University Ansan and Anam hospitals approved the work and waived informed consent because the image datasets were retrospective. Public or university funding does not guarantee an unbiased result, but there was no disclosed commercial sponsor with a product stake in the reported system.
The most defensible reading is narrow: in 107 selected Korean surgical cases, an editable confidence-aware AI tool improved the pooled accuracy and agreement of three experienced radiologists, while shifting sensitivity between the two target classes. That is a useful signal for clinician-in-the-loop research. It is not evidence that the system can diagnose all parotid masses, replace radiologists or improve outcomes outside the studied setting.[1]
What this means for people
- Radiologists gained accuracy overall but were also pulled toward the model's class-specific errors, so local testing and visible override controls matter.
- Patients could face different management depending on which benign tumour is suspected; image accuracy must therefore be linked to downstream decisions and harms.
- Hospital teams should not treat performance on selected surgical cases as proof that the system works on malignant, rare or tumour-negative examinations.
Global context
The work was conducted at two tertiary hospitals in South Korea. Different scanners and separate patient catchments add useful external testing, but both sites belonged to one national academic system and the evidence does not yet establish transfer across countries, community settings or unrelated health systems.
What the evidence does not yet show
- Retrospective surgical cohorts from two tertiary hospitals despite a prospectively planned reader study
- Only 107 external cases, two benign tumour classes and three similarly experienced radiologists
- Same still images read twice after a two-week washout, so residual recall is possible
- No clinical history, live probe operation, malignant lesions, negative examinations or full diagnostic workflow
- Opposite class-specific sensitivity changes and no established advantage over standalone AI
- No measurement of treatment decisions, harms, workload or patient outcomes
What to watch next
- Prospective multicentre trials in unrelated health systems with consecutive patients and broader disease spectra
- Class-specific calibration and safeguards against the fall in pleomorphic-adenoma sensitivity
- Randomized live-workflow studies measuring time, biopsies, operations, missed disease and patient outcomes
- Replication with trainees, general radiologists and head-and-neck specialists
Living evidence record
Impact record IAI-0IQCPFE
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
15 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what BMC Medicine published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 15 October 2026
Research question, topics and regions
Whether an editable confidence-aware AI system changes radiologists' image-based differentiation of two benign parotid tumours
Topics: Clinical AI · Ultrasound · Radiology · Human oversight · Diagnostic accuracy · Prospective evaluation
Regions: Asia · South Korea
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Healthcare, culture & human impact · Background explainer
Published 2026-07-01 · 44 min 28 sec
Is this the AI future we want? | The Royal Society
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
Can clinical AI recognise when the patient record does not support an answer?
A new clinical-agent benchmark raises a practical question for health systems: can an assistant explain what the record cannot establish? Our analysis examines evidence, local testing and the burden on staff.
6 min · 2 sources
Health & Life Sciences
Did AI heatmaps improve glioblastoma survival judgements?
Four clinicians became more accurate when shown an MRI model and its heatmaps—but the heatmaps did not significantly outperform the AI score alone, and an audit found that the model relied heavily on a small set of bright pixels.
9 min · 1 source
Health & Life Sciences
Can five preoperative signals predict kidney injury after hip-fracture surgery?
A single-hospital Chinese study of 616 older patients found moderate internal discrimination from a five-variable model. Only 54 developed acute kidney injury, the result changed with the creatinine definition, and no outside hospital has tested it.
8 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.