Qwen3 caught 320 of 335 radiology-report errors—what did it never see?
The model performed strongly on report text from two hospitals and an error-enriched comparison with three radiologists. It never saw the scans, was not used prospectively and did not measure patient outcomes.
The short version
- What happened
- Researchers retrospectively analysed 95,582 reports at one hospital and externally tested the selected model on 44,839 reports at a second hospital.
- Who it affects
- Radiologists could gain a rapid second check for report wording, while patients could benefit if genuine errors are caught before a report guides care. False alerts and missed image findings still require clinician ownership.
- What remains unproven
- Retrospective study
Keep in mind
Retrospective study
Two hospitals in one regional system
Text only—no images
300-report error-enriched comparison
Three radiologists
Best model selected from five
No prospective workflow or patient outcomes
Source analysis · 1 linked source. Primary analysed source date: 28 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
Two hospitals supplied 140,421 real reports
A general-purpose language model identified internal mistakes in radiology report text with high accuracy across two Chinese hospitals, according to a peer-reviewed retrospective study. The first centre contributed 95,582 reports: 10,036 from x-rays, 67,215 from CT and 18,331 from MRI. Two senior radiologists created a consensus reference using five error categories. They confirmed at least one error in 4,431 reports, or 4.6%. The researchers first compared five models on a random 10,000-report subset, then selected Qwen3 for the full analysis.
At the second hospital, Qwen3 was tested on an independent 44,839-report cohort collected under the same annotation protocol. Its balanced accuracy was 0.94, with a 95% confidence interval from 0.92 to 0.95. Results were strongest for explicit factual problems such as laterality or unit mistakes and weaker for interpretive inconsistencies. That external test is more informative than a single-centre benchmark, although both hospitals operated within one regional health system and the same two experts helped define both reference sets.[1]
The human comparison was deliberately difficult—and narrow
The headline human comparison used 300 reports sampled only from cases already confirmed to contain errors: 100 each from x-ray, CT and MRI, with 335 error instances in total. Qwen3 assigned the exact error category correctly for 320 instances, or 0.96. Three radiologists with ten, six and three years of experience classified 223, 178 and 152 instances correctly, corresponding to 0.67, 0.53 and 0.45. Mean review time was 5.8 seconds for the model versus 55.8 to 67.8 seconds for the clinicians.
This does not show that a model diagnoses scans better than radiologists. Everyone reviewed report text without images, clinical history or access to the original interpretation. Because every challenge case contained an error, the comparison also cannot show how often either side would raise false alarms during ordinary work, where most reports are correct. The study's broader first-centre analysis reported a positive predictive value of 0.90, meaning false alerts still matter when a tool is placed in front of busy staff.[1]
The distinction that matters
What the evaluation did—and did not—test
The strongest numbers concern internal consistency in report text.
Full cohorts
140,421 reports across two hospitals
Cross-centre text validation
Human comparison
300 known-error reports and 335 error instances
Qwen3 classified 320
Not tested
Images, prospective care and patient outcomes
Clinical benefit remains unproven
A second reader for words, not a replacement for image review
For patients and radiology teams, the plausible benefit is an additional text check before a report is finalised: flagging a left-right mismatch, inconsistent measurement or conclusion that conflicts with the description. The model cannot see an overlooked tumour, verify whether a fracture is present or decide whether the report matches the scan. Every alert still needs a clinician who can examine the images and the patient's history. The study also selected the best of five models before the main evaluation, which can make that model look stronger than an entirely prespecified deployment test.
Confidence should change with a prospective trial in routine reporting, a locked model and prompt, local error prevalence, workload and false-alert measurement, and patient-safety outcomes. Testing across more hospitals, languages and reporting styles would show whether the result travels. Until then, the evidence supports a promising text-quality-control assistant under human supervision. It does not establish safer care, faster diagnosis or permission to remove radiologist review.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
Wrong AI suggestions shifted five radiologists' bone-age estimates in a small Taiwan study
Six radiologists read 200 radiographs with accurate or deliberately shuffled AI advice. Five showed greater error with sham suggestions, but this controlled study cannot measure patient harm or a population-wide effect.
Read the next reportExplore the full source and evidence record
What this means for people
- Radiologists could gain a rapid second check for report wording, while patients could benefit if genuine errors are caught before a report guides care. False alerts and missed image findings still require clinician ownership.
Global context
Hospitals worldwide face radiology workload pressure, but report styles, languages, systems and error prevalence differ. A local prospective evaluation is needed before transferring the Chinese two-centre result into another health service.
What the evidence does not yet show
- Retrospective study
- Two hospitals in one regional system
- Text only—no images
- 300-report error-enriched comparison
- Three radiologists
- Best model selected from five
- No prospective workflow or patient outcomes
What to watch next
- Prospective workflow trial
- False alerts at local prevalence
- Locked model and prompt
- More hospitals and languages
- Image-aware evaluation
- Radiologist workload
- Patient-safety outcomes
Living evidence record
Impact record IAI-0YLP2W6
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
30 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Journal of Medical Internet Research published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 30 October 2026
Research question, topics and regions
Text-only large-language-model detection of internal errors in radiology reports across two hospitals
Topics: Clinical AI · Radiology · Quality assurance · Patient safety · External validation
Regions: China · Asia · United Kingdom · United States
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.