Open medical models passed Spain's residency exam—and still gave unsound reasons
Four open models answered 1,552 Spanish MIR questions. Three larger systems reached strong estimated ranks, yet 13–38% of their correct answers included at least one clinically unsound rejection of a distractor.
The short version
- What happened
- Researchers scored four open models of 8–27 billion parameters on 1,552 official MIR questions from eight examination years.
- Who it affects
- A trainee or clinician can receive the correct multiple-choice answer for the wrong medical reason. That is especially risky when the explanation—not the option letter—is what guides learning or care.
- What remains unproven
- The study is a preprint and has not been peer reviewed.
Keep in mind
The study is a preprint and has not been peer reviewed.
Only four open models, one Spanish prompt and direct answering were tested; larger or reasoning-focused systems may behave differently.
An LLM, not a physician, graded the model rationales; the authors identified judge errors on negatively worded options.
Rank mappings are estimates and the image questions may often be answerable from text alone.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
A high rank concealed several different weaknesses
The team reconstructed 2,595 MIR questions and used 1,552 scorable questions from 2018–2025 for the main ranking and calibration tests. Qwen3-8B, Qwen3-14B, Gemma-3-27B and MedGemma-27B answered deterministically under the exam's reward and penalty rules. The larger models scored well enough for many specialty choices, but the rank estimates depend partly on interpolated 2023 and 2024 allocation data.[1]
Correct answers were not always clinically well reasoned
For 123 questions with physician commentary, the researchers asked an external Llama model to compare each model's reasoning with the clinical explanation. Among correctly answered questions, 13–38% included at least one distractor rejected for a wrong or unsupported reason. No physician validated that automated judge, so the exact rates remain provisional.[1]
The distinction that matters
Four checks behind one exam headline
The paper separates performance that a single accuracy score would merge.
Rank
Score 1,552 MIR questions
Strong estimated placement for three larger models
Reason
Compare rationales on 123 questions
13–38% of correct answers had an unsound distractor reason
Image
Show, remove or swap 171 clinical plates
No material accuracy benefit from the correct plate
Hospitals should test the task, not trust the label
Medical fine-tuning reduced average overconfidence but also reduced accuracy and did not improve how well confidence separated right from wrong answers. The two vision models changed some answers when images changed, yet real images produced no detectable accuracy gain over no image or the wrong image. Before local deployment, clinicians need independent, task-specific tests with patient cases, expert review and real deferral costs.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring medical ai: compare the finding and its evidence limits.
How much does MedGemma improve medical AI?
A peer-reviewed study reports gains over similarly sized base models across medical image questions, chest X-ray classification and simulated agent tasks. The evidence comes from benchmarks and small specialist reviews, not prospective clinical deployment or patient outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- A trainee or clinician can receive the correct multiple-choice answer for the wrong medical reason. That is especially risky when the explanation—not the option letter—is what guides learning or care.
Global context
Spain's MIR makes confidence, abstention and specialty allocation unusually concrete, but other examinations and health systems use different scoring, languages and clinical expectations. No licensing-exam benchmark substitutes for local clinical validation.
What the evidence does not yet show
- The study is a preprint and has not been peer reviewed.
- Only four open models, one Spanish prompt and direct answering were tested; larger or reasoning-focused systems may behave differently.
- An LLM, not a physician, graded the model rationales; the authors identified judge errors on negatively worded options.
- Rank mappings are estimates and the image questions may often be answerable from text alone.
What to watch next
- Physician validation of rationale grading
- Prospective clinical-task evaluation
- Image-dependent cases that cannot be solved from text
- Independent comparisons of general and medically tuned open models
Living evidence record
Impact record IAI-0XLPNYF
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
23 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 23 October 2026
Research question, topics and regions
Whether exam rank, confidence, explanation quality and image sensitivity tell the same story about open medical language models
Topics: Medical AI · Clinical reasoning · Open models · Medical education · Multimodal evaluation
Regions: Spain · Europe · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.