Microsoft's diagnostic agent beat doctors on a benchmark—what has it proved?
A peer-reviewed Science paper reports higher accuracy and lower estimated testing costs on 304 difficult case reports and 325 emergency-department records. It is still a retrospective simulation, not evidence that the system improves care for real patients.
The short version
- What happened
- The benchmark turns 304 New England Journal of Medicine case reports and 325 de-identified emergency-department admissions into step-by-step diagnostic encounters.
- Who it affects
- A reliable sequential assistant could help clinicians consider alternatives and avoid low-value tests. A system that is wrong, overconfident or poorly integrated could instead delay treatment, order needless procedures or shift extra checking work onto already stretched teams.
- What remains unproven
- This was a retrospective benchmark study, not a prospective trial or evidence of improved patient outcomes.
Keep in mind
This was a retrospective benchmark study, not a prospective trial or evidence of improved patient outcomes.
The NEJM set deliberately contains diagnostically difficult published cases and does not represent the frequency of conditions in routine practice.
The gatekeeper can synthesise results absent from the original case, while another language model judges the answer; both components can introduce error or favour model-native reasoning.
The physician and AI conditions differ from team-based clinical care, where specialists, records, repeated observation and local workflows influence decisions.
The research is Microsoft-led and supports a Microsoft system. The accessible Science page was blocked, so final funding and conflict declarations could not be independently read in this review.
Multi-source analysis · 4 linked sources. Primary analysed source date: 22 October 2026. Earliest linked supporting source: 27 June 2025.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The strongest result is performance inside a harder benchmark
A Microsoft-led study published in Science on 22 October reports that its MAI Diagnostic Orchestrator, or MAI-DxO, reached 83% diagnostic accuracy on complex New England Journal of Medicine cases. Generalist physicians scored 36% when allowed textbooks and internet references and 22% without them. On a second set drawn from de-identified US emergency-department records, Microsoft reports 80% for the system and 47% for physicians. Those are large benchmark differences, but they do not show that an AI safely outperforms clinical teams at the bedside.[1][2][3]
The system asks questions and orders tests before answering
SDBench converts 304 difficult case reports and 325 emergency-department admissions into interactive encounters. The diagnostic agent starts with a short summary, then requests history, examination findings or tests before committing to an answer. MAI-DxO makes one model play five roles: maintaining a differential diagnosis, choosing tests, challenging early assumptions, checking cost and enforcing consistency. The earlier public manuscript compared 21 US and UK generalist physicians, with a median 12 years of experience; the peer-reviewed release adds the reference-enabled and emergency-care comparisons.[2][3][4]
The distinction that matters
How the benchmark differs from an exam—and from real care
The evaluation adds sequential decisions and estimated costs, while still operating on retrospective records through model-mediated components.
Start
Show a short case summary
Diagnosis is initially hidden
Investigate
Ask questions or order tests
Gatekeeper reveals or synthesises findings
Commit
Submit one diagnosis
Model judge scores accuracy; costs are estimated
Estimated cost improved, but the simulation supplies part of the world
Microsoft says a cost-focused configuration retained 81% accuracy on the difficult cases at less than half the estimated testing cost of comparison systems. Costs came from US procedural-price mappings, not hospital bills. More importantly, an AI gatekeeper reveals information from the source case and generates realistic, case-consistent results when a requested test was not recorded. Another model judges whether the final diagnosis matches the reference answer. Physician review found strong agreement with that judge in the earlier manuscript, but model-generated findings and model scoring still make this a simulation rather than a prospective clinical trial.[2][3][4]
The next study must test patients, workflow and harm
No patient was diagnosed or treated by MAI-DxO in this evaluation. Microsoft says it plans a prospective study with Beth Israel Deaconess Medical Center in which patients will interact with the system and independent physicians will assess performance. That work needs prespecified safety outcomes, subgroup analyses, missed and delayed diagnoses, unnecessary testing, clinician workload and the consequences of disagreements—not accuracy alone. Until those results exist, the paper supports a better way to test sequential reasoning and a promising research system, not a clinical product or a replacement for doctors.[1][2][3]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring emergency medicine: compare the finding and its evidence limits.
Can AI make life-or-death clotting results arrive sooner?
A peer-reviewed review found 27 studies applying AI to viscoelastic haemostatic assays. Some models may infer useful results within minutes, but small cohorts, internal validation and scarce calibration data mean faster predictions are not yet proven to improve patient outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- A reliable sequential assistant could help clinicians consider alternatives and avoid low-value tests. A system that is wrong, overconfident or poorly integrated could instead delay treatment, order needless procedures or shift extra checking work onto already stretched teams.
Global context
The cost model uses US prices and the emergency records come from one US clinical database. Diagnostic pathways, disease prevalence, test access and clinician roles vary substantially across countries, so neither accuracy nor cost results can be assumed to transfer.
What the evidence does not yet show
- This was a retrospective benchmark study, not a prospective trial or evidence of improved patient outcomes.
- The NEJM set deliberately contains diagnostically difficult published cases and does not represent the frequency of conditions in routine practice.
- The gatekeeper can synthesise results absent from the original case, while another language model judges the answer; both components can introduce error or favour model-native reasoning.
- The physician and AI conditions differ from team-based clinical care, where specialists, records, repeated observation and local workflows influence decisions.
- The research is Microsoft-led and supports a Microsoft system. The accessible Science page was blocked, so final funding and conflict declarations could not be independently read in this review.
What to watch next
- The planned prospective hospital study and its registered protocol
- Performance by age, sex, ethnicity, language, condition and clinical setting
- Missed diagnoses, unnecessary tests, delays and other patient-safety outcomes
- Independent replication using human clinical teams and non-US records
- Regulatory status, product design and who remains accountable for decisions
Living evidence record
Impact record IAI-0CGVIHW
Evidence stage
Studied
Confidence
Supported
Reporting basis
Multi-source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
22 October 2026
Source trail
4 direct sources across 3 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Evidence trail
Sources used for this report
Links checked 22 October 2026
Research question, topics and regions
Whether a multi-agent diagnostic orchestrator can improve sequential diagnostic accuracy and estimated test costs in retrospective case simulations
Topics: Clinical AI · Diagnosis · AI agents · Emergency medicine · Benchmark evidence
Regions: United States · United Kingdom · North America · Europe
This report is labelled multi-source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.