A dental guideline improved one chatbot’s answers—but the test graded against the same guideline
Claude gained 0.39 points across 18 fictional jaw-risk scenarios after receiving the complete guideline. No patient, diagnosis or treatment outcome was tested.
The short version
- What happened
- Researchers presented ChatGPT, Claude and DeepSeek with 18 fictional endodontic scenarios involving medication-related osteonecrosis of the jaw, before and after supplying the full 2022 AAOMS position paper.
- Who it affects
- Patients taking antiresorptive medicines need individual assessment; a chatbot benchmark cannot weigh their dental findings, medicine history and jaw-risk factors safely on its own.
- What remains unproven
- Only 18 fictional, standardised scenarios
Keep in mind
Only 18 fictional, standardised scenarios
Cases, prompting material and scoring reference came from the same guideline
Only two specialist raters with consensus resolution
No external case set or real clinical records
No diagnosis, treatment, safety or patient outcomes
Results can change with model versions and settings
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Supplying the guideline narrowed one model gap
A peer-reviewed study tested three chatbots on 18 fictional endodontic scenarios involving patients taking antiresorptive medicines, where treatment choices must balance tooth preservation against medication-related osteonecrosis of the jaw. ChatGPT, Claude and DeepSeek answered the cases first without and then with the complete 2022 American Association of Oral and Maxillofacial Surgeons position paper in context.
Two endodontic specialists independently scored anonymised answers across five clinical-decision domains on a 0–5 scale, then resolved disagreements by consensus. ChatGPT's baseline mean was 4.68, compared with 4.29 for Claude. After guideline prompting, Claude rose by 0.39 points to 4.68 and the earlier difference was no longer statistically significant. DeepSeek remained lower than the other two systems. Some domain-level sensitivity tests did not confirm the phase effect for consultation and medication management.[1]
Agreement with a supplied document is not clinical validation
The apparent gain is circular in an important way: researchers built the fictional scenarios and reference standard from the same guideline later supplied to the models. Better agreement therefore shows retrieval and application of that document under controlled conditions, not independent diagnostic reasoning. The same 18 cases were used in both phases, and there were no real patients, records, imaging findings, uncertain histories or consequences for treatment.
For dentists and patients, this is evidence that current guidance can make a chatbot's answer more guideline-concordant—not permission to delegate decisions. The safe use would remain a clinician-owned check with current local guidance, drug history, dental findings and specialist consultation where required. Confidence would require externally written cases, locked model versions, error analysis, prospective evaluation and evidence that support improves decisions without delaying care. The authors reported university support for the publication charge and no competing interests.[1]
The distinction that matters
What changed when the full guideline was supplied
The benchmark measured agreement with a guideline-derived reference, not patient outcomes.
Baseline
Models answered 18 fictional cases
Claude mean 4.29; ChatGPT 4.68
Guideline supplied
Same cases plus complete AAOMS paper
Claude rose to 4.68
Clinical care
No real patients or decisions
Safety and benefit untested
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical decision support: compare the finding and its evidence limits.
Why did cancer-care AI falter after diagnosis?
A two-centre Chinese preprint tested 30 language models across six stages of nasopharyngeal-cancer care. Diagnosis was the strongest phase; staging errors and follow-up omissions exposed a different, less reassuring pattern.
Read the next reportExplore the full source and evidence record
What this means for people
- Patients taking antiresorptive medicines need individual assessment; a chatbot benchmark cannot weigh their dental findings, medicine history and jaw-risk factors safely on its own.
Global context
The evaluation was conducted by a Turkish endodontics team using a US professional position paper. Drug pathways, referral practice and clinical guidance differ across health systems and change over time.
What the evidence does not yet show
- Only 18 fictional, standardised scenarios
- Cases, prompting material and scoring reference came from the same guideline
- Only two specialist raters with consensus resolution
- No external case set or real clinical records
- No diagnosis, treatment, safety or patient outcomes
- Results can change with model versions and settings
What to watch next
- Externally authored cases with independent reference standards
- Locked model versions and reproducible settings
- Prospective clinician-supervised evaluation
- Patient safety, delays and treatment outcomes
Living evidence record
Impact record IAI-0TEOWMS
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
26 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Scientific Reports published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 26 October 2026
Research question, topics and regions
Whether providing a full clinical position paper changes chatbot agreement with a guideline-derived reference on fictional MRONJ endodontic cases
Topics: Clinical decision support · Dentistry · Prompting · Patient safety
Regions: Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.