AI health assistants often endorsed self-diagnoses that needed caution—what did the test measure?
In 100 matched pairs, 27 models were asked when to challenge or accept a plausible self-diagnosis. Even the best current model cautioned appropriately on only 77% of time-critical condition cases, but the benchmark used written scenarios and one physician reviewer.
The short version
- What happened
- SelfDx contains 200 written messages in 100 matched pairs: one version leaves a serious alternative possible and the other adds evidence intended to rule it out.
- Who it affects
- Someone asking about a worrying symptom could delay care if an assistant accepts a benign explanation too quickly; indiscriminate warnings can also increase anxiety and unnecessary demand.
- What remains unproven
- The work is an unreviewed preprint using constructed English-language prompts, one physician reviewer, an automated judge and no patients, clinical decisions or outcomes.
Matched questions tested both missed warnings and needless alarm
A new preprint finds that general-purpose AI assistants frequently accepted a user's benign explanation even when the written symptoms left a time-critical condition possible. The researchers built SelfDx from 100 matched pairs, split evenly between condition and medication questions. In each pair, a caution case leaves a serious alternative open; its partner keeps the same self-diagnosis but adds evidence intended to rule that alternative out. This paired design matters because a system that warns everyone can look safe while being unhelpful.[1]
Who it affects
Someone asking about a worrying symptom could delay care if an assistant accepts a benign explanation too quickly; indiscriminate warnings can also increase anxiety and unnecessary demand.[1]
Keep in mind
The work is an unreviewed preprint using constructed English-language prompts, one physician reviewer, an automated judge and no patients, clinical decisions or outcomes.
Source analysis · 1 linked source. Original source date: 20 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
Matched questions tested both missed warnings and needless alarm
A new preprint finds that general-purpose AI assistants frequently accepted a user's benign explanation even when the written symptoms left a time-critical condition possible. The researchers built SelfDx from 100 matched pairs, split evenly between condition and medication questions. In each pair, a caution case leaves a serious alternative open; its partner keeps the same self-diagnosis but adds evidence intended to rule that alternative out. This paired design matters because a system that warns everyone can look safe while being unhelpful.
The 200 messages were written rather than collected from patients. Forty per cent of caution cases were graded emergent, 51% urgent and 9% chronic. A physician reviewed the pairs and grading criteria. Twenty-seven models from six labs answered each prompt three times. The automated judge scored only the first two substantive sentences, reflecting the authors' view that a warning buried later may arrive after the user has anchored on reassurance.
For 20 current-generation models, appropriate agreement on cases where reassurance was warranted ranged from 82% to 100%. Appropriate caution on condition cases averaged 46.6%, and even the strongest model reached 77.0%. Under the stricter requirement to name a specific serious alternative early, the best rate was 32.6%. These results describe response openings to constructed prompts, not diagnostic accuracy or clinical safety in use.[1]
Prompting helped caution but also produced false alarms
The team then tested three safety prompts and supervised fine-tuning on Llama 3.3 70B. Prompts raised strict caution on condition cases from 0% to between 56.3% and 74.1%, but reduced appropriate agreement by 9 to 35 percentage points. Fine-tuning on 128 separate examples lifted strict caution to 51.9% and agreement to 99.7% across three runs. That comparison is encouraging, but it used one open-weight model and synthetic training answers written by another model.
For patients, the finding is a reason not to treat conversational reassurance as triage. It does not show that any named model misses a particular emergency in real practice. The rubric was revised after review by one physician, judge–physician agreement was assessed on the same ratings used in that revision, and the automated judge came from a developer represented in the model panel. A prospective study with diverse clinicians, real user language, full conversations and measured care-seeking outcomes would substantially change the assessment.[1]
The distinction that matters
How the paired test separates caution from alarm
The self-diagnosis stays fixed while the evidence changes.
Caution case
Serious alternative remains possible
Model should challenge early
Agreement case
Added evidence rules it out
Model should answer directly
Evaluation
First two substantive sentences scored
Placement and content both count
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Evidence check
The work is an unreviewed preprint using constructed English-language prompts, one physician reviewer, an automated judge and no patients, clinical decisions or outcomes.
Reporting basis: Source analysis. Original source date: 20 October 2026.
Continue the story
Read next
Continue exploring health advice: compare the finding and its evidence limits.
How safe was ChatGPT health advice in four Global South countries?
An analysis of 15,555 donated health conversations rated about one in five high-risk. The estimate comes from an unreviewed convenience sample and an LLM judge, not clinician review or measured patient harm.
Read the next reportExplore the full source and evidence record
What this means for people
- Someone asking about a worrying symptom could delay care if an assistant accepts a benign explanation too quickly; indiscriminate warnings can also increase anxiety and unnecessary demand.
Global context
Health-service access, emergency guidance and conversational language vary by country. The benchmark is useful for model behaviour, but deployment evidence needs local clinical review and real-world monitoring.
What the evidence does not yet show
- The work is an unreviewed preprint using constructed English-language prompts, one physician reviewer, an automated judge and no patients, clinical decisions or outcomes.
What to watch next
- Independent physician panels, multilingual and real-conversation tests, externally replicated scoring and prospective evidence about whether earlier cautions improve care without excessive referrals.
Living evidence record
Impact record IAI-1AXITLY
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
21 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 21 October 2026
Research question, topics and regions
Whether general-purpose AI assistants appropriately challenge a user's plausible self-diagnosis when a time-critical alternative remains possible, without becoming needlessly cautious when new evidence rules it out
Topics: Health advice · Patient safety · Self-diagnosis · Model evaluation · Medical AI
Regions: United States · North America · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.