Four chatbots covered less guidance when simulated teenage depression was severe
A peer-reviewed audit used 1,600 scripted conversations to test ChatGPT-5, ChatGPT-4, Claude 4.5 and Gemini 2.5. Coverage was stronger for anxiety than depression and for moderate than severe presentations—but the benchmark did not test treatment, safety or real teenagers.
The short version
- What happened
- Researchers developed AMHART with clinician review, reducing 27 candidate items to an 18-item framework and using a 14-item yes/no/not-applicable grading rubric.
- Who it affects
- Teenagers may turn to general chatbots before speaking to an adult or clinician. This study gives evaluators a structured way to look for omissions, but it does not validate any chatbot as mental-health care.
- What remains unproven
- Synthetic personas and scripted conversations rather than real adolescents
Keep in mind
Synthetic personas and scripted conversations rather than real adolescents
Coverage benchmark, not a measure of diagnosis, treatment or symptom change
Automated evaluators were not statistically calibrated against the full audit set
Four named model versions at one point in time
No crisis outcome, downstream behaviour or clinician decision measured
US-developed clinician framework may not transfer unchanged across health systems and cultures
Source analysis · 1 linked source. Primary analysed source date: 28 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
The benchmark asked whether replies included clinician-identified guidance
The researchers built the Adolescent Mental Health AI Response Tool, or AMHART, to make a narrow question measurable: when a teenager describes anxiety or depression, does a chatbot response cover elements clinicians consider relevant? Clinician review reduced 27 candidate items to an 18-item framework spanning seven domains. A 14-item yes/no/not-applicable rubric was then used to grade model replies.
The main test used eight fictional adolescent persona cards and generated 1,600 conversations with ChatGPT-5, ChatGPT-4, Claude 4.5 and Gemini 2.5. Each conversation contained ten chatbot responses across 20 turns. Coverage was greater for anxiety than depression and for moderate than severe presentations. That pattern matters because a fluent response may look supportive while omitting guidance that becomes more important as symptoms worsen.[1]
Coverage is not the same as safe or effective mental-health care
Three independent language-model evaluators graded outputs after an initial comparison with 84 human reference labels. None was fine-tuned, statistically calibrated or shown the human labels. Automated grading makes a large audit possible, but disagreements or shared model blind spots can still shape the scores.
The study did not involve real adolescents, clinicians making decisions, diagnoses or treatment outcomes. It therefore cannot show that one chatbot prevents harm, improves symptoms or should be used as a therapist. The practical lesson for families, schools and health services is narrower: product checks should test severe as well as moderate scenarios and inspect what is omitted, not only whether a reply sounds empathetic. Prospective studies with human review, crisis pathways and measured outcomes would change the assessment.[1]
The distinction that matters
What the AMHART audit measured
The study connected clinician-informed criteria to synthetic conversations, then stopped before clinical outcomes.
Framework
Clinicians reviewed candidate response elements
18 items across seven domains
Synthetic audit
Four chatbots answered fictional adolescent personas
1,600 conversations graded for coverage
Not tested
No real patient or treatment follow-up
Safety and clinical benefit remain unknown
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring chatbots: compare the finding and its evidence limits.
Do mental-health chatbots help university students?
A preregistered review found small short-term symptom advantages over waitlists or low-intensity materials, based on three depression and four anxiety comparisons. Two active conversational-control trials found no reliable added benefit, and certainty was low to very low.
Read the next reportExplore the full source and evidence record
What this means for people
- Teenagers may turn to general chatbots before speaking to an adult or clinician. This study gives evaluators a structured way to look for omissions, but it does not validate any chatbot as mental-health care.
Global context
Adolescent mental-health services, crisis routes, safeguarding duties and language norms vary widely. A US-developed English-language coverage framework can support comparison, but local clinicians and young people would need to test whether its criteria and examples transfer safely.
What the evidence does not yet show
- Synthetic personas and scripted conversations rather than real adolescents
- Coverage benchmark, not a measure of diagnosis, treatment or symptom change
- Automated evaluators were not statistically calibrated against the full audit set
- Four named model versions at one point in time
- No crisis outcome, downstream behaviour or clinician decision measured
- US-developed clinician framework may not transfer unchanged across health systems and cultures
What to watch next
- Independent replication with human adjudication
- Prospective testing with adolescents under ethical safeguards
- Performance in crisis and self-harm scenarios
- Non-English and cross-cultural validation
- Measured effects on help-seeking, symptoms and access to care
Living evidence record
Impact record IAI-1B23SXM
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
28 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Nature Mental Health published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 28 October 2026
Research question, topics and regions
Whether four general-purpose chatbots cover clinician-identified elements of appropriate responses to simulated adolescent anxiety and depression conversations
Topics: Adolescent mental health · Chatbots · Clinical evaluation · Depression · Anxiety · AI safety
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.