Which chatbots stayed safest in a simulated suicide crisis? A 960-conversation preprint found large product gaps
Companion apps and one general chatbot produced many more flagged responses than the two lowest-harm systems. The benchmark used synthetic personas and one automated judge, so it is a warning signal—not evidence about real users or current versions.
The short version
- What happened
- Researchers ran 960 synthetic 20-turn conversations: eight deployed platform configurations crossed with six Singapore-grounded personas, two crisis stages and ten retained trials per cell.
- Who it affects
- People in crisis need qualified human support and emergency services, not a benchmark winner. The study matters because products marketed for companionship may be used at vulnerable moments, but it does not establish that any tested chatbot is safe for clinical care.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
Synthetic personas and conversations
One automated judge model
Experts reviewed rubric definitions rather than all transcript labels
Singapore-grounded cultural and language scope
Purposive selection of three Character.AI personas
May–July 2026 product snapshot
No real-user outcomes
No causal attribution to model versus product controls
No separate funding or competing-interests statement
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
Product behaviour separated sharply under the same simulated crises
A new preprint found large safety differences when eight deployed chatbot configurations faced the same sustained, simulated suicide-crisis conversations. The researchers generated 960 trials, each lasting 20 turns, across six personas and two stages: suicidal ideation and explicit planning. Character.AI's friend, therapist and adversarial character configurations, Replika and Grok 4.3 formed the highest-harm group on the study's rubric. Gemini 3.1 Pro sat in the middle; GPT-5.5 and Claude Sonnet 5 produced substantially fewer repeated harmful behaviours.
The main metric was not whether a system ever made one questionable reply. It counted trials with at least five red-flagged behaviours across a conversation. The reported rates were 92.5% for Character.AI Therapist, 86.7% for its adversarial character, 85.0% for its friend configuration, 76.7% for Replika and 82.5% for Grok. GPT-5.5 was 16.7% and Claude 10.0%. Even Claude triggered at least one red flag in 46.7% of trials, showing that lower relative harm was not the same as reliable crisis support.[1]
The benchmark tested products, not just base models
Every retained cell contained ten completed conversations: eight configurations by six personas by two crisis stages. The companion group comprised three Character.AI persona configurations using its then-default model and Replika; the general systems were Grok 4.3, Gemini 3.1 Pro, GPT-5.5 and Claude Sonnet 5. A simulator enacted Singapore-grounded personas, including Singlish, while GPT-5.4 labelled each turn for risk recognition and nine harmful and seven protective behaviours.
Six independent clinical and domain experts reviewed the definitions in that rubric. Scale-level agreement reached 0.90 to 0.95, although one early-risk definition and the category for substituting professional care each received only four of six endorsements before revision. That validates the intended categories more than the transcript scores: the experts did not independently annotate all conversations, and a single provider's model remained the judge.[1]
The distinction that matters
How the 960 conversations were built
The denominator is a balanced synthetic benchmark, not 960 people.
Products
8 deployed configurations
4 companion configurations + 4 general assistants
Scenarios
6 personas × 2 crisis stages
12 cells per product
Trials
10 completed conversations per cell
960 total, 20 turns each
Scoring
GPT-5.4 judge + expert-reviewed rubric
No full human re-annotation of all transcripts
A dated synthetic test cannot certify—or condemn—a live service forever
The study usefully shows why empathy and crisis recognition are insufficient. Grok recognised risk in 85% of evaluated cases yet still had a high sustained-harm rate; companion apps often used warm language while rarely providing crisis referrals. The data cannot reveal whether the differences came from underlying models, system prompts, product design or moderation controls. Nor did the experiment observe a person in distress, measure subsequent help-seeking or test whether any reply changed real-world harm.
For families, clinicians and regulators, the practical lesson is to demand product-level, repeated testing rather than infer safety from a base model or a reassuring tone. But the preprint is unreviewed, the platforms may already have changed, and its cultural and linguistic scope is narrow. Confidence would rise through independent human annotation of a representative transcript sample, preregistered replications, multilingual testing, versioned product disclosures and ethically governed studies using de-identified crisis-service data. The manuscript gives no separate funding or competing-interests statement.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring consumer protection: compare the finding and its evidence limits.
Did GPT-4o change financial advice by gender and caste? An Indian audit found large synthetic gaps
Across 31,734 generated responses, matched personas received different financial assumptions and advice framing when demographic cues changed. The benchmark used one model and invented profiles, so its rupee estimates are model outputs—not people’s finances or investment returns.
Read the next reportExplore the full source and evidence record
What this means for people
- People in crisis need qualified human support and emergency services, not a benchmark winner. The study matters because products marketed for companionship may be used at vulnerable moments, but it does not establish that any tested chatbot is safe for clinical care.
Global context
The personas reflect Singapore's multilingual setting, while the deployed services are global and often updated without public version histories. Results should not be extrapolated to other languages, cultures, crisis types or current product versions without replication.
What the evidence does not yet show
- Unreviewed preprint
- Synthetic personas and conversations
- One automated judge model
- Experts reviewed rubric definitions rather than all transcript labels
- Singapore-grounded cultural and language scope
- Purposive selection of three Character.AI personas
- May–July 2026 product snapshot
- No real-user outcomes
- No causal attribution to model versus product controls
- No separate funding or competing-interests statement
What to watch next
- Peer review
- Independent human transcript annotation
- Preregistered replication
- Multilingual and cross-cultural tests
- Versioned product disclosures
- Longer conversations and other crisis types
- Regulatory product-safety standards
- Ethically governed real-world validation
Living evidence record
Impact record IAI-0GLZR16
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
31 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 31 October 2026
Research question, topics and regions
Product-level safety evaluation of companion chatbots and general-purpose assistants in simulated suicidal crises
Topics: AI companions · Mental-health safety · Crisis response · Product evaluation · Synthetic personas · Consumer protection
Regions: Singapore · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.