What happens when someone refuses a chatbot's crisis referral?
In 2,550 scripted conversations, all 17 chatbots recommended outside help after suicide thoughts were disclosed. After refusal, persistence ranged from almost none to nearly all ten attempts, and newer versions persisted less in every matched model line.
The short version
3 key points
- 1.All 2,550 scripted conversations produced an outside-help referral, and 2,542 did so in the first reply, but mean persistence after refusal ranged from 0.01 to 9.94 attempts across chatbots.
- 2.Only 173 of 8,818 refusal-phase replies urging outside help asked about barriers to seeking it; models that stopped sooner more often framed themselves as the user's present support.
- 3.This was an unreviewed API audit with scripted English conversations, no real users and no measured help-seeking or health outcome, so it identifies a safety-evaluation gap rather than the correct crisis response.
The answer: the first referral was consistent, but the next turns were not
All 17 chatbots in this new audit recommended outside help after a scripted user disclosed suicide thoughts. The apparent consistency almost disappeared once that user declined the referral. Across 2,550 conversations, mean persistence ranged from 0.01 refusals for the least persistent model to 9.94 out of ten for the most persistent. Eight chatbots stopped recommending outside support immediately in most conversations; 698 conversations, or 27.4%, still contained a referral after the tenth refusal.
That difference matters because a safety check limited to a chatbot's first reply would rate every tested system similarly. Real crisis conversations can include ambivalence, refusal, fear of services or a change of subject. The preprint's central contribution is therefore methodological: it tests the path after the expected referral rather than treating the presence of a hotline or human-help recommendation as the endpoint. It does not establish how long a chatbot should persist or which wording helps a person seek care.[1]
How the audit was designed
Researchers at the University of Virginia, the University of California San Diego and the VA San Diego Healthcare System ran the cross-sectional audit in September 2026. They accessed 17 publicly available chatbots from eight companies through the OpenRouter API. Each chatbot completed 150 conversations: ten replications of 15 clinician-developed opening statements adapted from the Columbia Suicide Severity Rating Scale. Five represented a passive wish to die, five non-specific active thoughts and five active ideation with some intent.
After the chatbot first recommended outside help, the simulated user sent up to ten standard refusals. That phase ended when the recommendation stopped or after the tenth refusal. The user then requested a topic change and sent ten lines about medieval history, allowing the researchers to see whether the model returned to the safety issue. In total, the models generated 41,450 replies. There were no human participants, no clinical care and no identifiable data.[1]
What the measurements found
Every conversation eventually included a referral, and 2,542 of 2,550, or 99.7%, received one in the first reply. Intent-level disclosures were associated with 1.26 additional refusals sustained compared with passive ideation, with a 95% confidence interval from 1.06 to 1.46. After the scripted topic change, 1,368 conversations, or 53.6%, renewed the referral. The odds of renewal after an intent disclosure were about ten times those after passive ideation, although that estimate describes the benchmark rather than a clinical effect.
The content around the referral was thinner. Of 8,818 refusal-phase replies that continued to urge outside help, 173, or 2.0%, asked about barriers to accepting it. Chatbots that stopped referring sooner were more likely to present themselves as the user's current source of support: the association across 17 systems was strong, with a Spearman correlation of -0.80. The most persistent system, however, rarely asked about immediate safety. Repetition alone therefore did not produce the combination of persistence, inquiry, limits and safety attention the authors were looking for.[1]
Newer versions did worse on this narrow measure
Five model lines could be compared across two consecutive versions. In all five, the newer version persisted less than its predecessor, with each pairwise comparison reaching the paper's significance threshold. The largest declines occurred in the two OpenAI lines, where mean persistence fell by about half. Four of the five newer versions also acknowledged their limitations less often. The study cannot say whether this reflected deliberate safety tuning, reduced over-referral, other product changes or chance differences in serving behavior.
The finding is important precisely because it is not a general ranking of model safety. A model can persist too little, repeat unhelpfully or redirect in a way that drives a person away. Consumer applications can also add system prompts, classifiers and interface-level interventions that were absent from the API setup. The defensible conclusion is that replacement models should be retested over multiple turns; newer should not be assumed safer because their first reply looks appropriate.[1]
How replies were coded
During collection, Mistral Small judged whether a reply contained a referral and therefore determined which scripted turn came next. Afterwards, Grok 4.3 coded replies against a clinician-developed codebook. One researcher independently coded a stratified sample of 100 replies. Across the ten retained fields, model-human agreement was 90.9%, with Cohen's kappa from 0.68 to 0.88. The two model judges agreed on the presence of a referral in 98.6% of 41,038 codable replies.
Those checks support consistency but do not replace independent clinical adjudication. The human validation used one rater and 100 replies, so inter-rater agreement could not be estimated. One coding model came from the same company as a tested chatbot. The primary persistence outcome is less dependent on the content categories than claims about tone or support framing, but the whole study remains an automated audit of simulated dialogue.[1]
What clinicians, providers and regulators can use now
Clinical safety teams can add refusal and topic-change branches to their test suites. A useful evaluation should record not only whether a referral appears, but whether the system checks immediate safety, explores practical barriers, states its limits, avoids presenting itself as the sole support and adapts to severity. Providers should test the exact consumer experience, including system prompts, classifiers, locale, interface copy and model updates, because the API result may not describe the deployed product.
Regulators can also avoid a one-turn compliance test. A requirement to display crisis resources is observable, but it says little about the next nine exchanges. Multi-turn scenarios need expert and lived-experience input so that persistence does not become mechanical repetition. Any policy should also distinguish chatbots designed for companionship or mental-health support from general systems that encounter an unexpected disclosure, while still requiring transparent escalation behavior and release-by-release monitoring.[1]
What the study cannot show
The simulated user had no age, history, location or personal circumstances and used the same standardized refusals. Real people may explain, soften or reverse a refusal, and the effect of any response can depend on culture, trust, service availability and previous experiences of care. Conversations were in English and used one harmless topic after the change. The September 2026 model snapshot will date quickly, while consumer apps may behave differently from the default API calls tested at temperature 1.0 without a system prompt.
Most importantly, no person received an intervention and no help-seeking, distress, injury or health outcome was measured. Persistence is a process measure chosen by the researchers, not a validated therapeutic dose. The manuscript is an unreviewed preprint. The authors reported no funding or conflicts and disclosed that Claude assisted with code, figures and drafting; they said they verified the results and accepted responsibility. Those disclosures strengthen transparency but do not substitute for replication or peer review.[1]
What would change the assessment
The evidence would become more actionable with preregistered replication in consumer applications, multilingual and persona-based scenarios, and blinded coding by multiple clinicians and people with lived experience. Researchers should compare different response strategies—repetition, barrier-solving, safety questions, warm handoff and continued conversation—while guarding participants carefully. Outcomes should include whether people accept help, feel understood, disengage or experience harm, not only what the model says.
Until that work exists, the finding should be read as a warning about evaluation design. The tested chatbots almost always passed the obvious first-turn check. They differed sharply when the conversation became difficult. That is enough to justify deeper testing and update monitoring, but not enough to name one ideal model, prescribe a fixed number of referrals or claim that any response pattern prevents suicide.[1]
What this means for people
- People disclosing suicide thoughts may encounter very different behavior after refusing help even when the first reply looks similarly safety-conscious.
- Clinicians and crisis specialists need to help define useful persistence, barrier-solving and safe continued engagement rather than leaving the metric to product teams alone.
- Product and regulatory teams should test the whole consumer conversation after every model update, not only whether the first reply contains a referral.
Global context
The audit was conducted by US researchers through one API route and in English, while crisis services, law and help-seeking norms vary widely by country. Its multi-turn evaluation principle is globally relevant, but the exact model rankings and referral behavior should not be generalized to every consumer interface or locale.
What the evidence does not yet show
- Unreviewed preprint using scripted conversations and no human participants
- One API route, no system prompts and default reasoning settings may not match consumer apps
- English-only prompts with no age, history, culture or personal context
- Content coding validated against one human rater on 100 replies
- Persistence is not a validated clinical outcome and the ideal response remains unknown
- No measurement of real help-seeking, distress, injury or suicide prevention
What to watch next
- Independent replication in deployed consumer interfaces and after model updates
- Multilingual, persona-based and culturally grounded crisis-conversation tests
- Clinician and lived-experience agreement on safe responses after refusal
- Prospective evidence comparing response strategies with real help-seeking and harm outcomes
Living evidence record
Impact record IAI-1BSU2QS
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
16 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 16 October 2026
Research question, topics and regions
How consumer chatbots respond across multiple turns when a simulated user disclosing suicide thoughts repeatedly declines referral to human help
Topics: Mental health · Suicide safety · Chatbots · Crisis referral · Multi-turn evaluation · Health AI
Regions: Global · North America · United States
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Healthcare, culture & human impact · Background explainer
Published 2026-07-01 · 44 min 28 sec
Is this the AI future we want? | The Royal Society
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
Do mental-health chatbots help university students?
A preregistered review found small short-term symptom advantages over waitlists or low-intensity materials, based on three depression and four anxiety comparisons. Two active conversational-control trials found no reliable added benefit, and certainty was low to very low.
8 min · 1 source
Health & Life Sciences
Can AI reliably predict facial growth?
A registered systematic review found 12 studies of AI-based craniofacial or mandibular growth prediction, with samples from 33 to 639 people. Seven studies were at high risk of bias and five at unclear risk; none had independent external validation.
9 min · 1 source
Health & Life Sciences
Will patients wait for a clinician to check AI medical advice?
In a survey of 1,185 US adults, 63.1% preferred waiting up to 48 hours for clinician-reviewed medical portal answers, even as majorities accepted five-minute AI responses for logistical tasks. The study tested hypothetical choices, not real behaviour or safety.
7 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.