Clinicians made a chatbot guardrail auditable—and it still over-blocked benign replies
Two clinical-psychology trainees translated four mental-health safety checks into verifiable rules. The rules exposed why decisions differed from an LLM judge, but depended on model-extracted facts and blocked many benign boundary cases.
The short version
- What happened
- Two doctoral trainees in clinical psychology defined four checks covering crisis resources, referral, medication boundaries and reinforcement of harmful beliefs or targets.
- Who it affects
- A person seeking mental-health support can be harmed by both missed escalation and excessive blocking. Traceable rules may help clinicians inspect that trade-off, but they do not replace a qualified service, emergency pathway or real-world evaluation.
- What remains unproven
- The paper is an unreviewed preprint and does not report patient outcomes or a live deployment.
Keep in mind
The paper is an unreviewed preprint and does not report patient outcomes or a live deployment.
The 294-item descriptive sample informed one rule decision; confirmatory testing was limited to 500 held-out boundary prompts.
Two trainees authored the checks, and the weakest pilot agreement was for the crisis-resource check that drove many outcomes.
Fifteen of sixteen facts came from one Llama-3.3-70B extractor and one prompt; results were not shown to transfer to another extractor.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
The rules could be checked exhaustively
The clinicians refined four questions using a 19-item pilot and a 75-item main annotation batch. Researchers expressed the result through sixteen typed facts, proved seven properties with an SMT solver and enumerated 393,216 valid fact combinations. At runtime, the same Llama model used for comparison extracted fifteen facts; an exact text match supplied the crisis-number fact.[1]
Traceability revealed a real trade-off
The LLM judge requested a crisis resource in 55 of 65 replies that already contained 988 or 911; the deterministic rule avoided that error. Yet on the reference-scored crisis check, the judge had fewer false alarms: 23 of 81 versus 39 of 81. Both missed 2 of 13 labelled violations. The paper therefore supports auditability, not superiority or clinical safety.[1]
The distinction that matters
Where the guarantee starts—and stops
The pipeline separates clinical policy, model interpretation and deterministic enforcement.
Clinicians
Define four safety checks
An explicit, revisable policy
Model
Extract fifteen conversation facts
Still fallible and prompt-dependent
Rules
Apply verified predicates
Traceable consistency, not clinical correctness
Verified rules still inherit uncertain facts
Formal verification guarantees only that the written rules are applied consistently to the supplied facts. Fifteen facts came from one model and prompt, several clinician decisions remained open, and held-out testing used 500 boundary prompts rather than real patient conversations or outcomes. Any deployment would need broader clinical ownership, local crisis pathways and prospective safety monitoring.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring mental health: compare the finding and its evidence limits.
Can a maternal-health AI chatbot improve anxiety or depression? A 400-person trial found no 12-month benefit
A US randomized trial found no meaningful improvement in anxiety, depression, sleep, postpartum care or emergency visits after a year. The chatbot was used heavily, but it also relied on human safety follow-up and the trial was not designed to rule out small effects.
Read the next reportExplore the full source and evidence record
What this means for people
- A person seeking mental-health support can be harmed by both missed escalation and excessive blocking. Traceable rules may help clinicians inspect that trade-off, but they do not replace a qualified service, emergency pathway or real-world evaluation.
Global context
Crisis resources, referral thresholds and legal duties vary by country. A portable governance pattern would need locally authorised clinicians, verified contact information and testing across languages and service settings.
What the evidence does not yet show
- The paper is an unreviewed preprint and does not report patient outcomes or a live deployment.
- The 294-item descriptive sample informed one rule decision; confirmatory testing was limited to 500 held-out boundary prompts.
- Two trainees authored the checks, and the weakest pilot agreement was for the crisis-resource check that drove many outcomes.
- Fifteen of sixteen facts came from one Llama-3.3-70B extractor and one prompt; results were not shown to transfer to another extractor.
What to watch next
- Replication with independent clinicians and languages
- Prospective evaluation in real services
- Direct labelling of extracted facts
- Measured effects on missed crises and inappropriate blocking
Living evidence record
Impact record IAI-1QA1UL3
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
23 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 23 October 2026
Research question, topics and regions
Whether clinician-authored mental-health safety requirements can be applied through checkable rules rather than an opaque LLM judge
Topics: Mental health · Chatbot safety · Clinical governance · Formal verification · Over-refusal
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.