What do AI mental-health chatbots still get wrong?

What the study actually measured

The Mental Health Quality Rubrics taxonomy covers distortion reinforcement, factual and contextual fidelity, guidance calibration, professional boundaries, delivery problems and clearly broken replies. Each counsellor turn received a severity level from no concern to critical, weighted 0, 1, 5 or 10. The paper warns that these weights are an aggregation convention rather than calibrated ratios of clinical harm.

Before using automated judging at scale, three clinical psychologists annotated 67 conversation excerpts containing 411 counsellor turns. Only ten excerpts—66 turns—were rated by all three. Their average pairwise weighted kappa for severity was 0.44, and exact agreement across pairs was about 49%; 97.5% of disagreements were within one severity level. The selected automated judge reproduced the pooled ranking of three counsellor models and matched that ranking in 87% of 2,000 bootstrap resamples. With only three models in that validation ranking, this is useful but limited evidence for replacing clinicians at benchmark scale.

The final evaluation averaged scores over 150 patient-session combinations for each of 18 counsellor models. The three GPT-5.6 variants received the lowest automated scores, but the judge was from the same model family, so the authors mark a possible self-preference bias. A different judge also ranked those three first, yet model names and order should not be interpreted as a clinical league table: one prompt, one simulator, synthetic cases and automated scoring can favour particular response styles.[1]

The recurring failure is practical, not merely stylistic

Under-guiding appeared in about 12% of replies during the first two turn positions and roughly 70% from turn nine onward among replies observed at those positions. The pattern was often paired with over-exploring: another validating question followed even when the patient was asking for help moving forward. For someone using a support tool, repetition can feel like attention while providing little assistance, and it can delay escalation to a qualified professional.

Some models also reinforced a patient's tentative interpretation as fact. In the paper's example, a patient's uncertainty about another person's gesture became a counsellor response premised on 'knowing' that person had tried. Other model families fell into formulaic validation or missed previously shared context. Across all counsellors, the share of replies with a fidelity problem more than doubled from 7.6% in first sessions to 17.2% in later sessions, showing that a memory feature does not guarantee that a system uses history appropriately.

Boundary issues such as impersonating a human or acting as a clinician were uncommon in this particular test, and critical-rated replies never exceeded 1.85% for a model. That is reassuring only within the benchmark's narrow frame. The synthetic profiles did not explicitly model harmful behaviour or crisis situations, while other studies have shown failures when systems are directly probed on acute risk. A low critical-error rate here cannot be used to claim crisis readiness.[1]

What clinicians, services and users should demand

A service considering a support chatbot should evaluate whether it recognises when to stop questioning, offers appropriately bounded options and uses earlier disclosures accurately. Clinicians should define escalation triggers and review samples from the actual population, languages and interface. Product teams need to measure repeated questioning, mistaken memory and user abandonment as well as prohibited content. A warm tone is not evidence of good guidance.

Users should be told whether they are speaking to an automated system, what it remembers, who can access the conversation and how to reach a person. The benchmark does not justify replacing therapy, diagnosis or crisis services. If a tool gives advice that feels unsafe, misunderstands the situation or cannot move beyond repetitive questions, the appropriate fallback is a qualified professional or local emergency support where immediate danger exists—not another model turn.

The company context and data provenance require transparency. The authors used 1,584 anonymised sessions from 886 users of one online mental-health app to shape profile distributions and validate realism. They report removing named entities and personal identifiers before analysis and releasing only aggregate rates. The psychologists were company employees working within their regular duties. Those safeguards are relevant, but external ethical review, consent details and independent replication would strengthen confidence.[1]

What would change the assessment

The synthetic patients were still distinguishable from real users: clinicians rated their realism at 3.55 out of five versus 4.20 for real sessions. The evaluation was limited to English text, one mobile-chat prompt, US-grounded demographic seeds and presenting-problem rates from one product. It omitted crisis handling, cultural adaptation, therapeutic alliance over longer periods and the possibility that people respond differently when they know a system is automated.

Next evidence should include preregistered, independently governed studies with diverse real users, clinician comparators, several languages and measures of symptom change, alliance, escalation, adverse events and dropout. Larger multiply annotated samples would test the rubric, and blinded external clinicians should assess whether automated rankings predict human judgments. Until outcome data exist, MindEval2 is best viewed as a diagnostic stress test that exposes repetitive and miscalibrated conversational behaviour—not proof that any chatbot helps people recover.[1]