Can an AI review therapy sessions well enough to support supervisors?
A small US study found its best open model could classify cognitive-behavioural therapy skills with 0.72 balanced accuracy and distinguish most skills in an ongoing trial. Thirty gold-rated transcripts and three feedback raters leave clinical benefit unproven.
The short version
- What happened
- The development dataset contained 318 de-identified sessions from 45 people, but only 30 transcripts had expert gold labels; 10 of those were double-rated, with Cohen's kappa of 0.71.
- Who it affects
- Therapists and supervisors could receive more frequent, structured transcript review, but false positives, missed context and polished feedback may misdirect training. Human review remains essential before any effect on care is known.
- What remains unproven
- The gold-labelled set was 30 selected transcripts, with only 10 double-rated; the work is an unreviewed preprint.
Keep in mind
The gold-labelled set was 30 selected transcripts, with only 10 double-rated; the work is an unreviewed preprint.
The ongoing trial involved 22 people and one therapist delivering both arms, and its transcripts had no item-level expert labels.
Only three experts rated feedback, with weak agreement on relative preferences; high absolute scores do not establish behaviour change.
The source reports no separate funding or competing-interest statement and does not measure patient outcomes from AI-assisted supervision.
Source analysis · 1 linked source. Original source date: 20 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
The model was tested against labels and an external trial
The team began with 331 psychotherapy sessions, excluded 13 and retained 318 sessions from 45 participants. Experts gold-rated 30 transcripts across 60 CBT skill categories, producing 1,800 binary judgments; 10 transcripts were independently double-rated and reached kappa 0.71. The other 288 sessions helped the system retrieve examples and estimate uncertainty but did not expand the gold-label denominator.
Three open-weight models were compared. The strongest setup used Qwen3.5-9B, asked for evidence before a category decision and scored 0.72 balanced accuracy with a 95% bootstrap interval of 0.70–0.74; its Matthews correlation was 0.45, interval 0.41–0.49. A simpler full-transcript baseline scored 0.61 and 0.27 respectively. These are moderate classification results, not an automated replacement for a clinical supervisor.[1]
Promising validation stopped short of patient benefit
The researchers next processed 102 sessions from an ongoing trial involving 22 people: 57 CBT sessions and 45 comparison sessions, delivered by the same therapist. Four of five model-scored skill dimensions were significantly higher in CBT; guided discovery rose but was not significant. This known-groups test supports construct validity, yet the trial transcripts lacked item-level expert labels, so it cannot establish classification accuracy on that dataset.
Three expert users also made 30 pairwise comparisons and 30 absolute ratings of generated feedback. Retrieval-augmented feedback did not significantly beat basic feedback, and agreement on relative actionability and clarity was slight or near chance, although both versions received high absolute ratings. Audio had not been retained, so tone and pacing were invisible. A larger multi-site, prospectively labelled study must test whether feedback changes supervisor behaviour, therapist practice or patient outcomes.[1]
The distinction that matters
What each validation step can show
The study combines three different tests, each answering a narrower question.
Gold labels
Compare 1,800 model judgments with expert ratings
Estimates category classification
Known groups
Compare CBT and control sessions
Tests whether scores move in an expected direction
Expert feedback
Ask three raters to judge generated supervision text
Assesses perceived usefulness, not clinical effect
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring mental health: compare the finding and its evidence limits.
What do AI mental-health chatbots still get wrong?
A company-built benchmark scored 61,819 replies from 18 models across synthetic, three-session support conversations. Repetitive questioning and poorly calibrated guidance dominated the errors—but synthetic patients, one English prompt and automated judging cannot establish clinical safety or benefit.
Read the next reportExplore the full source and evidence record
What this means for people
- Therapists and supervisors could receive more frequent, structured transcript review, but false positives, missed context and polished feedback may misdirect training. Human review remains essential before any effect on care is known.
Global context
Supervision shortages make automated review attractive, especially in under-resourced services. This US CBT-specific evidence may not transfer to other therapies, languages, cultures, documentation practices or regulatory settings.
What the evidence does not yet show
- The gold-labelled set was 30 selected transcripts, with only 10 double-rated; the work is an unreviewed preprint.
- The ongoing trial involved 22 people and one therapist delivering both arms, and its transcripts had no item-level expert labels.
- Only three experts rated feedback, with weak agreement on relative preferences; high absolute scores do not establish behaviour change.
- The source reports no separate funding or competing-interest statement and does not measure patient outcomes from AI-assisted supervision.
What to watch next
- Prospective multi-site evaluation with independently labelled transcripts
- Randomised tests of supervisor and therapist behaviour
- Patient outcomes and safety events
- Performance across languages, therapy models and demographic groups
Living evidence record
Impact record IAI-0VQL170
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
22 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 22 October 2026
Research question, topics and regions
Whether small open-weight language models can identify cognitive-behavioural therapy skills in transcripts and generate useful supervision feedback
Topics: Mental health · Psychotherapy · Clinical supervision · Open models · Evaluation
Regions: United States · North America
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.