Can an LLM screen students for depression reliably? One setup changed its answer in 23.6% of cases
In 6,395 Chinese undergraduates, some language-model prompts matched conventional models at one threshold. But performance changed with the cutoff, XGBoost gave better probability estimates, and repeated GPT calls disagreed for 532 of 2,258 students.
The short version
3 key points
- What changed
- The study used records from 6,395 Huangshan University students, splitting first-, second- and third-year cohorts into development, test and within-institution holdout sets.
- Why it matters
- Students could be wrongly reassured or sent for unnecessary review if a screening label changes between identical calls. Human assessment and an accountable referral pathway remain essential.
- What to watch
- External university cohorts and clinician-confirmed outcomes
Similar headline scores concealed different reliability
A language-model screening result could change when the same student record was submitted again, a peer-reviewed benchmark has found. Researchers invited 6,596 first- to third-year undergraduates at Huangshan University; 6,395 completed a 146-item assessment in 2024. They used first-year records for development, second-year records for testing and third-year records as a separate within-university holdout. The target was whether a student's 20-item Self-Rating Depression Scale score crossed 53—not a diagnosis of depression.[1]
Who it affects
Students could be wrongly reassured or sent for unnecessary review if a screening label changes between identical calls. Human assessment and an accountable referral pathway remain essential.[1]
Keep in mind
The outcome was an SDS questionnaire threshold, not a clinician-confirmed diagnosis.
All participants came from one Chinese university during one survey period; the holdout was not external or temporal.
The inputs required several other psychological questionnaires and did not reduce assessment burden.
Fairness analysis was limited to sex and did not examine socioeconomic status, disability, discipline or intersectional groups.
Source analysis · 1 linked source. Original source date: 19 October 2026.
Read the full report · 4 min The evidence, methods, implications and limitations
Similar headline scores concealed different reliability
A language-model screening result could change when the same student record was submitted again, a peer-reviewed benchmark has found. Researchers invited 6,596 first- to third-year undergraduates at Huangshan University; 6,395 completed a 146-item assessment in 2024. They used first-year records for development, second-year records for testing and third-year records as a separate within-university holdout. The target was whether a student's 20-item Self-Rating Depression Scale score crossed 53—not a diagnosis of depression.
At a fixed 0.5 cutoff on the 2,186-person test set, the leading GPT and Gemini prompts recorded macro-F1 scores of 0.732 and 0.730, versus 0.674 for XGBoost. That apparent lead was not stable. At a development-selected specificity target, XGBoost reached 0.738. It also had the lowest Brier score, indicating better probability estimates. Pairwise intervals found no clear difference among the three leading language-model prompts at the fixed cutoff.[1]
Repeated calls exposed a deployment problem
The most consequential test repeated each finalist 10 times across all 2,258 holdout records at temperature 0.1. The GPT condition changed its binary answer for 532 students, or 23.6%; the Gemini condition changed for 34, or 1.5%. In a representative 400-person subset, setting temperature to zero did not eliminate the GPT instability. A screening programme cannot treat an answer as dependable merely because a single batch-level score looks competitive.
The comparison was unusually careful about calibration, thresholds and structured-output failures, but it remained an offline benchmark. All three cohorts came from one university and one survey period. Screen-positive prevalence rose from 16.3% in development to 24.4% in test and 27.9% in holdout. That shift makes transport and clinical actionability especially important. The models also relied on 126 variables from other psychological questionnaires. They did not shorten the assessment burden or discover depression from ordinary conversation.[1]
The distinction that matters
What was tested
The three cohorts were separated by year group, but remained inside one university and survey period.
1,951 development
Fit baselines and choose thresholds
First-year cohort
2,186 test
Compare 12 LLM prompts and two baselines
Select two finalists
2,258 holdout
Retest four models
Within-university transport
10 repeats
Check answer stability
23.6% vs 1.5% inconsistent
What a university should require before any trial
For students and university health services, the paper is a warning against automated referral or reassurance. A model-selected label would still need a locally chosen action threshold, qualified professional review, consent, secure handling of sensitive data and a route to assessment. The study did not compare the systems with a clinical interview, test treatment decisions or establish fairness beyond an exploratory female–male analysis. It also did not measure waiting times, staff workload or whether flagged students sought help. A conventional score or tabular model may be easier to audit for this structured task.
Confidence would require external validation at different universities, prospectively adjudicated clinical outcomes and a workflow trial measuring false reassurance, unnecessary review and access to care. The work was publicly and institutionally funded; one author was employed by AIFORIA (Hangzhou). Until those tests exist, the useful result is methodological for service designers: prompt choice, cutoff, calibration and repeated-call stability are separate questions, and none can be replaced by one flattering accuracy number.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Evidence check
The outcome was an SDS questionnaire threshold, not a clinician-confirmed diagnosis.
All participants came from one Chinese university during one survey period; the holdout was not external or temporal.
The inputs required several other psychological questionnaires and did not reduce assessment burden.
Fairness analysis was limited to sex and did not examine socioeconomic status, disability, discipline or intersectional groups.
Reporting basis: Source analysis. Original source date: 19 October 2026.
Continue the story
Read next
Also about large language models
LLMs found depression signals in one-minute diaries—but not a diagnosis
Seven language models produced scores that moved with self-reported mood in 108 US adults. The peer-reviewed study is an early measurement test, not evidence that passive screening is accurate or safe in clinical care.
Read the 4 min reportAlso about student mental health
Can exercise patterns identify student distress without becoming a diagnosis?
A Chinese study linked self-reported exercise patterns with a constructed mental-health category among 1,127 vocational medical students. Its best model classified 88.9% of one internal test set correctly, but missed roughly one in four severe-category cases and has not been externally validated.
Read the 7 min reportExplore the full source and evidence record
What this means for people
- Students could be wrongly reassured or sent for unnecessary review if a screening label changes between identical calls. Human assessment and an accountable referral pathway remain essential.
Global context
Universities differ in languages, questionnaires, referral capacity and privacy rules. Any prospective use would need local validation and governance; the reported scores cannot be transported directly to another campus.
What the evidence does not yet show
- The outcome was an SDS questionnaire threshold, not a clinician-confirmed diagnosis.
- All participants came from one Chinese university during one survey period; the holdout was not external or temporal.
- The inputs required several other psychological questionnaires and did not reduce assessment burden.
- Fairness analysis was limited to sex and did not examine socioeconomic status, disability, discipline or intersectional groups.
What to watch next
- External university cohorts and clinician-confirmed outcomes
- Prospective trials with human review and referral consequences
- Stability tests after provider or model updates
- Broader subgroup audits and privacy controls
Living evidence record
Impact record IAI-1TFF8QU
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
19 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Psychiatry published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 19 October 2026
Research question, topics and regions
Whether language models can predict an SDS depression-screening threshold from other structured questionnaires consistently enough to justify prospective evaluation
Topics: Student mental health · Screening · Model stability · Large language models · Clinical evidence
Regions: China · Malaysia · Global health
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.