Did AI patients train medical students as well as human actors?
A three-arm pilot randomized 61 students and analysed 58. All groups improved on the main communication checklist, with no detectable between-group difference—but the study was too small to establish equivalence and compared text-based AI with voice-based human practice.
The short version
- What happened
- The study combined 350 blinded expert assessments across five language models, 155 live AI consultations by 31 students and a three-arm pilot randomized study that analysed 58 participants.
- Who it affects
- Medical students could practise difficult questions on demand and educators could see which safety-critical details were missed. Patients would benefit only if that practice later improves real consultations; this study did not test that transfer.
- What remains unproven
- The pilot analysed 58 participants and was underpowered for small between-group differences or an equivalence claim.
Keep in mind
The pilot analysed 58 participants and was underpowered for small between-group differences or an equivalence claim.
AI training used text chat while human standardized patients used voice calls, so agent and communication modality cannot be separated.
Training and assessment reused the same five scenarios, increasing familiarity and potentially compressing differences between arms.
The single-institution Chinese-language study measured immediate performance, not retention, real-patient communication or clinical outcomes. The global rating scale had poor reliability (ICC 0.15).
The educational study was not prospectively registered. The authors report no specific funding and no competing interests.
Source analysis · 1 linked source. Primary analysed source date: 23 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The pilot found comparable gains, not equivalent training
A peer-reviewed Chinese study found no detectable difference in immediate communication-skill gains among students assigned to text-based AI patients, voice calls with human standardized patients or the usual curriculum. Sixty-one medical students were randomized; three withdrew before training, leaving 21 in the AI arm, 18 in the human-patient arm and 19 controls. Blinded examiners scored recorded consultations.
All three groups improved on the prespecified 14-item checklist, whose inter-rater reliability was acceptable at ICC 0.72. The between-group comparison was not significant (p=0.483), and baseline-adjusted analysis reached the same conclusion. But this was a pilot: post-hoc power was below 25% for the observed between-group effect, and the authors estimate at least 216 participants would be needed for 80% power. A null result therefore cannot show that AI and human practice are equivalent.[1]
Structured disclosure made the simulation responsive to better questioning
The system withheld some clinically important details until a student asked directly or used empathetic probing. In an earlier live phase, 31 students completed five consultations each, for 155 sessions, and discovered 65.6% of hidden items. The wider validation included 350 blinded expert assessments across five models, but every model used the same architecture, so the study cannot isolate the contribution of that design from prompting, persona writing or language style.
For educators, AI patients could make practice available without booking actors or simulation rooms. Yet the trial reused the same five scenarios for training and assessment, tested immediate outcomes at one Chinese institution and entangled agent with modality: AI was text-based, people used voice. The study did not measure retention, real-patient communication or clinical outcomes. Before replacing human practice, a larger registered multicentre trial would need new assessment cases, matched modalities and longer follow-up.[1]
The distinction that matters
What each phase established—and what it did not
The programme moved from scripted validation to live use and then a pilot comparison.
Phase 1
Seven educators score 350 scripted assessments across five models
Tests fidelity, not learning
Phase 1b
31 students complete 155 AI consultations
65.6% of hidden items discovered
Phase 2
Randomize 61 students; analyse 58
No detectable checklist difference; equivalence remains unproven
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring medical education: compare the finding and its evidence limits.
Did a GPT-4o virtual patient prepare medical students better? This study found no measurable gain
Only 49 of 96 responding students offered the tool used it. Nervousness, preparedness, self-efficacy and perceived competence were no better than in the comparison group, despite positive user ratings.
Read the next reportExplore the full source and evidence record
What this means for people
- Medical students could practise difficult questions on demand and educators could see which safety-critical details were missed. Patients would benefit only if that practice later improves real consultations; this study did not test that transfer.
Global context
AI simulated patients could widen access where actor time and simulation facilities are scarce, but cost, language, privacy, model drift and curriculum fit will vary across institutions. Multilingual, multicentre evidence is needed before treating this pilot as a replacement case.
What the evidence does not yet show
- The pilot analysed 58 participants and was underpowered for small between-group differences or an equivalence claim.
- AI training used text chat while human standardized patients used voice calls, so agent and communication modality cannot be separated.
- Training and assessment reused the same five scenarios, increasing familiarity and potentially compressing differences between arms.
- The single-institution Chinese-language study measured immediate performance, not retention, real-patient communication or clinical outcomes. The global rating scale had poor reliability (ICC 0.15).
- The educational study was not prospectively registered. The authors report no specific funding and no competing interests.
What to watch next
- A prospectively registered trial with at least the estimated confirmatory sample
- New assessment scenarios that students did not rehearse during training
- Matched text and voice conditions that separate modality from AI-versus-human effects
- Retention at four to twelve weeks and transfer to real patient encounters
Living evidence record
Impact record IAI-0ZIXC4V
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
24 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what npj Digital Medicine published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 24 October 2026
Research question, topics and regions
Whether structured AI patient simulations can support medical-student communication training and produce outcomes comparable with human standardized-patient practice
Topics: Medical education · Simulated patients · Communication training · Randomized study · Clinical skills
Regions: China · Asia
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.