What has AI medical-education research actually measured?
A review found 23 of 27 student-facing studies measured reactions and 16 measured learning, but none tracked sustained behaviour. The map is useful for curriculum design, not proof that generative AI improves future clinical care.
The short version
3 key points
- 1.The review included 27 student-facing studies: 23 measured learner reactions, 16 measured knowledge, skills or attitudes, and none measured sustained behavioural change.
- 2.Only three studies were classed as explicitly grounded in educational theory under the authors' adapted framework; just two examined mechanisms explaining how learning changed.
- 3.This was a scoping review without formal quality appraisal, and its September 2025 search cutoff means it cannot establish effectiveness or represent the newest year of a fast-moving field.
The answer: mostly reactions and short-term learning
Research on generative AI in undergraduate medical education has mostly measured whether students liked an intervention and whether knowledge, skills or attitudes changed soon afterwards. In this scoping review, 23 of 27 included studies measured learner reactions and 16 measured learning outcomes; categories were not mutually exclusive. None met the authors' definition of sustained behavioural change. Three were judged to use educational theory explicitly enough to qualify for the review's adapted system-level category, and only two explored mechanisms in enough depth to ask how the tool changed learning.
That distribution matters because positive satisfaction or a better score after a single exercise does not show that a future clinician will reason, communicate or act more safely in practice. The review does not conclude that generative AI is ineffective. It shows that the evidence chain is short: many studies stop before repeated behaviour, clinical transfer or patient consequences can be assessed. For medical schools, the practical message is to decide what durable capability an intervention is supposed to build before choosing a chatbot, simulated patient or automated-feedback system.[1]
How the evidence map was assembled
The authors searched Ovid MEDLINE and Embase, PubMed, Web of Science and Scopus between 22 and 24 September 2025, without language or date restrictions. Search terms covered generative AI and large language models alongside medical students or medical education. The search produced 2,316 records; after deduplication, 923 were screened. The team identified 269 potentially eligible records, sought 45 full texts after tightening the criteria to student-facing interventions, and included 27 studies in the final map.
Screening was done independently in assigned pairs, with team meetings and a joint test on 40 records. The reviewers did not report a numerical agreement statistic. Data were extracted in pairs using a piloted spreadsheet and finalized by consensus. Twenty-one of the 27 included studies were published in 2025. Six came from the United States and five from China, with the remainder spread across ten other countries. Twenty-three used quantitative designs. The review therefore maps a young, geographically varied but methodologically uneven literature rather than pooling a stable body of trials.[1]
What students were actually asked to use
Thirteen studies used generative AI for simulated-patient experiences, ten for knowledge or decision support and four for educational-content generation. Twenty-five used ChatGPT either alone or inside a wider platform or workflow. Other tools included DeepSeek, Kimi Chat, Stable Diffusion, DALL-E 3, Leonardo.ai and Whisper-3. Applications ranged from history-taking and clinical-reasoning practice to mentoring, question generation, diagnostic images and case videos.
The intervention count should not be mistaken for evidence that every approach worked. Level-one studies commonly reported satisfaction, engagement or perceived usefulness. Level-two studies addressed knowledge, skill or attitude, but often through one-off assessments, simulations or pre-post tests. Those are legitimate early-stage outcomes, especially in preclinical education, yet they do not establish retention or transfer. The review found no study showing the same behaviour repeatedly over time under its broadened definition of sustained practice.[1]
A larger literature often left the learner out
The screening process also exposed two much larger groups that were excluded from the 27-study analysis. One hundred and nineteen reports discussed generative AI in medical education without a student-facing intervention, often concentrating on content generation, awareness or attitudes. Another 110 tested how well a model performed on examinations or licensing-style questions without involving students. Those papers may answer technical or safety questions, but they cannot show what happens to a learner who studies with the system.
This distinction is especially useful for university leaders evaluating claims about an 'AI medical tutor.' A model's examination score is not a student outcome, and a generated lesson's apparent quality is not evidence of learning. Procurement and curriculum decisions need measures tied to people: what students understand, how they check uncertain output, whether gains persist, whether performance transfers to new cases, and whether the intervention changes access or workload across different groups.[1]
What the review's framework adds—and changes
The researchers organized outcomes with an adapted Kirkpatrick framework. Its conventional levels are reaction, learning, behaviour and results. They broadened behaviour to mean practice demonstrated consistently over time rather than a single simulation. They also broadened the highest level to include either clinical or patient outcomes or theoretically grounded educational research. That choice made explicit theory count as a system-level contribution even without measured care outcomes; readers should not interpret the three level-four studies as three demonstrations of patient benefit.
The authors then proposed a Technology-Integrated Educational Design, or TIED, road map. It starts with whether a tool is trustworthy, appropriate and responsive to a learner need; connects reactions and learning to sustained practice; and asks what mechanism produced an outcome, for whom and under which conditions. This is a design proposal derived from the evidence map, not a validated intervention. Its value is as a checklist against technology-first projects whose evaluation ends at enthusiasm or a short test.[1]
What educators and students can do now
A medical school can use the review to write a measurement plan before rollout. For a simulated-patient chatbot, for example, the immediate questions might include realism, accessibility and feedback quality. The learning question should specify the targeted history-taking or reasoning skill and an appropriate comparator. Follow-up should test retention and performance on unfamiliar cases. If the intended benefit is clinical transfer, the institution needs supervised longitudinal observation and safeguards that separate educational practice from unsupervised patient care.
Students should also be treated as participants in design rather than endpoints for satisfaction surveys. That means documenting which tasks the tool supports, what happens when it is wrong, how personal or patient data are protected, what alternatives exist for students who cannot or do not wish to use it, and how teachers will respond to over-reliance. A positive reaction can still be useful, but it should sit alongside error checking, cognitive workload, equity and demonstrated learning rather than substitute for them.[1]
Limits of the review
This was a scoping review, so the authors did not formally assess risk of bias or study quality and did not calculate a pooled effect. The protocol was neither published nor registered, and the team refined parts of the eligibility and analytical framework as it learned more about the literature. Those choices are described, but they increase the importance of replication. Agreement statistics for screening were not recorded, and the expanded meanings assigned to the upper Kirkpatrick levels are interpretive rather than standard.
The search ended in September 2025, a year before publication, so later interventions and newer models are absent. The review covered undergraduate medical students, not residents, continuing professional development, nursing or other health professions. Its result is therefore a map of what a defined literature had measured by the cutoff—not a verdict on all medical education or on current products. No financial support or conflicts were declared; the authors disclosed GPT-5.3 use for proofreading.[1]
What would change the assessment
The assessment would strengthen if an updated registered review formally appraised study quality and separated randomized comparisons, observational studies and design reports. Better primary studies would preregister outcomes, use appropriate non-AI or standard-teaching comparators, report denominators and attrition, and test retention and transfer rather than only immediate performance. Multi-institution studies should examine whether results vary by language, prior knowledge, disability, access and clinical context.
Most importantly, the field needs a longer chain from intervention to practice. That does not require exposing patients to unproven systems. Researchers can begin with repeated simulated cases, delayed testing, structured observation and blinded assessment, then progress to supervised clinical settings when justified. Until that evidence exists, the defensible conclusion is narrower than the hype: generative AI has produced many student-facing experiments, but the published evidence has rarely followed students far enough to know whether short-term enthusiasm becomes durable professional competence.[1]
What this means for people
- Medical students are often measured as satisfied users rather than followed as developing clinicians, leaving retention and transfer uncertain.
- Teachers can use the review to connect a specific learner need to a comparator, delayed assessment and an explicit mechanism before deploying a tool.
- Patients should not be told that educational chatbot studies prove safer care; none of the 27 included studies demonstrated sustained behaviour under the review's definition.
Global context
The included studies came from 12 countries, led by the United States and China, but the review was led by institutions in Singapore and China and searched literature only through September 2025. Its evidence map is internationally relevant without being a prevalence estimate or a current inventory of every medical school.
What the evidence does not yet show
- Scoping review without formal study-quality or risk-of-bias appraisal
- Unregistered protocol and eligibility refinements made during screening
- September 2025 search cutoff despite publication in October 2026
- No pooled effect size and substantial heterogeneity of tools, interventions and outcomes
- Adapted Kirkpatrick level 4 includes theoretical grounding, so it is not equivalent to demonstrated patient or system benefit
- Undergraduate medical education only, excluding residents and other health professions
What to watch next
- Registered updates covering studies published after September 2025 with formal quality appraisal
- Delayed and repeated assessments showing retention and transfer to unfamiliar cases
- Comparative studies that measure equity, workload, error checking and cognitive offloading
- Supervised longitudinal evidence linking educational use to clinical behaviour without placing patients at avoidable risk
Living evidence record
Impact record IAI-0COZOJW
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
15 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Journal of Medical Internet Research published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 15 October 2026
Research question, topics and regions
What outcomes student-facing generative-AI interventions in undergraduate medical education have measured, and where the evidence remains missing
Topics: Medical education · Generative AI · Learning evidence · Curriculum design · Students · Evidence synthesis
Regions: Global · Asia · Singapore · China
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
UNESCO
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Education & human agency · Background explainer
Published 2025-01-23 · 8 min 30 sec
Teacher to Teacher: Is AI reshaping education for the better?
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Education
Does generative AI improve self-regulated learning?
A meta-analysis of 17 comparative studies found a positive average effect, but results varied sharply: heterogeneity reached 94.85% and the prediction interval included harm as well as benefit. Almost all evidence came from higher education.
7 min · 1 source
Education
Does AI help musicians practise independently?
A review of 16 heterogeneous studies maps how recording tools, error detectors and generative assistants may support practice. Short-term findings are encouraging, but no study tested whether gains persist after the support is removed.
6 min · 1 source
Education
Does AI help trainee teachers reflect better after metacognitive training?
In a randomized 60-person experiment, a configured Socratic AI improved written reflection only when paired with a 20-minute metacognitive lesson. The immediate, single-university result does not show that generic chatbot access improves teaching practice.
7 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.