AI lecture summaries raised quiz scores by 0.28 points—did students learn more?
In an unreviewed year-long study, 120 US medical students were randomized to receive AI-generated lecture summaries or take the same quizzes without them. Scores leaned towards the summary group, but the individual-level difference was small and statistically uncertain.
The short version
- What happened
- The single-school study enrolled 137 third-year medical students; 120 submitted at least one analysed quiz, producing 317 responses across 11 neurology clerkship cohorts.
- Who it affects
- Medical students may value a concise, checked summary when clinical rotations leave little study time. This study cannot tell a school that summaries improve durable learning, and it gives no evidence that an unreviewed AI output should replace faculty teaching or source material.
- What remains unproven
- This is an unreviewed preprint from one US medical school and one neurology clerkship.
Keep in mind
This is an unreviewed preprint from one US medical school and one neurology clerkship.
The principal student-level effect was small and statistically uncertain; the cohort-direction test was one-sided and its two-sided value was just above 0.05.
Repeated quiz responses were clustered within students, the quizzes had a strong ceiling effect and no delayed-retention outcome was measured.
Survey results were self-reported, used varying denominators and offered no neutral response option.
Two authors founded a medical-education product company, although the paper states it did not fund or participate in the study.
Source analysis · 1 linked source. Primary analysed source date: 30 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 5 min The evidence, methods, implications and limitations
The clearest result is a small, uncertain score difference
A year-long study at one US medical school found that third-year neurology students offered AI-generated lecture summaries scored, on average, 8.14 out of 10 on weekly quizzes. Students sent the same quiz without the summary averaged 7.86. That 0.28-point difference favoured the summary group, but its 95% confidence interval ran from 0.23 points lower to 0.80 higher, and the individual-level comparison was not statistically significant (P = 0.28). The study therefore does not establish that the summaries improved learning, despite a consistent-looking pattern across rotations.
The researchers enrolled 137 students across 11 sequential clerkship cohorts from January 2025 to January 2026. Seventeen submitted no quiz; 120 students contributed 317 analysed responses, with 62 assigned to receive summaries and 58 to receive no summary. Assignment happened within each cohort before the first quiz. Both groups answered the same ten questions, while the intervention group received a one-page summary generated from that week’s recorded Neurology Grand Rounds lecture. Participation was voluntary, scores did not count towards grades, and completing a quiz counted towards attendance.[1]
A favourable cohort pattern does not resolve how much students learned
The summary arm scored higher in nine of 11 cohorts. A pre-specified one-sided randomization test on that direction produced P = 0.026; the two-sided value was 0.053. But counting direction gives a small cohort the same weight as a large one. When the analysis weighted differences by response count, the advantage was 0.22 points and did not reach significance (one-sided P = 0.21). Repeated quizzes also clustered within students: the authors estimated that 317 responses provided precision closer to 195 independent observations. Their mixed-effects estimate remained 0.27 points, with a confidence interval crossing zero.
The quiz design made a difference harder to interpret. Nearly two-thirds of responses were already eight or above, 17% were perfect scores, and most questions tested recall rather than clinical application. Sixty-three students answered the end-of-cohort survey, with changing denominators by item and no neutral response option. Among those reporting receipt of a summary, 26 of 33 said it made them more likely to review the lecture and 14 of 17 said it reduced the time needed to understand it. Those perceptions may matter to busy trainees, but they are self-reports rather than evidence of durable knowledge or better patient care.[1]
The distinction that matters
What the study’s three signals actually show
All figures come from the same unreviewed, single-school intervention; they answer different questions and should not be combined as if they were one result.
Student averages
62 summary vs 58 no-summary students
+0.28 of 10; 95% CI -0.23 to +0.80; P = 0.28
Cohort direction
11 rotations compared by which arm scored higher
9 of 11 favoured summaries; two-sided P = 0.053
Student reports
63 survey respondents, with item-specific denominators
26/33 reviewed more; 14/17 reported faster understanding
Calibration was careful, but the evidence remains preliminary
Before the classroom phase, the team compared six prompts for summaries and six for questions across two models, generating 2,692 rubric annotations. Claude 3.5 Sonnet with one prompt produced the best summaries, while GPT-4 with another produced the best questions; no model led both tasks. Outputs were checked against lecture transcripts and reviewed for relevance before release. This is a useful design lesson: institutions should test prompts and content types rather than assume a preferred model works equally well for every educational task.
The paper is a medRxiv preprint and has not been peer reviewed. It comes from one school, one specialty and short quizzes delivered near the lectures; it did not test delayed recall, clinical reasoning, examination results or patient outcomes. Detecting the observed student-level effect with conventional power would have required roughly 394 students per arm, far more than the 62 and 58 analysed. Two authors founded Dendro Education, which develops medical-education products; the paper says the company did not fund or participate in the study, and the study received no external funding. A stronger assessment would need an independently run, preregistered, multi-school trial using harder application questions and delayed testing, with a fixed primary outcome and enough students to estimate both benefit and harm precisely.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring medical education: compare the finding and its evidence limits.
Can AI simulated patients prepare medical students for psychiatry?
Sixteen UK medical students and nine psychiatry teachers saw flexible, low-pressure practice in an AI simulated patient—but after only a ten-minute trial, they also flagged emotional realism, bias, safeguarding and over-reliance.
Read the next reportExplore the full source and evidence record
What this means for people
- Medical students may value a concise, checked summary when clinical rotations leave little study time. This study cannot tell a school that summaries improve durable learning, and it gives no evidence that an unreviewed AI output should replace faculty teaching or source material.
Global context
Medical schools are testing AI-generated learning materials faster than robust outcome evidence is accumulating. The practical question is not whether a model can produce a plausible summary, but whether a calibrated, faculty-governed resource improves lasting knowledge across institutions without introducing errors or widening access gaps.
What the evidence does not yet show
- This is an unreviewed preprint from one US medical school and one neurology clerkship.
- The principal student-level effect was small and statistically uncertain; the cohort-direction test was one-sided and its two-sided value was just above 0.05.
- Repeated quiz responses were clustered within students, the quizzes had a strong ceiling effect and no delayed-retention outcome was measured.
- Survey results were self-reported, used varying denominators and offered no neutral response option.
- Two authors founded a medical-education product company, although the paper states it did not fund or participate in the study.
What to watch next
- Peer review and protocol registration details
- Independent replication at several medical schools
- Delayed recall and clinical-reasoning questions
- A pre-specified two-sided primary outcome
- Error rates in released summaries and quizzes
- Results by prior attainment and student access needs
Living evidence record
Impact record IAI-06JMW0Q
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
1 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what medRxiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 1 November 2026
Research question, topics and regions
AI-generated medical education materials
Topics: Medical education · Learning · Lecture summaries · Randomized study · Large language models · Preprint
Regions: North America · United States
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.