Can AI write safe clinical teaching cases without expert review?
A peer-reviewed review of 72 studies found that AI-generated cases were often plausible and clear, but most evaluations relied on small expert panels, only 25 studies involved students and recurring errors, shallow realism and bias kept human validation essential.
The short version
- What happened
- The review screened 1,280 unique records and included 72 studies; 59 concerned medical education, while nursing, pharmacy and other health professions were much less represented.
- Who it affects
- Students may receive larger and more varied case libraries, but flawed prevalence, stereotypes or weak item construction can teach the wrong clinical pattern.
- What remains unproven
- The review did not formally appraise the methodological quality or risk of bias of the 72 included studies.
Keep in mind
The review did not formally appraise the methodological quality or risk of bias of the 72 included studies.
Study designs, models, prompts, specialties, sample sizes and outcomes were too heterogeneous for a pooled effectiveness estimate.
The search ended on 10 May 2026 and covered only English-language work in seven databases plus citation searching.
Most evaluations relied on expert panels, often ten or fewer people; student and performance evidence was less common.
Many studies generated or evaluated small output sets, which may not expose uncommon but consequential errors.
Text-only case materials were included; images, audio and video scenarios were outside the review.
The evidence was concentrated in medicine, Asia and North America, limiting transfer to other professions and regions.
Source analysis · 1 linked source. Primary analysed source date: 3 November 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 9 min The evidence, methods, implications and limitations
The direct answer: no—the evidence supports assisted drafting, not unsupervised release
Large language models can produce useful first drafts of clinical cases, multiple-choice questions and simulation scenarios. Across the 72 studies in this scoping review, experts often judged outputs clinically plausible, clearly written and aligned with an educational aim, especially after refinement. The same literature repeatedly reported factual inaccuracies, superficial clinical realism, item-construction defects, omissions, bias and inconsistent performance between models. That combination supports a workflow in which AI accelerates drafting while educators retain responsibility for validation, adaptation and final approval.
The review does not establish that one model, prompt or workflow reliably produces safe material without review. It found no model that consistently outperformed the others, and the underlying studies used different tasks, specialties, prompts, scoring criteria and sample sizes. Many generated only a handful of outputs. A favourable average rating on a small batch cannot guarantee that the next case is accurate, that an exam item discriminates appropriately, or that a synthetic patient reflects the population learners will encounter.[1]
How the reviewers assembled the evidence
The French research team followed Joanna Briggs Institute scoping-review methods and PRISMA-ScR reporting guidance. Seven databases—PubMed, Embase, Scopus, Web of Science, CINAHL, ERIC and IEEE Xplore—were searched on 10 May 2026 for English-language research published from 2017 onward. Citation searching added another eligible study. From 2,777 records, the team removed 1,497 duplicates, screened 1,280 unique titles and abstracts, retrieved 110 full texts and included 72 studies.
Two reviewers independently screened records at each stage and resolved disagreements through discussion, with a third reviewer involved twice. One researcher charted each study and another independently verified the extraction. The team recorded profession, learner level, model, prompting, source material, refinement, evaluation method, output count, findings and limitations. The complete charting table and search materials are public. This is a systematic map of what has been studied, not a meta-analysis of a common outcome.[1]
The distinction that matters
What the review found across 72 studies
The evidence is broad enough to show recurring patterns, but too heterogeneous for a pooled effectiveness claim.
Generated
Cases, MCQs and simulation scenarios
Often plausible after refinement; errors and shallow realism persisted
Expert review
Used in 56 of 72 studies
Usually ten or fewer experts
Student evidence
25 studies involved students
Performance outcomes were much rarer than ratings or surveys
Decision
Use AI for drafting within a governed workflow
Clinical and educational experts remain accountable for release
What the 72 studies actually covered
Medical education dominated: 59 of 72 studies, or 82%. Seven covered nursing and two pharmacy; five included other fields such as dentistry, occupational therapy, physiotherapy, respiratory therapy or paramedicine. Forty-two studies involved undergraduate education, eight postgraduate training and one continuing education, while 22 did not specify a learner level. Case-based multiple-choice questions were the most common product at 28 studies, followed by 22 studies of clinical cases or vignettes and 18 of simulation scenarios.
Geographic coverage was uneven. Thirty-five studies involved Asia and 27 North America, compared with ten Europe, three Africa and one South America; none came from Oceania. Country totals overlap because some studies covered more than one place. The United States accounted for 19 studies and Türkiye ten. The evidence therefore reflects particular education systems, languages and clinical contexts. A case judged realistic in one setting may use the wrong prevalence, terminology, referral pathway or scope of practice elsewhere.[1]
Most systems used familiar models and relatively simple prompts
The GPT-4 family appeared in 40 studies, GPT-3.5 in 21, Bard or Gemini in 13, Claude in seven and DeepSeek in five; studies could use more than one model. Fifty-nine studies relied on prompt-only generation rather than supplying a textbook, guideline or prior cases. Sixty-two published their prompts, six reported only extracts and four did not publish them. Instructional and contextual prompting appeared in nearly all studies, while only 15 used prompt chaining and six used one- or few-shot examples.
Output volumes were often small: 29 studies generated ten or fewer items and 25 generated between 11 and 100. Only 14 produced more than 100, and four did not specify the number. Twenty-five studies refined prompts, 16 refined outputs and five asked the model to self-refine. These choices matter because a polished example selected after iteration may not represent the system's routine error rate. Reproducible evaluation needs the complete prompt, model version, generation settings, all attempted outputs and transparent exclusion rules.[1]
Evaluation was common, but usually narrow
Fifty-six studies used expert review, yet 42 of the 72 studies involved ten or fewer experts. Six did not state the expert-panel size. Student assessment appeared in 25 studies, most commonly with 11 to 100 learners, while ten studies were exploratory and had no formal evaluation. Among evaluated studies, clinical accuracy, educational alignment and realism were common criteria, but definitions varied from free-text judgments to rating scales. That variation makes a score from one paper difficult to compare with another.
Measures closer to learning or assessment quality were less common. Eleven studies reported item discrimination and ten difficulty indices; four assessed distractor effectiveness and four reliability. Only three tracked learner performance during an intervention or before and after it, and four reported correct-response rates. Student surveys more often measured perceptions such as clarity, engagement or usefulness. Satisfaction can identify usability problems, but it does not prove that a case teaches the right concept, improves reasoning or performs safely in a high-stakes examination.[1]
The recurring risks go beyond hallucinated facts
Factual errors and omissions were one problem, but the review also found shallow or stereotyped cases, weak alignment with objectives, implausible distractors, inconsistent difficulty and suboptimal psychometric performance. Bias was formally analysed in relatively few studies. Where researchers looked, some generated cases overrepresented stereotyped groups, diverged from real disease prevalence or underrepresented minority, Indigenous and LGBTQIA+ people and social determinants of health. An apparently accurate case bank can still train learners on a distorted picture of who develops disease and where care occurs.
No model won consistently. Different comparisons favoured different systems for correctness, narrative detail, diversity, instructional design or local-language realism, and every system still needed expert correction in at least some settings. That instability argues against procurement based on a single leaderboard or demo. Institutions need a validation standard tied to their curriculum and patients: source checking, clinical review, bias and prevalence checks, item-writing rules, learner testing where appropriate, version control and a documented way to withdraw flawed material.[1]
Evidence check and what would change the assessment
The review has strong transparency features: a reproducible search across seven databases, dual screening, verified data extraction, a PRISMA flow diagram and public charting data. It was supported by a French government France 2030 grant, and the authors declared no conflicts of interest. They disclosed using ChatGPT-5 under human supervision for proofreading, summarising and reformatting, with the authors retaining responsibility. The central evidence limit is that a scoping review maps literature but does not formally rate risk of bias or combine studies into a dependable effect estimate.
The search ended on 10 May 2026 and excluded non-English work, grey literature, multimodal cases and databases outside the chosen seven, so a fast-moving field will already have changed. Confidence in educational effectiveness would rise with preregistered comparative studies that publish every generated item, use independent blinded assessors, report correction and rejection rates, test diverse patients and settings, and measure learner performance over time. If a workflow repeatedly produces accurate, calibrated and inclusive material with fewer faculty hours and no loss of learning, AI could move from optional drafting aid toward a validated production tool. Until then, expert oversight is a safety control, not an inefficiency to remove.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring medical education: compare the finding and its evidence limits.
Can AI write medical exam questions that are ready to use?
Almost never in this epilepsy benchmark. Seven language models produced 420 questions, but two blinded clinicians judged only two—0.48%—directly usable without revision. Answer accuracy ranged from 61.67% to 98.33%, and weak distractors were the recurring problem.
Read the next reportExplore the full source and evidence record
What this means for people
- Students may receive larger and more varied case libraries, but flawed prevalence, stereotypes or weak item construction can teach the wrong clinical pattern.
- Educators can save drafting time only if review, correction, versioning and learner testing are planned rather than left as invisible extra work.
- Patients are affected indirectly because case materials shape the examples through which future clinicians learn diagnosis, communication and judgement.
Global context
The included studies came from several regions, but medicine, Asia and North America dominated and no study came from Oceania. Health-professions education depends on local disease prevalence, clinical guidelines, language, professional roles and patient expectations. A model can produce fluent material that is locally unsafe, so international reuse requires more than translation: cases need clinical, cultural and curricular validation in the setting where learners will use them.
What the evidence does not yet show
- The review did not formally appraise the methodological quality or risk of bias of the 72 included studies.
- Study designs, models, prompts, specialties, sample sizes and outcomes were too heterogeneous for a pooled effectiveness estimate.
- The search ended on 10 May 2026 and covered only English-language work in seven databases plus citation searching.
- Most evaluations relied on expert panels, often ten or fewer people; student and performance evidence was less common.
- Many studies generated or evaluated small output sets, which may not expose uncommon but consequential errors.
- Text-only case materials were included; images, audio and video scenarios were outside the review.
- The evidence was concentrated in medicine, Asia and North America, limiting transfer to other professions and regions.
What to watch next
- Preregistered comparisons that publish every generated output and correction
- Independent tests of learner performance and retention rather than satisfaction alone
- Bias and prevalence audits across demographic groups and clinical settings
- Faculty-time, cost and rejection-rate measurements for governed production workflows
- Evidence from nursing, pharmacy, allied health, Africa, South America and Oceania
Living evidence record
Impact record IAI-0DMA918
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
4 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what JMIR Medical Education published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 4 November 2026
Research question, topics and regions
How large language models are being used and evaluated when generating case-based materials for health-professions education
Topics: Medical education · Clinical cases · Assessment quality · Expert review · Generative AI
Regions: Global · Asia · North America · Europe · Africa · South America
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.