Can AI write medical exam questions that are ready to use?
Almost never in this epilepsy benchmark. Seven language models produced 420 questions, but two blinded clinicians judged only two—0.48%—directly usable without revision. Answer accuracy ranged from 61.67% to 98.33%, and weak distractors were the recurring problem.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1Seven models each completed the same 60 prespecified epilepsy tasks, producing 420 single-best-answer questions under a common one-shot prompt and without web search, tools or model-specific tuning.
- 2Only 2 of 420 questions were judged directly usable; 119 needed minor revision and 299 needed substantial revision or were not recommended.
- 3The study measured expert review of generated items, not how real learners performed on them, whether the questions were psychometrically valid or whether using them improved education or care.
Research topic
A controlled seven-model benchmark of epilepsy multiple-choice question generation, evaluated for structure, keyed-answer accuracy and expert-rated educational usability
The answer: AI produced drafts, not a ready-made question bank
A peer-reviewed benchmark published in Frontiers in Medicine found that current language models could usually reproduce the visible structure of an epilepsy multiple-choice question, but almost none of the resulting items were ready for direct educational use. Two blinded clinical reviewers rated only 2 of 420 questions as usable without revision. Another 119, or 28.33%, could be used after minor revision, while 299, or 71.19%, needed substantial revision or were not recommended.
That is the practical result for medical educators. A correctly formatted stem, four options, one keyed answer and a concise explanation can look finished while still containing weak alternatives, imprecise reasoning or a clinical error. The paper supports using a model as an expert-supervised drafting aid. It does not support publishing an automatically generated examination, question bank or continuing-education test without independent clinical and educational review.[1]
The Impact Brief · Free
Follow the evidence in education.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
How the researchers made the comparison fair
The China-based research team created 60 prespecified tasks, divided equally across four areas: 15 on foundational knowledge, 15 clinical cases, 15 treatment decisions and 15 on diagnosis, evaluation and longitudinal management. Each task specified a knowledge point, intended difficulty and cognitive level. GPT-5.6, Gemini-3.5, DeepSeek-V4, Qwen3.8, Doubao-2.1, Claude-5 and Kimi-K3 each received all 60 tasks, giving 420 model-question pairs.
Every model was accessed through an official application programming interface and received the same single-turn prompt. The systems could not search the web or use external tools, and the researchers did not optimise the prompt for individual models. A single standard corrective prompt was allowed when a first response failed the requested structure. This design makes the head-to-head comparison easier to interpret, but it tests one controlled workflow rather than the best result a skilled educator might obtain through retrieval, iteration or fine-tuning.[1]
Accuracy varied, and high accuracy did not guarantee a good assessment item
The researchers checked the keyed answer against an epilepsy-specific reference framework assembled before outcome assessment. It drew on current guidelines and consensus recommendations, International League Against Epilepsy classification papers, established textbooks and selected educational references. Keyed-answer accuracy ranged from 37 of 60 for Doubao-2.1, or 61.67%, to 59 of 60 for Claude-5, or 98.33%. GPT-5.6 and Kimi-K3 each reached 57 of 60, or 95%, while Gemini-3.5 reached 54 of 60, or 90%.
Even the upper results leave room for consequential error when questions are used for training or certification. More importantly, knowing the keyed answer is not the whole job. A useful single-best-answer item needs plausible but clearly inferior distractors, an unambiguous stem and alignment with the intended cognitive level. Median distractor-quality ratings were only two or three on the five-point scale across models. That weakness can make a question trivial, confusing or misleading even when the nominal answer is correct.[1]
Two clinicians scored every item, with substantial agreement
Two clinicians with expertise in epilepsy or clinical neurology and experience in medical education independently rated content rigour, clarity, distractor quality, educational usefulness, cognitive alignment and overall usability. They were blinded to which model produced each item. Intraclass correlation coefficients across domains ranged from 0.794 to 0.859, indicating substantial consistency between their ratings without making the judgments objective or universal.
GPT-5.6 had the highest composite expert score, with a median of 3.75 and an interquartile range of 3.50 to 4.50. Claude-5 followed at 3.67 and Kimi-K3 at 3.50. Yet Claude-5 had the highest keyed-answer accuracy while GPT-5.6 led the composite rating and produced the only two directly usable questions. The split illustrates why a single leaderboard number cannot substitute for examining the dimension that matters in a particular educational setting.[1]
What teachers and clinical training teams should do with this evidence
An educator could use a model to create a first draft, then require a named subject expert to verify the source, keyed answer, explanation, difficulty and every distractor. A second review should check whether the question rewards clinically relevant reasoning rather than word patterns or test-taking tricks. Institutions also need version control: model behaviour changes, so an item approved today should not silently be regenerated later and treated as equivalent.
The study does not justify replacing trained item writers. Nor does it show that automated drafting saves time after review and revision are counted. A model that produces many plausible drafts may help a small faculty widen coverage, but a high rejection rate can move work rather than remove it. For examinations affecting progression, licensing or patient-facing competence, local governance should require human sign-off, a traceable evidence source and ordinary psychometric testing with learners.[1]
The limits are as important as the model ranking
Each model generated only one response for each of 60 tasks. The design therefore cannot estimate run-to-run variability or test-retest reliability. The ranking is time-sensitive because model versions and serving configurations change. Two experts supplied the subjective ratings, and the threshold for usability was an expert Likert score rather than observed learner performance. Pooling correlations across models may also reflect model-specific writing styles.
No students answered the questions, so the study cannot establish item difficulty, discrimination, fairness, retention or effects on learning. It did not test whether flawed questions would change clinical judgments or patient outcomes. The work was supported by Chinese public and university research programmes, and the authors declared no commercial or financial conflicts. Those disclosures reduce one concern but do not resolve the design limits.[1]
What would change the assessment
Evidence would strengthen if several independent generations per task produced stable results, if larger and more diverse clinical panels reached similar judgments and if educators measured the total time from prompt to approved item. A stronger validation would give reviewed questions to learners, estimate difficulty and discrimination, compare them with established human-written items and test whether any subgroup faced systematic ambiguity or disadvantage.
For now the defensible conclusion is narrow: structured prompting can make the surface form of specialist medical questions easy to reproduce, but educational readiness remains scarce. The most useful deployment is a transparent drafting workflow with expert accountability, not autonomous assessment production. Better model scores would change that view only if they were accompanied by repeatability, psychometric validity and evidence that review effort and error risk genuinely fall.[1]
What this means for people
- Medical educators can treat model output as a draft queue, but every keyed answer, explanation and distractor still needs qualified review.
- Learners should not be evaluated with unvalidated generated items simply because the questions are well formatted or the model scores highly on other benchmarks.
- Clinical organisations need traceable sources, version control and named accountability before AI-generated assessments affect training or progression.
Global context
The authors work across hospitals and universities in China, while the tested models include systems developed in several countries. The benchmark is internationally relevant but does not establish that the same questions, language or clinical guidance transfer across curricula, health systems and regulatory settings. Local reference standards and learner validation remain necessary.
What the evidence does not yet show
- Only one generation was collected for each model-task pair, preventing estimates of stochastic variability and repeatability.
- Two expert raters judged usability; no learners completed the items and no psychometric validity was measured.
- The single-turn, no-tools protocol may understate the performance of carefully grounded or iterative workflows.
- Model versions and API configurations change quickly, so the ranking may not persist.
- The study did not measure authoring time, total review cost, learning outcomes or patient outcomes.
What to watch next
- Replications using repeated generations and independently documented model versions
- Learner-based measures of item difficulty, discrimination, fairness and learning effect
- Total staff time and error rates for AI-assisted versus conventional item writing
- Whether retrieval-grounded workflows improve distractors without introducing citation or context errors
Living evidence record
Impact record IAI-16HEDV1
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
13 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Medicine published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 13 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Education
Did a global AI course prove clinicians learned more?
Not yet. In a peer-reviewed evaluation, 63 completers from 12 countries reported large gains in confidence after an eight-week course, but there was no objective knowledge test, control group, retention measure or patient outcome.
7 min · 1 source
Education
Can AI grade medical students without hiding mistakes?
A prospective US deployment processed 72,907 OSCE rubric scores for 222 students and sharply reduced human scoring passes, but validation focused on the low-scoring tail and physician adjudicators could see which score came from AI.
7 min · 1 source
Education
Can AI make clinical skills exams fairer?
An Oxford study found that an examiner-aware AI second marker improved agreement with a panel-derived reference across 442 ratings from 120 students, especially for borderline fails. That is an audit result in virtual-reality exams—not proof of fairer real-world assessment or a licence for automated grading.
9 min · 1 source
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.