Can GPT-4 write a valid chemistry test? The items mostly fitted—but their intended difficulty levels did not
Researchers generated 27 German electrochemistry questions and tested them with 95 upper-secondary students. Most items met many writing criteria and largely fitted a Rasch model, but language, distractors and competence-level alignment exposed why teachers still need expert and psychometric checks.
The short version
- What happened
- GPT-4 generated 27 electrochemistry items—nine for each of three intended competence levels—and researchers administered the unchanged test to 95 upper-secondary students.
- Who it affects
- Teachers can use generated questions as drafts but retain responsibility for content and score meaning.
- What remains unproven
- One 27-item German electrochemistry test, one GPT-4 process and 95 students.
Keep in mind
One 27-item German electrochemistry test, one GPT-4 process and 95 students.
No human-written comparison instrument.
Person reliability was 0.66 and level alignment unstable.
Researchers manually checked prompt compliance and textbook correctness.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 7 min The evidence, methods, implications and limitations
The direct answer: GPT-4 produced a workable draft, not a ready-to-trust assessment
A peer-reviewed Frontiers in Education study published on 4 November reports a mixed result for AI-authored school assessment. GPT-4 generated 27 German-language multiple-choice questions on electrochemistry, divided evenly across three curriculum-defined competence levels. Ninety-five upper-secondary students completed the unchanged test after an electrochemistry unit. Many questions met established writing criteria, and most behaved acceptably under Rasch-model checks. Yet the claimed progression from recall to explanation and complex transfer did not reliably match empirical difficulty. Several distractors attracted almost nobody, and measurement precision for individuals was limited.
That combination matters more than a simple pass rate. A question can be grammatically plausible and fit a statistical model while failing to measure the intended competence. Teachers may use AI to reduce blank-page work, but a consequential test needs evidence that scores mean what the test claims. This single-topic study supports supervised drafting followed by expert review, student piloting and psychometric analysis. It does not support copying generated questions into an examination or assuming that a prompt label such as 'level III' creates a genuinely higher-order task.[1]
How the test was generated
Researchers pasted definitions and competence expectations from the North Rhine-Westphalia curriculum into an iteratively refined prompt. GPT-4 first produced nine items, then generation continued until there were nine at each of three levels: recall, explanation in a new context and complex transfer. Each item had four options with one correct answer. The prompt requested relevant background in the stem and language suited to the target group. Researchers manually checked format and topic and checked correctness against a chemistry textbook.
The questions were not edited before the study. That makes the evaluation more informative about generated output, but it is not how a careful teacher would normally deploy a test. The prompt was refined over several iterations until outputs met the requested structure, so this was neither one-shot generation nor an automatic pipeline. Human choices shaped the curriculum material, instructions, stopping point, correctness check and final set. The study evaluates an AI-assisted development process, not isolated model capability independent of expert scaffolding.[1]
The distinction that matters
A generated question must pass more than one test
The study evaluated wording, student-response behaviour and curriculum-level claims separately.
Generation
27 electrochemistry items
Nine requested at each competence level
Writing review
22 criteria, two coders
87.71% average with language weaknesses
Student test
95 analysed pupils
Most items fitted; person reliability 0.66
Distractors
Wrong-answer patterns
Twelve distractors in ten items drew under 5%
Level claims
Labels versus empirical groups
Only seven items were bootstrap-stable
Writing quality was broadly strong, with recurring weaknesses
Two authors independently coded all 27 items against 22 binary criteria, with Cohen's kappa of 0.93 before disagreements were resolved. The average pass rate was 87.71%, but that headline hides large differences. All items met criteria for contextualisation, curriculum content and specific competencies, avoidance of bias and several option-format rules. Only 33.33% reduced specialised language sufficiently, and 48.15% balanced answer-option length. Three items had spelling errors and six had syntax errors. Plausible distractors were present in 77.78% of items.
Aggregate quality scores can therefore mislead. Technical chemistry necessarily uses specialist terms, but unnecessary jargon can turn a subject test into a reading barrier. Unequal option length can cue test-wise students. Syntax errors may alter meaning, and an implausible distractor makes an item easier without measuring intended knowledge. A teacher should use a structured checklist rather than ask whether a question merely looks polished. Independent review is important because fluent wording can hide a cue, ambiguity or content assumption that the generator and first reviewer both overlook.[1]
The Rasch results were promising, not a certificate of validity
The test went to 98 students; three were excluded for missing responses, leaving 95. One item underfit and one overfit the Rasch model; remaining fit statistics were 0.79 to 1.28 under the authors' criteria. Item reliability was 0.91, indicating a stable relative hierarchy in this sample and model. Person reliability was 0.66, limiting precision for distinguishing individual students. Measurement information was concentrated around the middle of the observed ability range rather than equally strong at the extremes.
Rasch fit is not proof that every question is correct, fair or educationally useful. It asks whether response patterns conform reasonably to model expectations. It does not validate content by itself, establish that scores predict learning or show equivalence to an established test. The sample was modest and from one target context, and no age or gender variables were collected. The identical item order means order effects were not tested. Replication across schools, cohorts and item orders is needed before treating parameters as stable beyond this administration.[1]
Weak distractors and unstable level labels are the practical warning
Twelve distractors across ten questions were chosen by fewer than 5% of students, suggesting those wrong answers contributed little discrimination. Of 77 estimable distractors, 61—79.22%—had the expected negative relationship with ability, while 16 did not. A distractor that attracts almost nobody may need replacement; one that behaves unexpectedly can reveal ambiguity, misconception or scoring trouble. Teachers cannot judge this from wording alone. It becomes visible when representative students answer and response patterns are analysed.
The central weakness concerned competence levels. Intended curriculum labels showed weak agreement with empirical groups and were sensitive to anchoring rules. Only seven of 27 items retained their classification in at least 70% of bootstrap samples. Median adjusted agreement was 43.48%, with weighted kappa 0.314. Difficulty and cognitive demand are not identical: a complex task can be easy for a well-taught class, while a recall item can be hard because of obscure wording. A model's level label is a hypothesis for review, not metadata a school should trust automatically.[1]
What changes for teachers—and what would change the assessment
Teachers can use a model to propose curriculum-anchored drafts, then check content, language, cues, accessibility and distractor plausibility before piloting. AI may stimulate variants, but the final test needs a named professional owner. Generated items also need version records covering the prompt, model, curriculum input, human edits, source check and pilot statistics. Pupils can be harmed by ambiguous wording and hidden reading load when scores influence feedback, placement or confidence. High-stakes use demands stronger evidence than this study provides.
There was no matched human-written test, so the study cannot say whether GPT-4 was better, faster or cheaper than expert authorship. Confidence would rise with a preregistered multi-school comparison at matched curriculum coverage, using blinded expert review and repeated samples. Outcomes should include errors, accessibility, teacher time, cost, reliability and subgroup functioning. It would weaken if generated items repeatedly misclassified demand or disadvantaged particular pupils. This is consequential research but not Breaking: it offers assessment evidence, not an urgent school instruction or active safety event.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring assessment: compare the finding and its evidence limits.
Do better prompts make AI fairer for students? Outputs improved, but learning and equity were not tested
A structured prompt raised expert and model ratings across 800 outputs. A separate student survey linked prompt confidence with perceived gains, but neither stage measured learning or unequal outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- Teachers can use generated questions as drafts but retain responsibility for content and score meaning.
- Pupils can be mismeasured when language, cues or level labels fail.
- Schools need pilot data before generated items influence progression or support.
Global context
Curriculum language, assessment culture and qualification rules differ internationally. A workflow using North Rhine-Westphalia definitions cannot be assumed valid elsewhere. The transferable lesson is methodological: wording, psychometric behaviour and claims about cognitive demand need separate checks. Multilingual and under-resourced schools also need evidence that generated language does not add inequitable reading demands or unfamiliar assumptions.
What the evidence does not yet show
- One 27-item German electrochemistry test, one GPT-4 process and 95 students.
- No human-written comparison instrument.
- Person reliability was 0.66 and level alignment unstable.
- Researchers manually checked prompt compliance and textbook correctness.
What to watch next
- Matched expert-written comparisons.
- Accessibility and subgroup item functioning.
- Teacher time and revision burden.
- Whether level alignment improves with new prompts.
Living evidence record
Impact record IAI-1QUY8QK
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Education published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Whether GPT-4-generated secondary chemistry questions meet item-writing standards, function psychometrically and match curriculum competence levels
Topics: Assessment · Chemistry education · GPT-4 · Rasch modelling · Teachers · Validity
Regions: Germany · Europe
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.