Do better prompts make AI fairer for students? Outputs improved, but learning and equity were not tested
A structured prompt raised expert and model ratings across 800 outputs. A separate student survey linked prompt confidence with perceived gains, but neither stage measured learning or unequal outcomes.
The short version
- What happened
- The study combined a 380-student self-report survey in China with an experiment rating 800 outputs from four language models across four university task types.
- Who it affects
- Students may benefit from being shown a repeatable way to specify a task, but teachers still need to assess the underlying learning rather than the polish of AI-assisted output.
- What remains unproven
- Cross-sectional, self-reported survey of 380 opt-in undergraduate LLM users in China
Keep in mind
Cross-sectional, self-reported survey of 380 opt-in undergraduate LLM users in China
No observed grades, retention, family income or rural–urban measure
Synthetic prompt experiment measured outputs rather than student learning
Human and model ratings were aggregated without reported formal inter-rater reliability
Published ethics and data-availability statements conflict with the described research
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Structured prompts improved rated outputs
A peer-reviewed study found that a structured prompt scaffold improved the rated quality of language-model outputs across four higher-education tasks. Researchers wrote 25 basic prompts for each task, then refined them using Role, Goal, Input, Process and Output instructions. Four models—DeepSeek V3, Doubao lite, GPT-5 nano and Claude 3.5 Haiku—produced 400 matched pairs, or 800 outputs.
Two human experts and GPT-5 nano rated seven dimensions on a five-point scale. The aggregated score rose from 3.318 for basic prompts to 4.603 for refined prompts: +1.285 points, with a 95% confidence interval of 1.255–1.315. Gains appeared across every task and model. The experiment therefore supports a narrow claim about this scaffold and these rated outputs—not a general claim that elaborate prompts always improve results.[1]
The study did not test learning or equity
A separate online survey of 380 undergraduate LLM users in China found that self-rated prompt competence was strongly associated with perceived academic gains. That cross-sectional relationship cannot show that prompting caused learning, and the survey did not measure grades, knowledge retention, family income or rural–urban background. The experimental stage tested output quality, not student achievement or gaps between groups.
There are further reasons for caution. The paper aggregates human and automated ratings without reporting formal inter-rater reliability, even though their directions differed between conditions. Its published ethics statement says no new human-participant study or human data were involved, and its data statement says no datasets were generated or analysed—both conflict with the reported survey and experiment. That publication-record inconsistency needs clarification. Educators can teach structured prompting as a provisional workflow, but should not present this study as proof that it improves learning or makes AI access fairer.[1]
The distinction that matters
What each stage actually tested
The paper joins two different forms of evidence. Keeping them separate prevents output ratings from being mistaken for student outcomes.
Student survey
380 users rated their own prompting and academic gains
Association, not a causal learning test
Prompt experiment
400 basic/refined pairs produced 800 outputs
+1.285 points on an aggregated five-point rating
Still untested
No observed grades, retention or group outcome gap
Equity and learning claims remain open
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring assessment: compare the finding and its evidence limits.
Can coursework show how students used AI without relying on detection?
A two-course study made AI use, checking and revision more inspectable. Independent practical tests did not show a clear learning gain.
Read the next reportExplore the full source and evidence record
What this means for people
- Students may benefit from being shown a repeatable way to specify a task, but teachers still need to assess the underlying learning rather than the polish of AI-assisted output.
Global context
The student survey was conducted in China and the four-model experiment included Chinese and international systems. Access, language, teaching practice and assessment rules differ across countries, so the equity interpretation should not be generalized without direct outcome studies elsewhere.
What the evidence does not yet show
- Cross-sectional, self-reported survey of 380 opt-in undergraduate LLM users in China
- No observed grades, retention, family income or rural–urban measure
- Synthetic prompt experiment measured outputs rather than student learning
- Human and model ratings were aggregated without reported formal inter-rater reliability
- Published ethics and data-availability statements conflict with the described research
What to watch next
- Randomized classroom trials with observed learning and delayed retention
- Preregistered independent human scoring with inter-rater reliability
- Publisher clarification of the ethics and data-availability record
- Replication across socioeconomic groups, languages and institutions
Living evidence record
Impact record IAI-0KQENK1
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
26 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what International Journal of Educational Technology in Higher Education published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 26 October 2026
Research question, topics and regions
Whether prompt-engineering competence is associated with perceived academic gains and whether a structured scaffold improves rated LLM output quality
Topics: Higher education · Prompt engineering · Student equity · Assessment
Regions: Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.