AI lifted proof homework scores—but the advantage disappeared in the exam
In a 153-student course, earlier access to two AI tools improved supported homework by 2.56 points but produced no detectable gain on the next AI-free exam. Later chatbot use was observational, so it cannot establish that the chatbot caused lower scores.
The short version
- What happened
- Researchers randomly assigned 153 students in one undergraduate proof course to receive an AI chatbot and proof-feedback tool earlier or later.
- Who it affects
- A student can submit stronger AI-assisted work without becoming better prepared to solve a new problem alone. Teachers need separate evidence of independent understanding before treating higher homework marks as learning.
- What remains unproven
- The study is a preprint and has not been peer reviewed.
Keep in mind
The study is a preprint and has not been peer reviewed.
It covers one proof-oriented university course; 153 of 308 enrolled students consented, and the two AI interfaces were introduced together.
Later tool use was self-selected, so the association between message volume and exam scores is not a causal estimate.
The study ended before the final exam because of budget limits and did not measure cognitive effort directly.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Better homework did not become a measurable exam gain
Students were split into early- and later-access groups in an ordinary Carnegie Mellon proof course. The tools arrived together: an open-ended GPT-4o chatbot and an interface that required a written proof before giving feedback. Across two homework assignments, early access produced a small adjusted gain. On the next midterm, completed without AI, the groups did not differ significantly.[1]
The later usage result needs a causal warning
Once all students could use both tools, every additional 100 chatbot messages was associated with 3.02 fewer points on the following exam after adjustment for prior performance and self-efficacy. Students with lower self-efficacy also used the tools more. Because students chose how much help to request, unmeasured struggle, outside assistance or study time could explain part of the pattern; cognitive offloading was not measured directly.[1]
The distinction that matters
What the study can and cannot attribute
Randomisation applied to access timing, not to each tool or to later usage volume.
Randomised
Earlier versus later access to both tools
+2.56 homework points; no detected exam gain
Observed
Self-selected chatbot message count
Associated with lower later scores
Unproven
Cognitive offloading mechanism
Plausible interpretation, not a measured causal effect
Teachers need to test what students can do alone
The practical distinction is between help that critiques a learner's attempt and help that supplies missing reasoning. Schools evaluating an AI tutor should compare supported work with a later task completed independently, and should not use chat counts as a disciplinary signal. A stronger test would randomise students separately to answer-giving chat, hint-only chat and feedback on an attempted solution.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring ai tutoring: compare the finding and its evidence limits.
Do learning agents improve results? A 32-person study cannot isolate the agent effect
An unreviewed China–Singapore study found larger short-term assessment gains with an agent-organised learning programme than a comparison tutor. Different materials, practice and assessment designs mean the result does not show that multi-agent coordination caused the difference.
Read the next reportExplore the full source and evidence record
What this means for people
- A student can submit stronger AI-assisted work without becoming better prepared to solve a new problem alone. Teachers need separate evidence of independent understanding before treating higher homework marks as learning.
Global context
The finding matters beyond US mathematics courses because many education products report completion or satisfaction while leaving later unaided performance untested. Local assessment rules, language and subject design could change the result.
What the evidence does not yet show
- The study is a preprint and has not been peer reviewed.
- It covers one proof-oriented university course; 153 of 308 enrolled students consented, and the two AI interfaces were introduced together.
- Later tool use was self-selected, so the association between message volume and exam scores is not a causal estimate.
- The study ended before the final exam because of budget limits and did not measure cognitive effort directly.
What to watch next
- Peer review and replication in other subjects
- Three-arm trials separating answers, hints and feedback on attempts
- Delayed retention and transfer tests
- Whether teachers can detect and support struggling high-use students without punitive monitoring
Living evidence record
Impact record IAI-1724Z5K
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
23 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 23 October 2026
Research question, topics and regions
Whether access to open-ended chat and feedback-on-attempt tools improves supported proof work and later independent performance
Topics: AI tutoring · Mathematics education · Independent learning · Cognitive offloading · Assessment
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.