Do agent-style AI tutors improve learning? Two trials suggest gains—but attrition clouds the result
A Tsinghua preprint found a structured legal tutor beat plain chat and a coding workbench improved expert-rated mastery. The programming analysis included only 54 completers from 140 recruits, and the preregistrations are not linked.
The short version
- What happened
- In a three-arm legal-learning trial, 111 analysed students using the same base model gained 22.8 points with AI Cosmos, 18.5 with a teacher prompt and 15.7 with plain chat; the planned AI Cosmos-versus-chat contrast was seven points.
- Who it affects
- Students may learn differently from systems that require them to explain, build and debug instead of delivering an answer. These trials are too small and selective to justify replacing teachers or buying an agentic tutoring platform on their own.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
Preregistrations not linked
Programming analysis only 54 completers from 140 recruits
No intent-to-treat analysis
Per-arm exclusions not reported
Small programming arms
Underlying model version not named
Delayed legal comparison underpowered
No ordinary-classroom deployment
No funding or competing-interests statement
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
Structured interaction improved one immediate legal outcome
Tsinghua University's AI Cosmos team reports two three-arm randomized studies that kept the underlying language model constant while changing how students interacted with it. In the legal study, 131 students were assigned to plain chat, the same model prompted as a teacher, or a structured scenario with role-based dialogue. Minimum-engagement exclusions left 111 participants—37 per arm—for analysis. They completed matched civil- and criminal-law pre-tests, learning tasks and immediate post-tests scored on a 100-point scale.
Average gains rose from 15.7 points with plain chat to 18.5 with the teacher prompt and 22.8 with AI Cosmos. The prespecified Cosmos-versus-chat contrast was seven points, with Cohen's d of 0.56 and p=0.018; an analysis adjusting for baseline gave a 4.5-point difference with the same p-value. The overall test was only marginal, and gains on open responses did not differ. At four weeks, 18 people per arm retained a directionally larger Cosmos gain, but the between-group comparison was inconclusive.[1]
The programming result distinguished test scores from mastery
The second study compared plain chat, a teacher-style assistant and AI Cosmos with an executable coding workbench. Participants practised newly written algorithm problems, then independently attempted two unseen problems one week later. Two problem authors, blind to condition, rescored code using a five-level mastery rubric. Among submissions receiving positive automated-judge credit, 68% in Cosmos met the mastery threshold, against 45% with plain chat and 32% with the teacher baseline. The Cosmos-versus-teacher comparison had p=0.009; the difference from plain chat did not reach significance.
Mean post-test scores were 68.0, 61.5 and 49.8 respectively, but the overall comparison was not statistically significant (p=0.20). The expert review matters because an automated judge can award points to incomplete or brute-force code. Yet it also creates a narrower claim: the study supports a difference in the rubric-rated quality of completers' code more clearly than a general improvement in programming performance.[1]
The distinction that matters
What the two trials can—and cannot—show
Random assignment strengthens the comparisons, but exclusions and attrition mean the reported analyses do not represent every person initially assigned or recruited.
Legal trial
131 assigned
111 analysed
Programming trial
140 recruited
54 completed
Programming score
Three-arm comparison
Not significant
Code mastery
Cosmos vs teacher
Higher among completers
Attrition prevents a confident causal claim about all recruits
The main limitation is participant flow. Of 140 programming recruits, 100 completed the diagnostic and only 54 completed the multi-session protocol; the analysed arms contained 17–19 people. Advancement required full-score submissions on three training problems, a gate the intervention itself could influence. The paper does not report per-arm dropout at each stage or an intent-to-treat analysis, so the groups of completers may differ in ways random assignment no longer protects against. The legal study likewise omits per-arm counts for its 20 engagement exclusions.
The authors describe both studies as preregistered but do not link a registry, and the paper does not name the base-model version or provide funding and competing-interest statements. For teachers and institutions, the useful signal is that interaction design and independent assessment may matter as much as model choice. Confidence would rise with linked protocols, complete participant-flow tables, intent-to-treat estimates, a named model, peer review and replication in ordinary classrooms over longer periods.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring ai tutoring: compare the finding and its evidence limits.
AI tutor access lowered grades in a university trial — but few students used it
A randomized Maryland study found lower grades and course-platform activity after instructors received a course-integrated assistant. Uptake was only about 15%, and the paper is not peer reviewed.
Read the next reportExplore the full source and evidence record
What this means for people
- Students may learn differently from systems that require them to explain, build and debug instead of delivering an answer. These trials are too small and selective to justify replacing teachers or buying an agentic tutoring platform on their own.
Global context
The studies were led by a Chinese university and used legal-learning and competitive-programming tasks. The paper does not establish effectiveness across school ages, languages, curricula, accessibility needs or national legal systems.
What the evidence does not yet show
- Unreviewed preprint
- Preregistrations not linked
- Programming analysis only 54 completers from 140 recruits
- No intent-to-treat analysis
- Per-arm exclusions not reported
- Small programming arms
- Underlying model version not named
- Delayed legal comparison underpowered
- No ordinary-classroom deployment
- No funding or competing-interests statement
What to watch next
- Peer review
- Linked preregistration records
- CONSORT-style participant flow
- Intent-to-treat estimates
- Named model and version
- Larger classroom replication
- Longer retention tests
- Teacher workload and accessibility outcomes
Living evidence record
Impact record IAI-1LS5BX4
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
31 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 31 October 2026
Research question, topics and regions
Whether structured AI interaction and an executable workbench improve independently assessed legal reasoning and programming competence
Topics: AI tutoring · Legal education · Programming education · Randomized trials · Learning outcomes · Agentic systems
Regions: China · Asia
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.