Did a Socratic AI raise student engagement?
Across eight course sections at four Chinese universities, students offered EduBot reported larger within-semester engagement gains. Section-level allocation, one course per university and no follow-up beyond the semester limit causal and general claims.
The short version
3 key points
- 1.The study enrolled 583 students in eight sections of four introductory courses; 555 complete cases comprised 278 intervention and 277 control students.
- 2.Post-test differences favoured the AI sections for behavioural engagement (d=0.71), agentic engagement (d=0.57), cognitive engagement (d=0.55) and emotional engagement (d=0.34).
- 3.Only eight sections were allocated, university and subject were confounded, outcomes were self-reported and the last measurement occurred immediately after access ended. The study does not show durable learning or higher grades.
The answer: engagement improved, but learning was not proved
Students in four course sections given a Socratic conversational agent reported stronger engagement trajectories over one semester than students in four parallel sections taught as usual. The largest standardised post-test difference was behavioural engagement at 0.71, followed by agentic engagement at 0.57, cognitive engagement at 0.55 and emotional engagement at 0.34. Group-by-time interactions remained statistically significant after adjustment for the four tested dimensions.
That is encouraging evidence for a structured learning aid, not proof that a chatbot improves achievement. Engagement was measured with student questionnaires, and the study did not establish higher exam marks, deeper knowledge retention or effects after the semester. Only eight sections were allocated. A difference attached to a section may reflect class dynamics or other unmeasured factors, and each university contributed a different subject, making university and discipline impossible to separate.[1]
How the eight-section comparison worked
Four Chinese research universities contributed one introductory course each: psychology at Zhejiang, computer science at Nanjing, business at Wuhan and biology at Fudan. Every course ran in two parallel sections with the same instructor, syllabus, materials, assessments and teaching-assistant support. Students had enrolled through normal timetabling before the study announcement. A course coordinator used a coin toss to assign one section to EduBot and one to the control condition.
The researchers obtained consent from 583 students and analysed complete data for 555: 278 in intervention sections and 277 in controls. Scores were collected at weeks 1, 7 and 14, with EduBot available for 12 weeks. The analysis used mixed-effects models to account for repeated observations and nesting in course sections, plus mixed ANOVA and sensitivity analyses. Qualitative evidence came from interviews and focus groups and was used to generate explanations, not to prove statistical mediation.[1]
What changed on the seven-point scales
Behavioural engagement in the intervention group rose from 4.82 to 5.47, while the control group moved from 4.79 to 4.76. Cognitive engagement rose from 4.91 to 5.36 with EduBot and changed from 4.88 to 4.85 in control sections. Agentic engagement—students contributing questions and shaping their learning—rose from 4.45 to 5.11 versus 4.41 to 4.52. Emotional engagement showed the smallest contrast: 5.04 to 5.19 versus 5.01 to 4.82.
Usage was uneven. Intervention students logged a mean of 31.42 sessions and a median of 28, but 18 students, 6.5%, never used the tool. Within the intervention group, more sessions were associated with higher post-test engagement after adjustment for baseline and covariates. That dose-response pattern is compatible with benefit, yet motivated students may also choose to use the tool more. It cannot by itself show that extra sessions caused the higher scores.[1]
Why the Socratic design may matter
EduBot was designed to ask questions and scaffold reasoning instead of immediately supplying answers. The thematic analysis identified three proposed mechanisms: students described a low-judgment place to repeat questions, prompts that made them persist through a problem and access outside office hours. Teachers and assistants reported that routine questions could be handled before students sought human help. These accounts are plausible and useful for design, but they were not tested as measured causal pathways.
The intervention was a package. It changed availability, conversational format, personalisation and questioning style at once. The comparison therefore cannot tell a university whether Socratic prompts, round-the-clock access or simply extra contact produced the difference. Nor can it show whether the same design would help in a seminar, a vocational course or a less digitally prepared institution. Component tests and replicated courses are needed before choosing one feature as the active ingredient.[1]
Practical implications for teachers and universities
Teachers considering a similar tool should begin with a bounded purpose: concept clarification and guided practice, not unsupervised grading or replacement of contact. They should inspect how the system responds to misconceptions, require citations where appropriate, protect student data and keep a route to a human teacher. Engagement should be tracked alongside learning outcomes. A busier discussion board or higher self-report score is not enough if misconceptions persist or students become dependent on prompts.
Universities should pay particular attention to non-users. The 18 students who never opened EduBot may represent access, trust, disability, motivation or digital-literacy barriers. Exploratory results suggested larger agentic gains for first-year students and larger cognitive gains for students with lower prior GPA, but only two of eight moderation tests survived correction and they were not pre-registered. Institutions should not present those subgroup findings as settled evidence of equity benefits.[1]
Evidence limits and what would change the assessment
The strongest limitation is the small number of clusters. Students were not individually randomised, only eight sections participated and unmeasured section effects remain possible despite balanced measured characteristics. University and course were completely confounded, so a subject effect could look like an institutional effect. Measures were self-reported and the last survey occurred in the week after access ended, leaving durability unknown. The integrated package also prevents attribution to any single design feature.
The study was funded by China's National Social Science Fund through a project on university teachers' digital literacy; the authors declared no commercial conflict. Confidence would rise with a preregistered cluster-randomised trial across many replicated courses, direct knowledge tests and grades, longer follow-up, documented model behaviour and analysis of who declines or stops use. For now, the result supports supervised pilots of Socratic tutoring. It does not justify replacing teachers or claiming that AI raised attainment.[1]
What this means for people
- Students may gain a private, flexible place to practise questions, but current evidence measures engagement rather than durable learning or grades.
- Teachers can use AI for guided practice while preserving human feedback, pastoral contact and responsibility for assessment.
- Universities should investigate non-use and unequal access rather than infer equity from exploratory subgroup findings.
Global context
The study involved four highly resourced Chinese universities and one introductory course at each. Classroom norms, digital readiness, language, assessment and access vary internationally, so the effect should not be assumed to transfer unchanged.
What the evidence does not yet show
- Only eight course sections were allocated, limiting cluster-level causal inference
- University and discipline were perfectly confounded
- Engagement outcomes were self-reported rather than direct learning measures
- No follow-up tested whether differences persisted beyond the semester
- The intervention bundled several features, preventing component attribution
What to watch next
- Preregistered trials with many sections and replicated subjects
- Exam performance, knowledge retention and independent problem-solving
- Longer follow-up after AI access ends
- Accuracy, privacy and teacher-override monitoring
- Participation and outcomes for students who initially avoid the tool
Living evidence record
Impact record IAI-0VC6HF8
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
15 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Psychology published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 15 October 2026
Research question, topics and regions
Whether access to a Socratic conversational AI was associated with changes in behavioural, cognitive, emotional and agentic engagement over one university semester
Topics: Higher education · Student engagement · AI tutoring · Quasi-experiment · Teachers
Regions: China · Asia
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
UNESCO
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Education & human agency · Background explainer
Published 2025-01-23 · 8 min 30 sec
Teacher to Teacher: Is AI reshaping education for the better?
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Education
Did an AI teaching assistant improve one Python course?
A peer-reviewed Chinese study reports stronger engagement and assessment results in one 60-student class using a course-restricted AI assistant than in one conventional class. With only one intact class per condition, the result is promising but not a clean test of AI alone.
7 min · 1 source
Education
Can an AI-labelled course match restore student engagement?
In a simulated experiment with 293 Chinese university students, a report claiming an 85% AI-derived match raised self-reported engagement after forced reassignment to the level seen with self-selection. No AI actually matched anyone, and the study measured intentions after a micro-lecture—not attendance, grades or learning.
8 min · 1 source
Education
Did AI-assisted teaching improve computational thinking?
A 14-week pilot at one Chinese medical college found larger gains in an AI-assisted class than in a conventional class. But one intact class received each condition, so the 120-student study cannot separate the teaching package from class, teacher, time-on-task or other group differences.
8 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.