Does AI help trainee teachers reflect better after metacognitive training?
In a randomized 60-person experiment, a configured Socratic AI improved written reflection only when paired with a 20-minute metacognitive lesson. The immediate, single-university result does not show that generic chatbot access improves teaching practice.
The short version
3 key points
- 1.Sixty third-year trainee teachers at one Chinese university were randomized equally across metacognitive or placebo training and AI Socratic or conventional digital-tool conditions.
- 2.With metacognitive training, the AI condition scored 0.56 adjusted points above traditional tools; without the training, the difference was -0.03 and not significant.
- 3.The AI condition bundled generation, adaptive questioning and dialogue, the outcome was immediate written reflection and each cell held only 15 participants.
The answer: preparation mattered more than access alone
A configured AI system improved trainee teachers' written reflection only when users first received a short lesson in planning, self-questioning, monitoring and evaluation. Among 60 participants randomized across four groups, the adjusted AI-versus-traditional-tool difference was 0.56 points when metacognitive training was present. Without that training, the difference was minus 0.03 points and statistically indistinguishable from zero. The interaction between training and tool condition was significant, with a partial eta-squared of 0.165. The useful finding is conditional: how learners approach the tool may be as important as giving them access.
This is not evidence that any chatbot makes teachers more reflective. The AI condition used a DeepSeek V3 web interface deliberately configured to avoid ready-made answers and ask progressively probing questions. It bundled generative output, dialogue, adaptive follow-up and Socratic instructional design. The comparison group used a text editor and a university digital repository. Because those features were not separated, the experiment cannot tell whether generation, questioning, novelty or another component caused the difference. Nor did it observe a classroom, a pupil or a teaching decision beyond the study task.[1]
How the experiment worked
Sixty third-year pre-service teachers at Changzhou Institute of Technology completed the full procedure. Stratified random allocation placed 15 in each cell of a two-by-two design: metacognitive training or a time-matched placebo lesson, combined with the configured AI or conventional digital tools. Before any intervention, everyone had 30 minutes to analyse a teaching vignette and submit a written reflection. One week later, participants received a 20-minute lesson—either metacognitive strategies or PowerPoint design—and completed a matched post-test case with their assigned tool.
Trained raters scored the written reflections on a rubric covering evidence, attribution, perspectives, strategy and clarity. The researchers used an analysis of covariance to compare post-test performance while controlling for the pretest. Baseline scores did not differ significantly across groups, and reported checks supported the model assumptions. Sensitivity analyses excluding influential observations, using robust standard errors and restricting the sample to 53 participants who reported following their assigned condition retained the interaction. Those checks make the internal result more credible, though they cannot compensate for the small cells or single setting.[1]
What changed in the written reflections and prompts
The adjusted post-test mean was 4.13 for metacognitive training plus AI, compared with 3.57 for metacognitive training plus conventional tools, 3.33 for placebo plus AI and 3.36 for placebo plus conventional tools. The interaction was strongest for depth and attribution, theory and perspective taking, and strategy generation and transfer. It was not significant for basic description and evidence or for structure and clarity. That pattern is compatible with the idea that preparation helped participants use dialogue for higher-order analysis rather than simply polishing a response.
The researchers also coded 487 prompts from the two AI groups. Among participants who received metacognitive training, 44.1% of prompts were classified as higher-order and 21.3% as lower-order. In the placebo group, 12.2% were higher-order and 54.5% lower-order. Two doctoral coders achieved a Cohen's kappa of 0.86 on an initial 20% sample. This behavioural difference supports the proposed mechanism, but it does not prove mediation: the log analysis did not statistically connect each participant's prompt pattern with the treatment and final reflection score through an indirect-effect model.[1]
What teachers and course designers can use now
Teacher educators considering AI-supported reflection should teach a method for using it, not just demonstrate prompts. Learners can be required to state an initial interpretation, identify evidence, test an alternative explanation, consider another perspective and revisit their conclusion after the system responds. The AI can be configured to withhold complete answers until those steps occur. This creates productive friction and gives instructors observable process evidence. It also reduces the risk that a reflective exercise becomes a request for a fluent improvement plan written by the system.
The experiment does not establish that the combined method is better than an equally structured non-AI Socratic scaffold. Its traditional condition offered a repository and text editor, not a rule-based questioning tool or trained human tutor. Institutions should therefore pilot the smallest intervention that meets the learning goal and compare it with credible alternatives. They should also keep assessment focused on the learner's reasoning and ability to justify action. A well-formed prompt is not evidence that a trainee can adapt safely in a classroom or recognise when an AI suggestion conflicts with a pupil's needs.[1]
Why the result remains narrow
Each experimental cell contained only 15 people, making estimates vulnerable to sampling variation even though the reported effect survived several checks. Participants were volunteers with basic information-technology skills at one Chinese university. Local norms around teacher authority, criticism and technology could affect both how reflection is written and how comfortably a student challenges an AI system. The adapted reflection questionnaire was examined in the same small sample, so its psychometric structure is exploratory rather than a definitive cross-cultural validation.
The intervention was brief and the post-test immediate. The study cannot tell whether participants retained metacognitive habits, used them during teaching practice or improved outcomes for pupils. It also cannot separate the effect of DeepSeek V3 from the system prompt and interaction design. The authors report no external funding and no commercial conflicts. That independence is useful context, but reproducible configuration details, preregistration and independent replication would be more important for judging whether the result survives across models and institutions.[1]
What would change the assessment
A larger preregistered trial across universities should retain random allocation, use blinded raters and compare four credible supports: configured AI, a non-AI Socratic scaffold, human-guided reflection and ordinary digital tools. A dismantling design could separate adaptive questioning from generative content and dialogue. Researchers should publish the configuration and test more than one model so that a finding about pedagogy is not mistaken for a claim about a particular product.
Follow-up during teaching placements would provide the most important evidence. Researchers could examine lesson revisions, responses to classroom dilemmas, use of evidence, willingness to challenge automated suggestions and effects on pupils or mentors. Benefits should persist when the novelty has faded and when participants face messy, emotionally charged cases rather than matched written vignettes. Until then, the careful conclusion is that metacognitive preparation improved short-term use of one Socratic AI environment; generic access alone did not.[1]
What this means for people
- Trainee teachers may need explicit self-questioning and evidence-evaluation routines before an AI dialogue supports rather than substitutes for reflection.
- Teacher educators can assess the reasoning process and compare AI with simpler scaffolds instead of treating access as an intervention.
- Pupils are not represented in this study, so claims about classroom quality or learning remain untested.
Global context
The experiment involved one Chinese teacher-education institution. Cultural expectations, programme design, model access and assessment practices differ internationally, making cross-institutional and cross-cultural replication essential.
What the evidence does not yet show
- Only 60 participants completed the experiment, with 15 in each factorial cell.
- The single-university volunteer sample limits institutional and cultural generalisability.
- The AI condition bundled generation, adaptive questioning, dialogue and a Socratic design.
- Outcomes were immediate written reflections, not retained skill, classroom practice or pupil benefit.
- Prompt patterns were consistent with a mechanism but were not tested as statistical mediation.
What to watch next
- Preregistered multicentre replications with blinded scoring
- Comparisons with non-AI Socratic scaffolds and human coaching
- Long-term transfer into teaching placements and classroom decisions
- Component tests separating generation from adaptive questioning
Living evidence record
Impact record IAI-085FCCP
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
13 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Psychology published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 13 October 2026
Research question, topics and regions
Whether brief metacognitive preparation changes the effect and use of a configured AI Socratic reflection system among pre-service teachers
Topics: Teacher education · Metacognition · Generative AI · Reflective practice · Socratic tutoring
Regions: China · Asia
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Education
Does AI help students think about learning?
A 33-study meta-analysis found a positive average for technology-supported metacognition, but almost all effects came from self-report and the review did not show that AI outperformed other educational technology.
8 min · 1 source
Education
How are Kuwaiti health students using AI to write in English?
In a peer-reviewed survey of 241 nursing and health-sciences undergraduates, 94.6% reported previous generative-AI use and 83.4% used it for academic work at least sometimes. Students also reported accuracy and dependency concerns, while the single-institution self-report design cannot show that AI improved writing or learning.
7 min · 2 sources
Education
Can a two-hour workshop teach future teachers AI skills?
Spanish trainee teachers who attended a coding, robotics and machine-learning workshop reported stronger familiarity and attitudes than a separate control group. With no pretest or performance task, the study cannot show that the workshop caused durable skill gains.
7 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.