Can automated feedback help peer tutors draw out learner reasoning? A randomized trial found modest gains
Across two SAT mathematics bootcamps, transcript-based feedback increased three measured talk moves by 8–15%. It did not reduce tutors’ 91.5% share of speaking time or improve attendance, practice-test performance or satisfaction after correction.
The short version
- What happened
- The trial randomized 1,199 tutor assignments across four conditions in two online SAT mathematics bootcamps; the pooled complete-case analysis covered 5,653 sessions from 899 tutors.
- Who it affects
- Volunteer tutors may receive specific prompts for drawing out learner thinking when human coaching is unavailable, but the tool did not reduce their dominance of speaking time.
- What remains unproven
- Unreviewed preprint from one online SAT platform; most tutors were high-school volunteers and generalisability to schools or professional tutoring is unknown.
Keep in mind
Unreviewed preprint from one online SAT platform; most tutors were high-school volunteers and generalisability to schools or professional tutoring is unknown.
The four-week trial had incomplete delivery, prior exposure among some controls and no log showing whether or when tutors opened feedback.
Discourse outcomes came from automated transcripts and classifiers that detect the presence, not the quality or appropriateness, of a talk move.
No subsequent SAT scores were available; pooled attendance, practice-test and satisfaction outcomes were null.
Interviews were small, coded by one researcher and conducted by platform staff; several authors were affiliated with Schoolhouse.world.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The direct answer: feedback changed selected moves, not who held the floor
Automated feedback can nudge volunteer peer tutors to invite and build on learner thinking, but it did not make the sessions broadly learner-led. In an unreviewed preregistered randomized trial on Schoolhouse.world, tutors assigned to receive transcript-based feedback elicited learner ideas about 8% more often, took up learner contributions about 9% more often and were followed by about 15% more coded learner reasoning than tutors assigned no feedback. These are modest, statistically supported changes in discourse markers—not evidence of higher examination scores or durable learning.
The strongest null result is just as important. Tutors in the control group occupied 91.5% of speaking time, and the estimated treatment effect on that share was effectively zero. Adding a participation message for learners did not further change discourse. Attendance, practice-test performance and satisfaction measures also showed no significant pooled effect from tutor feedback. The trial therefore separates two claims that are often blurred: a system can change particular conversational techniques without redistributing participation or improving measured educational outcomes over four weeks.[1]
Two bootcamps, four conditions and one tutor-level randomization
The setting was two consecutive four-week SAT mathematics bootcamps in spring 2024. Each offered eight 75-minute Zoom sessions to groups of up to ten learners. Schoolhouse.world tutors are volunteers, mostly high-school students, who qualify through SAT mathematics performance and platform training rather than professional teaching credentials. Across both bootcamps the study involved 1,030 distinct tutors, 7,443 learners and 6,508 eligible recorded sessions before complete-case restrictions. The platform serves learners across more than 180 countries, although the paper does not report a country denominator for this trial.
Before sessions began, 1,199 tutor assignments were randomized at tutor level. Control tutors received no feedback; a tutor-feedback group received automated analysis of its own discourse; and two joint groups received the same tutor feedback while learners saw a participation message framed either around personal goals or benefits to the group. Randomization was stratified by SAT score band and section size. A tutor running several sections received the same assignment within a bootcamp, while 169 tutors participating in both bootcamps were randomized again for the second.[1]
The distinction that matters
What changed in the randomized trial
The intervention improved three coded discourse measures but did not alter talk balance or measured short-term learning.
Eliciting
+1.41 per hour
8% above the control mean
Uptake
+0.52 per hour
9% above the control mean
Reasoning
+0.24 per hour
15% above the control mean
Talk share
No detectable change
Tutors still spoke about 91.5% of the time
Learning
No pooled effect
Practice tests and attendance did not improve
What the intervention actually showed tutors and learners
Tutor feedback arrived after a recorded session and usually before the next one. It compared the tutor’s talk share and proportion of participating learners with earlier sessions and the bootcamp average. Classifiers highlighted examples of eliciting ideas, taking up contributions and asking for reasoning. GPT-4 Turbo generated one to three suggestions grounded in the transcript roughly every other session. The sequence followed a fixed curriculum rather than choosing whichever metric looked weakest. Tutors could revisit earlier feedback and received email reminders when they had not opened it.
Learner feedback was thinner. In the joint conditions, learners saw a modal when entering their next session showing the group’s share of talk and a message encouraging participation. They could not revisit it and received no reminder. One framing linked speaking to their own score goals; the other argued that questions could help peers. The display reported a group statistic rather than each learner’s contribution. That design difference matters when interpreting the null result: tutor and learner arms did not receive equivalent information, timing, persistence or opportunity to act.[1]
The analysis used intention to treat—and delivery was incomplete
The preregistered primary analysis compared groups as assigned, whether or not a particular tutor viewed feedback. It excluded the baseline first session, the practice-test fifth session, recordings shorter than 30 minutes and sessions missing required covariates. The pooled complete-case sample contained 5,653 sessions from 899 tutors; learner-level models used up to 14,839 learner-sessions. Ordinary least-squares models included baseline discourse measures, tutor experience and gender, cohort composition, SAT band, session-number and bootcamp effects, with standard errors clustered by tutor and Benjamini–Hochberg correction within outcome families.
Delivery was not complete. Feedback reached 77.9% of treated learner-sessions in the first bootcamp and 56.0% in the second, where an error withheld much early feedback. Some second-bootcamp control tutors had also received feedback in the first bootcamp or a pilot. Both issues could reduce the difference between assigned groups. Instrumental-variable and sensitivity analyses produced somewhat larger estimates, but they require additional assumptions. The intention-to-treat result remains the most policy-relevant estimate of offering this imperfectly delivered system under the tested conditions.[1]
What increased—and what did not
Across bootcamps, treated tutors elicited learner ideas 1.408 more times per hour than controls, whose mean was 17.161; uptake increased by 0.519 over a control mean of 5.914; and coded learner reasoning increased by 0.238 over 1.637. The associated p-values were .005, .004 and .001, and all three survived the family correction. Effect sizes were about 0.12 control standard deviations. Signs were broadly consistent across the two bootcamps, although no single-bootcamp estimate survived correction on its own.
Other outcomes did not clear the threshold. Learner questions and the proportion speaking moved in a favourable direction but were not conventionally significant after correction. Tutor speaking share changed by minus 0.001, p=.803. Attendance, cohort or learner practice-test performance, ratings and net promoter scores showed no significant pooled effect. A few uncorrected satisfaction contrasts favoured learner feedback, but none would survive correction. The authors appropriately conclude that observable discourse changed while measured learning and the balance of talk did not.[1]
Interviews explain barriers, but they are not a second experiment
The team interviewed 17 treated tutors and 13 learners from the first bootcamp. Most tutors said the summary statistics and concrete generated suggestions were useful, and several described replacing a closed invitation such as ‘any questions?’ with wording that assumed questions existed. Ten tutors named learner engagement as their biggest challenge. The highlighted transcript excerpts were less valued: most interviewed tutors said they ignored them or saw them as redundant beside the generated suggestions.
Learners described why a group-level participation number was hard to act on: one learner could speak more yet see little movement if peers stayed silent, some preferred listening and learner-to-learner interaction was rare. But the interviews were small, conducted by a platform staff member and coded by one person; no control tutors were interviewed. They help generate design explanations, not estimate effects. The platform later changed its feedback based partly on these accounts, but the paper does not evaluate whether that redesign works.[1]
The practical lesson for schools and tutoring platforms
For tutoring programmes, the experiment supports a limited use case: automated transcript feedback can increase selected, observable talk moves at scale. Adoption should still start with reliable delivery, consent, transcript security, accessibility and audits of classifier errors. Programmes should not equate a higher count of questions or reasoning-labelled utterances with better teaching. The classifiers detect whether a move appears, not its quality, timing or mathematical correctness. Only two classifiers were validated on this platform, and speech-recognition errors can affect every downstream measure.
The study cannot show that feedback raises SAT scores because subsequent examination scores were unavailable and practice-test estimates were null. It lasted four weeks on one nonprofit platform with mostly high-school volunteer tutors. Schoolhouse.world staff co-authored the work, though the quantitative analysis was led by an unaffiliated researcher and the plan was preregistered. Confidence would rise with independent replication, complete delivery logs, validated learning measures, longer follow-up and trials of session structures that genuinely give learners more opportunities to speak. Until then, the intervention is a modest coaching aid—not an automated substitute for educator development.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring randomized trials: compare the finding and its evidence limits.
Do agent-style AI tutors improve learning? Two trials suggest gains—but attrition clouds the result
A Tsinghua preprint found a structured legal tutor beat plain chat and a coding workbench improved expert-rated mastery. The programming analysis included only 54 completers from 140 recruits, and the preregistrations are not linked.
Read the next reportExplore the full source and evidence record
What this means for people
- Volunteer tutors may receive specific prompts for drawing out learner thinking when human coaching is unavailable, but the tool did not reduce their dominance of speaking time.
- Learners produced more reasoning-labelled utterances without a measured improvement in practice-test performance, so programmes should not promise learning gains from discourse changes alone.
- Schools and platforms need consent, transcript security, reliable delivery and human oversight before scaling automated analysis of recorded teaching sessions.
Global context
Schoolhouse.world serves learners across more than 180 countries, but the trial does not provide a country distribution and uses English-language SAT mathematics sessions. Speech recognition, tutoring norms, test stakes, connectivity and consent rules differ across education systems. The study supports further local trials, not automatic transfer to classrooms or multilingual settings.
What the evidence does not yet show
- Unreviewed preprint from one online SAT platform; most tutors were high-school volunteers and generalisability to schools or professional tutoring is unknown.
- The four-week trial had incomplete delivery, prior exposure among some controls and no log showing whether or when tutors opened feedback.
- Discourse outcomes came from automated transcripts and classifiers that detect the presence, not the quality or appropriateness, of a talk move.
- No subsequent SAT scores were available; pooled attendance, practice-test and satisfaction outcomes were null.
- Interviews were small, coded by one researcher and conducted by platform staff; several authors were affiliated with Schoolhouse.world.
What to watch next
- Peer review and independent replication on another platform.
- Trials with reliable delivery and direct opening logs.
- Longer follow-up using validated learning outcomes and subsequent test scores.
- Multilingual evaluation of transcription and discourse classifiers.
- Session redesigns that create real learner-to-learner interaction rather than only reporting talk ratios.
Living evidence record
Impact record IAI-16O9663
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
Whether transcript-based automated feedback changes volunteer peer tutors' discourse, learner reasoning, participation and short-term learning outcomes
Topics: Peer tutoring · Automated feedback · Education · Classroom discourse · Randomized trials · Learner participation
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.