Do positive views of AI mean students use it reflectively? A 27-person study says not necessarily
In one Chinese programming course, students generally liked the AI tools even as logged activity became more restricted near the final task. The study was exploratory, did not measure learning gains and cannot show that the tool caused the change.
The short version
- What happened
- The peer-reviewed study followed 27 educational-technology undergraduates across eight sessions in a required Python course at one Chinese university.
- Who it affects
- Teachers and students may need to check task progress and the quality of reasoning directly; neither a positive AI survey score nor a low dashboard count reveals whether learning is deep, complete or stalled.
- What remains unproven
- Exploratory sample of 27 students
Keep in mind
Exploratory sample of 27 students
One Chinese university and one course
Five enrolled students excluded
Small and unequal clusters
Different tasks across stages
Profiles clustered independently at each stage
No control group or causal estimate
No learning-outcome measure
Generated feedback quality not independently rated
Source analysis · 1 linked source. Primary analysed source date: 27 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
Positive ratings did not map neatly onto reflective activity
A small peer-reviewed study suggests that students can value generative AI while making limited use of the reflective features built around it. Researchers followed educational-technology majors in a required Python course at one Chinese university. The students used a DeepSeek-V2 programming assistant and a separate dashboard that displayed their own and classmates' activity. At the final stage, four behaviour profiles ranged from broad exploration to a 13-person Limited Explorer group with little recorded reading or peer-oriented activity.
The 27 students still rated AI generally positively. Conventional and permutation-based tests found no statistically significant differences between the four final-stage profiles in perceived usefulness, ease of use, intention to use or attitude. That null result does not prove the profiles were equivalent: small, unequal groups limit statistical power. It does show why a favourable survey score should not be treated as evidence that students are comparing alternatives, checking feedback or integrating peers' ideas.[1]
The logs show a change in activity, not its cause
The study began with 32 second-year students who already had experience with C programming. Five were excluded because they did not review generated reports or had incomplete logs, leaving 27 complete records: 11 women and 16 men. Across eight one-hour sessions, students moved from structured visualization exercises to an open-ended final project. The dashboard recorded seven measures, including opening, moving and reading self- and peer-generated nodes; researchers also coded AI conversations and administered a post-course questionnaire.
Dashboard actions fell from 1,448 in the initial stage to 770 in the final stage. Separate K-means analyses produced three initial profiles and four final profiles, including the Limited Explorer group. Because each stage was clustered independently, similarly named groups do not necessarily contain the same students. The later task was also different. Lower activity could mean efficient completion, fatigue, loss of direction or disengagement; the logs cannot distinguish those explanations, and the design cannot show that the dashboard or AI caused the shift.[1]
The distinction that matters
What the study observed—and what it did not
The evidence separates logged behaviour, survey responses and unmeasured learning outcomes.
Observed
Count dashboard actions across two task stages
1,448 initial-stage actions and 770 final-stage actions
Compared
Test AI perceptions across four final-stage profiles
No statistically significant differences in four survey measures
Not tested
Measure learning gains or causal effects
No control group, performance outcome or causal estimate
For teachers, task status matters more than a label
The practical lesson is narrower than the profile names suggest. A teacher should not assume that a low activity count means a student is struggling, or that enthusiasm for AI means reflective learning is happening. Near a deadline, a student who has finished may need a prompt to verify and justify the final solution; someone with incomplete work may need diagnostic feedback and a clear next step. The authors present those supports as design hypotheses, not tested interventions.
The study measured platform behaviour and perceptions, not code quality, exam performance or durable learning. It came from one course, one institution and one cohort, and the generated feedback reports were not independently rated for quality. Stronger evidence would retain student-level profiles over time, synchronize events across both systems, compare learning outcomes with a suitable control and replicate the work in larger, more diverse courses. Until then, the result is a warning against equating acceptance with effective use—not proof that AI dashboards improve or damage learning.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring generative ai: compare the finding and its evidence limits.
Does generative AI improve self-regulated learning?
A meta-analysis of 17 comparative studies found a positive average effect, but results varied sharply: heterogeneity reached 94.85% and the prediction interval included harm as well as benefit. Almost all evidence came from higher education.
Read the next reportExplore the full source and evidence record
What this means for people
- Teachers and students may need to check task progress and the quality of reasoning directly; neither a positive AI survey score nor a low dashboard count reveals whether learning is deep, complete or stalled.
Global context
The course used DeepSeek-V2 and a locally designed learning-analytics tool with Chinese educational-technology majors. The observed profiles should not be assumed to transfer to other models, subjects, age groups, countries or assessment systems.
What the evidence does not yet show
- Exploratory sample of 27 students
- One Chinese university and one course
- Five enrolled students excluded
- Small and unequal clusters
- Different tasks across stages
- Profiles clustered independently at each stage
- No control group or causal estimate
- No learning-outcome measure
- Generated feedback quality not independently rated
What to watch next
- Larger preregistered replications
- Student-level transition analysis
- Independent feedback-quality ratings
- Code-quality and learning outcomes
- Controlled tests of stage-specific prompts
- Replication outside programming and China
Living evidence record
Impact record IAI-14AYBOI
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
27 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Humanities and Social Sciences Communications published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 27 October 2026
Research question, topics and regions
How 27 Chinese undergraduates used self- and peer-oriented learning analytics alongside a generative-AI programming assistant, and whether stage-specific behaviour profiles aligned with perceptions of AI
Topics: Generative AI · Programming education · Learning analytics · Student engagement · Higher education
Regions: China · Asia
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.