More GenAI use did not predict stronger independent reasoning in Saudi students
A seven-university study found that frequency, duration and weekly time using generative AI were not associated with scores on a new case-based assessment. University context explained substantially more variation—but the cross-sectional, unproctored study cannot establish why.
The short version
- What happened
- Among 431 health-informatics students at seven Saudi universities, 425 (98.6%) reported prior GenAI use; the mean score on a new 90-point case-based assessment was 57.14.
- Who it affects
- Teachers and course leaders should not treat frequent chatbot use as evidence that students are developing independent reasoning. Assessment design, curriculum and supervised practice remain central—and this study does not justify restricting useful AI support on its own.
- What remains unproven
- Convenience sampling and self-reported exposure limit generalisability and may introduce selection or recall bias.
Keep in mind
Convenience sampling and self-reported exposure limit generalisability and may introduce selection or recall bias.
The one-time design cannot establish direction or causality, and the remote assessment was not proctored.
The case assessment was newly developed for this study; its problem-solving subscale was less reliable than the overall instrument.
University was strongly associated with scores, but the study did not identify which institutional differences mattered.
Source analysis · 1 linked source. Primary analysed source date: 24 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Near-universal use was not the same as stronger assessed reasoning
Researchers surveyed 431 second-year-to-internship health-informatics students across seven Saudi universities between May and July 2026. They measured self-reported GenAI experience and administered a newly developed, expert-reviewed 30-item case assessment covering analysis, evaluation and problem-solving. Scores ranged to 90; the mean was 57.14 with a standard deviation of 15.41.
Duration, frequency and weekly hours of GenAI use did not predict higher scores. Academic-use intensity had a small negative correlation before adjustment, but the association disappeared after university was included. Formal AI training was also associated with a 3.48-point lower score after adjustment, an unexpected result that could reflect who sought training rather than an effect of training itself.[1]
The clearest signal was variation between universities
Average scores ranged from 45.8 to 68.0 between universities, and the reported intraclass correlation was 0.27. That makes curriculum, assessment culture and student mix more plausible targets for follow-up than raw AI-use time. It does not prove any particular university practice caused better performance.
The sample was voluntary and cross-sectional, the use measures were self-reported, and the remote assessment was not proctored. The instrument was new: overall reliability was strong, while its problem-solving subscale was less consistent. A longitudinal, independently proctored study using validated tasks and observed AI-use behaviour would materially change the assessment.[1]
The distinction that matters
What was measured—and what was not
The study linked reported use habits to one independent assessment at a single point in time.
Exposure
Survey GenAI duration, frequency, hours and academic use
Self-reported habits, not observed use quality
Outcome
Complete a 30-item case assessment
Mean 57.14/90 across 431 students
Interpretation
Adjust for student and university factors
No independent advantage from more use; university differences remained
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring generative ai: compare the finding and its evidence limits.
How are Australian engineering students actually using generative AI?
A peer-reviewed survey of 131 students at two Australian universities found broad but uneven use: coding was reported by 46.6% of one cohort and 7.1% of the other. Because the cohorts differed in size, year level and location, the study cannot identify which factor caused the gap.
Read the next reportExplore the full source and evidence record
What this means for people
- Teachers and course leaders should not treat frequent chatbot use as evidence that students are developing independent reasoning. Assessment design, curriculum and supervised practice remain central—and this study does not justify restricting useful AI support on its own.
Global context
Universities globally are trying to distinguish AI-assisted task completion from durable knowledge and reasoning. This study adds evidence from Saudi Arabia and health informatics, but its main lesson is about measurement: exposure counts alone say little about how students think without the tool.
What the evidence does not yet show
- Convenience sampling and self-reported exposure limit generalisability and may introduce selection or recall bias.
- The one-time design cannot establish direction or causality, and the remote assessment was not proctored.
- The case assessment was newly developed for this study; its problem-solving subscale was less reliable than the overall instrument.
- University was strongly associated with scores, but the study did not identify which institutional differences mattered.
What to watch next
- Longitudinal studies that test students before and after structured AI use
- Proctored, validated assessments alongside observed AI interaction logs
- Randomised comparisons of specific teaching and AI-literacy approaches
- Replication across disciplines, institutions and countries
Living evidence record
Impact record IAI-0AXNHVU
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
24 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Advances in Medical Education and Practice published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 24 October 2026
Research question, topics and regions
Generative-AI use and independently assessed higher-order cognitive performance
Topics: Generative AI · Higher-order thinking · Health informatics · Assessment · AI literacy
Regions: Saudi Arabia · Middle East · Higher education
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.