What do student prompts reveal about agency? The task phase mattered more than the deadline
A University of Sydney preprint classified 962 prompts from 105 first-year students. Most showed some student contribution, but explicit constraints were rare and self-reported journals cannot establish learning or the true prevalence of delegation.
The short version
- What happened
- Researchers classified 962 prompts from 105 students across ten journalled weeks; 84.2% contained some student contribution beyond forwarding a task, while 15.8% were classed as delegation.
- Who it affects
- Teachers gain a possible formative lens, not a validated grading scale.
- What remains unproven
- Unreviewed preprint from one Australian course and one low-code project context.
Keep in mind
Unreviewed preprint from one Australian course and one low-code project context.
Self-reported journals, declining participation and possible under-recording of delegation.
One-prompt coding excludes earlier context, attachments and learning outcomes.
Course phase, instruction, tools and assessment changed together.
Source analysis · 1 linked source. Primary analysed source date: 3 November 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 7 min The evidence, methods, implications and limitations
The direct finding: student agency changed with the work in front of them
A University of Sydney study suggests that the form of student agency visible in AI prompts changes across a software project rather than sitting at a fixed level for each learner. Researchers analysed 962 prompts recorded by 105 students in a first-year software-engineering course where generative AI was permitted and taught. They classified 152 prompts, or 15.8%, as simply delegating the problem. The remaining 810 showed some student contribution, ranging from a factual lookup to supplying constraints, diagnosing an artefact or framing design alternatives.
The strongest pattern tracked the development lifecycle. In week four, when students defined requirements, 48.1% of prompts explored a design space. During later build and verification work, diagnostic prompts became more common: after allowing for repeated observations from the same student, the odds were 2.88 times those in the initial analysis phase, with a 95% confidence interval from 1.41 to 5.86. That does not make later prompts better or worse. The engineering act changed as students moved from deciding what to build to finding why something they built did not work.[1]
How an invisible process was made countable
The paper introduces the Human Agency Depth Taxonomy, or HADT. Its six tiers begin with T0, where a student forwards a brief without adding a constraint, decomposition or evidence. T1 asks for generic knowledge; T2 requests a local component; T3 specifies system constraints; T4 brings an existing artefact or observation for diagnosis; and T5 frames alternatives or trade-offs while retaining the design decision. The tiers combine project-specific context, engineering scope and who defines the problem. The authors stress that they are context-sensitive categories, not a score to maximise.
One author coded every prompt without seeing student identities. The second independently coded a random 261-prompt sample, about 27% of the dataset. Agreement was 87.4%; unweighted Cohen's kappa was 0.82 and linearly weighted kappa 0.88. Most disagreements sat between T3 and T5, where a request built from student-supplied requirements could be read as constrained generation or design exploration. Both coders also taught and coordinated the course, giving them context while raising the possibility that outside coders could interpret the same prompts differently.[1]
The distinction that matters
What the six prompt tiers describe
The taxonomy classifies the engineering contribution visible in one student prompt. It does not rate learning or the AI output.
T0
Forward the task
AI defines the problem
T1–T2
Look up knowledge or request a component
Some student scoping is visible
T3
Specify system constraints
Only 5.3% of prompts
T4
Bring evidence for diagnosis
More common during build and verification
T5
Frame alternatives and trade-offs
48.1% in the requirements week
The denominator is students, prompts and weeks—not independent decisions
The cohort contained 145 students, 140 in their first university semester. Forty declared no AI use, leaving 105 who contributed the analysed prompts from weeks four to thirteen. Prompt counts declined from 187 in week four to between 58 and 75 in later weeks. Because prolific journal keepers could dominate a simple analysis, the authors used generalised estimating equations for their main phase comparisons. They separately examined 27 students who submitted a journal in at least half the eligible weeks, contributing 608 prompts, to ask how behaviour varied within individuals.
In that longitudinal subset, the median student used five of the six tiers. A phase-adjusted model attributed 32% of tier variation to differences between students and 68% to variation within the same students. That supports the paper's central interpretation: observed agency was substantially situational. But treating ordered tiers numerically for this variance estimate is an approximation, and the 27 regularly observed students may not represent the other 78 journal contributors, let alone the 40 who declared no use.[1]
Deadlines looked important until the analysis respected student clusters
At the raw prompt level, delegation was more common in deadline weeks: 74 of 374 prompts, or 19.8%, compared with 78 of 588 prompts, or 13.3%, in other weeks. The unadjusted odds ratio was 1.61. Once the analysis allowed prompts from the same student to be correlated, the estimate fell to 1.28 and the 95% confidence interval crossed no effect, from 0.93 to 1.75. A paired comparison inside the 27-student longitudinal subset was also inconclusive.
This negative result blocks a simple story that pressure makes students hand their thinking to a model. The data point more clearly to the work being done. Yet phase, teaching and assessment changed together: students received different AI instruction, used different tools and produced different deliverables across the semester. The study cannot isolate whether lifecycle stage caused the prompt shifts, or whether instruction and assignment design produced the same pattern.[1]
Explicit constraints were rare, but the prompts do not explain why
Only 51 of 962 prompts, or 5.3%, were placed in T3, where the student requests an artefact and specifies the wider constraints it must satisfy. The authors interpret this as a possible knowledge bottleneck: novices may have been shown how to constrain an AI response but still lacked the domain knowledge to name the right data model, platform limit or acceptance criterion. That is plausible in a course assuming no prior programming, but the data cannot distinguish lack of knowledge from a choice to omit information already available to the model.
Follow-up prompts were deliberately coded on their text alone. A short message such as 'fix it' counted as delegation even if earlier turns contained rich project context. Attachments and the journal's reflection fields were not analysed. HADT measures what a student contributes visibly in that one prompt; it does not measure the whole conversation, cognitive effort, conceptual understanding, output accuracy or learning. Schools should not turn the 84.2% figure into a claim that most AI use was educationally beneficial.[1]
What teachers can use now—and what would change the assessment
For teachers, the taxonomy offers a language for formative review. A project team could discuss whether prompts at a given stage show the intended human contribution, while recognising that lookup can be appropriate during implementation and diagnosis during testing. The authors advise against awarding marks for higher tiers because students could optimise prompts to satisfy the rubric. Keeping journals unmarked may have encouraged honesty, although selective reporting remains possible and delegated interactions may be the least likely to be disclosed.
Confidence would grow if independent researchers reproduced the coding across subjects, institutions, countries and experience levels; analysed complete conversation histories; linked profiles to validated learning outcomes; and preregistered comparisons. It would weaken if outside coders cannot reproduce the categories or recorded prompts differ systematically from actual use. The study was funded by a University teaching-and-learning grant and received ethics approval. It is a timely, methods-rich preprint, not peer-reviewed evidence that HADT should determine grades or police students. It is not Breaking.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 8 min 30 sec · Background explainer
Teacher to Teacher: Is AI reshaping education for the better?
YouTube loads only when you choose; playback starts when you press play.
A conversation about how teachers can guide AI use while preserving pupils’ judgement. Watch for the difference between helping a learner think and supplying an answer.
Evidence check: Teachers’ perspectives provide context. They do not establish that a particular AI tool improves learning outcomes.
Published 2025-01-23. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the education evidencePhoto: Vitaly Gariev / Unsplash · Licence
Continue the story
Read next
Continue exploring generative ai: compare the finding and its evidence limits.
Did ChatGPT improve writing scores in a 16-week Taiwan course?
Twelve university students used ChatGPT throughout a five-stage inquiry-writing course. Their closed-book writing scores rose by 2.79 points, but the change was not statistically significant and the study had no comparison group.
Read the next reportExplore the full source and evidence record
What this means for people
- Teachers gain a possible formative lens, not a validated grading scale.
- Students should not be labelled from one prompt; most variation occurred within the same students.
- Assessment designers need evidence of process and learning, not just polished AI-assisted artefacts.
Global context
Universities are trying to distinguish a student's contribution from the sophistication of an AI-generated artefact. Existing cognitive taxonomies often describe the task or output. HADT instead describes problem ownership visible in a prompt. Its value depends on independent validation and keeping process evidence separate from claims about learning, integrity or authorship.
What the evidence does not yet show
- Unreviewed preprint from one Australian course and one low-code project context.
- Self-reported journals, declining participation and possible under-recording of delegation.
- One-prompt coding excludes earlier context, attachments and learning outcomes.
- Course phase, instruction, tools and assessment changed together.
What to watch next
- Independent cross-course replication.
- Links to measured learning.
- Full-conversation coding.
- Whether rubric use changes prompt writing.
Living evidence record
Impact record IAI-0UGMBPQ
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
How much engineering contribution is visible in first-year students' AI prompts, how that changes across a project, and whether a prompt taxonomy can support assessment without claiming to measure learning
Topics: Higher education · Student agency · Assessment · Generative AI · Software engineering
Regions: Australia · Asia-Pacific
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.