AI watched 115 hours of professional work—and still missed rejection
Fifty professionals used agents on real projects, but frontier models struggled to predict when people would modify or reject the output. The preprint tests understanding of collaboration, not productivity or job loss.
The short version
3 key points
- 1.Kolab records 115 hours from 50 professionals in 40 occupations, covering 100 problems, 508 workflow steps and more than 90 software tools.
- 2.Of 692 annotated AI contributions, professionals adopted 65.9%, modified 19.1% and rejected 15.0%.
- 3.The strongest full-set model reached macro-F1 0.492 when forecasting adoption, modification or rejection, while every model lost to a simple mode baseline on rating error.
Professionals often changed what left the chat
A new workplace study recorded what happened after AI produced an answer—not just what appeared in the chat window. Fifty professionals from 40 occupations completed two hour-long sessions on projects they had chosen from their own fields. Across 115 hours, researchers mapped 100 problems into 508 workflow steps and labelled 692 agent contributions. Professionals adopted 65.9% unchanged, modified 19.1% and rejected 15.0%. The finding matters because an apparently fluent answer can become useful, altered or discarded only after it meets the rest of a person's work.
The occupations spanned ten broad sectors and 21 countries, including architects, lawyers, journalists, engineers, educators and clinicians. Participants mainly worked with Claude but could use other agents, and their tasks had to require domain expertise, at least five interdependent steps and a concrete deliverable. This is closer to professional practice than a one-shot benchmark, although the sample was small and unusually AI-confident: 37 of 50 rated their generative-AI familiarity between eight and ten out of ten.[1]
Frontier models confused acceptance with understanding
Researchers then asked multimodal models to recognise or forecast whether a contribution would be adopted, modified or rejected. On the full video-only set, Claude Opus 5.5 reached macro-F1 of 0.676 when the human response was visible and 0.492 when it had to forecast that response. GPT-6 Astra was tested on fewer cases because of cost, so its headline score is not directly comparable with the full-set results. On the shared subset, Claude led recognition and GPT-6 led forecasting, but both still found modification and rejection harder than simple adoption.
The error pattern is the useful result. Gemini 3.1 Pro labelled 52% of true modifications and 71% of true rejections as adoption. Chat logs alone were especially weak at detecting rejection, because the decisive action often happened later in another program. Adding longer context did not reliably solve the problem: most video and workflow-graph changes stayed below 0.07 macro-F1, and none of the video gains was statistically significant. More recorded history was not automatically better understanding.[1]
The distinction that matters
What happened to 692 agent contributions
The study followed outputs beyond the chat window and recorded the professional's next action.
Adopt
Used substantially as given
65.9% (456)
Modify
Changed before use
19.1% (132)
Reject
Discarded or replaced
15.0% (104)
The benchmark cannot tell employers whether agents help
Participants rated collaboration and output quality highly, with roughly 85% of sessions scoring four or five on both measures. Yet every tested model had worse mean absolute error than simply predicting the most common rating. Rank agreement with people was also weak: the highest reported Spearman correlations were 0.379 for collaboration satisfaction and 0.352 for output quality. A system that routinely overpredicts approval could miss when a worker needs clarification, revision or control returned.
The preprint does not compare work with and without AI, measure time saved, audit deliverables independently or estimate employment effects. Recruitment through word of mouth, Prolific and Upwork, payment of $100 plus bonuses, heavy representation of advanced users and one default agent all limit generalisation. The study had ethics approval, removed personal identifiers and was partly supported by a University of Minnesota fellowship. Confidence would rise with larger occupational samples, independent quality assessment, low-familiarity users and controlled comparisons of error, time and workload. For employers, the message is practical: keeping a person in the loop is not enough if the system cannot recognise when that person has rejected its work.[1]
Evidence check
Unreviewed preprint; 50 paid, mostly AI-familiar participants; one main agent interface; self-chosen tasks; no no-AI comparator or independent productivity outcome.
Reporting basis: Source analysis. Original source date: 15 October 2026.
Explore the full source and evidence record
What this means for people
- Workers may have to repair or reject AI output outside the chat interface, where an agent that assumes success can add coordination and checking work.
Global context
Participants came from 21 countries, but 20 of 50 were in the United States and the study was run through a US university; workplace authority, language and software access vary elsewhere.
What the evidence does not yet show
- Unreviewed preprint; 50 paid, mostly AI-familiar participants; one main agent interface; self-chosen tasks; no no-AI comparator or independent productivity outcome.
What to watch next
- Larger preregistered workplace studies, independent deliverable scoring, novice users, comparisons with no-AI work and agent designs that respond appropriately to rejection.
Living evidence record
Impact record IAI-1NE128W
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
17 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 17 October 2026
Research question, topics and regions
Whether frontier multimodal models can recognise and predict how professionals adopt, modify or reject AI contributions in long workflows
Topics: Human-agent collaboration · Professional work · Computer-use agents · Workflows · Worker judgement
Regions: Global · United States · Asia · Africa · Europe · Latin America
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
UNESCO
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
AI, jobs & human choice · Background explainer
Published 2024-10-14 · 11 min 27 sec
UNESCO Talks: Interview with 2024 Nobel laureate Professor Daron Acemoglu
Economist Daron Acemoglu discusses how choices about AI affect workers, inequality and society. Pair the interview with our comparison of task exposure, hiring and skills evidence.
Evidence check: An expert interview from 2024, offering a perspective on policy and work rather than a measured forecast of job losses.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Work & Skills
AI on the factory floor: who gets the gains, and who carries the risk?
A robot may take over a task while a worker keeps responsibility when something goes wrong. IBM’s survey of 800 leaders points towards job redesign, but its promises of higher productivity and fewer staff are expectations, not measured results.
9 min · 2 sources
Work & Skills
What breaks when an AI agent meets a messy customer brief?
Across 95 simulated delivery jobs, the leading configuration resolved only 14 without an incident. Finding the missing facts was necessary—but still not enough to deliver the right result.
4 min · 1 source
Work & Skills
Should AI make workplaces clearer—or workers more visible?
Eleven US caregivers and employment-support professionals described hidden workplace expectations as an access barrier for autistic and IDD workers. Their technology ideas also exposed a surveillance risk: support data could be repurposed for monitoring or discipline.
7 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.