When AI work looks faster, who counts the mistakes? A preprint tests the missing ledger
New analysis today of an unreviewed 23 October preprint: reanalysis shows that one productivity dataset supports different slowdown estimates, while 40 paired AI-review trials found that an inflated reliability claim cut checking and let wrong figures through. This does not measure human worker productivity.
The short version
- What happened
- The unreviewed preprint reanalyses 246 software issues completed by 16 developers and finds that the same randomized dataset supports a 19% geometric-mean slowdown or a 24% increase in total hours, depending on the summary used.
- Who it affects
- Workers may be judged against speed metrics that stop before checking, rework or customer harm appears. Recording those outcomes can change which tool or workflow looks productive, but this preprint does not estimate the effect on human jobs or wages.
- What remains unproven
- Unreviewed preprint by one independent author
Keep in mind
Unreviewed preprint by one independent author
Claude used substantially in research and writing
Author works on an AI-software company not evaluated in the paper
One drafter model and one reviewer model
Seeded rather than naturally occurring errors in the decisive experiment
40 paired drafts in the over-trust comparison
Prompt-assigned time and error costs
No human reviewers or workplace deployment
Wide or model-dependent bounds in parts of the reanalysis
Source analysis · 1 linked source. Primary analysed source date: 23 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 5 min The evidence, methods, implications and limitations
One dataset can produce more than one honest productivity headline
A new unreviewed preprint argues that an AI tool's visible time saving should be read beside the errors and later work it creates. The author reanalysed public data from METR's randomized study of experienced open-source developers: 246 issues completed by 16 people. METR's published specification estimated that access to early-2025 AI tools made work 18.8% slower. Using a different but defensible summary—the ratio of total mean time—the reanalysis estimates 24.0% more hours after adjusting for developers' forecasts, or 33.7% without that adjustment.
Those figures are not competing experiments. They describe the same observations with different estimands: the geometric mean asks about a typical proportional change, while the ratio of means is closer to total hours consumed. The preprint also calculates very wide partial-identification bounds: the data are compatible with AI helping on 2.6% to 82.1% of issues. That range has no sampling-error interval and should not be read as a precise success rate. It shows how little an average effect reveals about who or which tasks benefited.[1]
A small experiment made over-trust faster—and less reliable
The paper then tests one mechanism with AI agents. Draft briefs were built from US Securities and Exchange Commission filings dated after the models' training cutoff. A Claude Haiku model drafted the briefs and Claude Sonnet reviewed nine checkable figures. In the decisive preregistered comparison, the true draft-error rate was 30%. Across 40 paired drafts, the reviewer was either told that truth or told the drafter erred only 3% of the time.
Given the inflated reliability claim, the reviewer made 7.2 fewer lookups out of nine and allowed a wrong figure through in 22.5% of briefs, compared with none in the truthful condition. The reported difference was 22.5 percentage points, with a cluster-bootstrap 95% interval from 10 to 35 points. Every time the reviewer inspected a line exposing the seeded error, it corrected it; the failures arose because it checked less. A warning to treat the claimed track record as unverified did not restore the missing effort.[1]
The distinction that matters
Stopwatch versus ledger: what is being counted?
The paper separates visible task time from the checking and escaped errors that determine whether a workflow saved time overall.
Stopwatch
Time the visible task
Reports faster completion before downstream consequences
Verification
Record checks and corrections
Shows where review time is spent or skipped
Ledger
Add escaped-defect costs
Finds the defect cost that erases the apparent saving
The result is a measurement warning, not a verdict on workers
The experiment does not show that people behave like these agents or that workplace AI usually creates more errors than it saves. It tested one reviewer model, one structured financial task and costs written into the prompt. The defects were seeded; the planned comparison with natural errors could not run because the drafter made none in 140 briefs. The author also used Claude extensively to search, code, run the experiment and draft the paper, then reviewed the work. The paper is not peer reviewed.
Its useful proposal is procedural: before calling an AI workflow faster, name the time summary, record the checking performed and count mistakes that escape. Organisations can then ask a concrete question: how costly would each missed defect have to be before the apparent saving disappears? For procurement and workforce decisions, that ledger is more informative than a stopwatch alone—but it still needs testing with human reviewers, real consequences and independently replicated tasks.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 11 min 27 sec · Background explainer
UNESCO Talks: Interview with 2024 Nobel laureate Professor Daron Acemoglu
YouTube loads only when you choose; playback starts when you press play.
Economist Daron Acemoglu discusses how choices about AI affect workers, inequality and society. Pair the interview with our comparison of task exposure, hiring and skills evidence.
Evidence check: An expert interview from 2024, offering a perspective on policy and work rather than a measured forecast of job losses.
Published 2024-10-14. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the AI and jobs guidePhoto: Dylan Gillis / Unsplash · Licence
Continue the story
Read next
Continue exploring workplace ai: compare the finding and its evidence limits.
For small African firms, AI's first draft was only the start of the work
Interviews with 27 workers in Kenya and Nigeria found that client-ready output depended on verification, local knowledge and repair. The fieldwork was conducted in 2023, so this is evidence about early workplace practice—not current model performance.
Read the next reportExplore the full source and evidence record
What this means for people
- Workers may be judged against speed metrics that stop before checking, rework or customer harm appears. Recording those outcomes can change which tool or workflow looks productive, but this preprint does not estimate the effect on human jobs or wages.
Global context
The reanalysed developer study involved experienced contributors to mature open-source projects, while the agent experiment used US financial filings. Neither setting represents the full range of occupations, languages, institutions or labour conditions in which AI-assisted work is being introduced.
What the evidence does not yet show
- Unreviewed preprint by one independent author
- Claude used substantially in research and writing
- Author works on an AI-software company not evaluated in the paper
- One drafter model and one reviewer model
- Seeded rather than naturally occurring errors in the decisive experiment
- 40 paired drafts in the over-trust comparison
- Prompt-assigned time and error costs
- No human reviewers or workplace deployment
- Wide or model-dependent bounds in parts of the reanalysis
What to watch next
- Peer review and independent replication
- Human-reviewer experiments
- Natural rather than seeded error rates
- Whole-workflow time and rework records
- Task- and worker-level distribution of benefits
- Pre-registered comparisons across models and languages
Living evidence record
Impact record IAI-0U8Z8IT
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
28 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 28 October 2026
Research question, topics and regions
How productivity studies should count time, errors and verification—and whether misleading reliability information changes an AI reviewer's checking effort
Topics: AI productivity · Verification · Human oversight · Measurement · Workplace AI · Preprints
Regions: International · United States
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.