Agentic AI's average business gain did not predict what a new deployment would deliver
A peer-reviewed synthesis of 19 retail and supply-chain studies found a large positive average effect, but near-total variation between studies meant a new setting could plausibly see harm as well as benefit.
The short version
- What happened
- The peer-reviewed meta-analysis combined 19 studies with 5,292 observations across retail, supply chain, e-commerce and inventory contexts.
- Who it affects
- Workers and customers bear the consequences when a promising average is treated as a guaranteed local result. Pilots should measure correction work, errors and service outcomes alongside speed or cost.
- What remains unproven
- The 19 studies differed profoundly, producing 99.5% heterogeneity and a prediction interval spanning negative and positive effects.
Keep in mind
The 19 studies differed profoundly, producing 99.5% heterogeneity and a prediction interval spanning negative and positive effects.
The evidence base included journal articles, conference papers and preprints, and mixed simulations with real-world studies.
Operational domains did not explain differences reliably under the random-effects model; the e-commerce subgroup contained only three studies.
A selective-reporting adjustment reduced the pooled effect and removed conventional statistical significance.
Source analysis · 1 linked source. Primary analysed source date: 24 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The average was positive; the variation was the decision-relevant result
Researchers at Tshwane University of Technology combined 19 studies containing 5,292 observations across retail, supply-chain, e-commerce and inventory work. They converted reported outcomes to Hedges' g, a standardised effect size, and used a random-effects model with restricted maximum-likelihood estimation and a Hartung-Knapp adjustment for confidence intervals. The pooled result was large and positive: 1.58, with a 95% confidence interval from 0.26 to 2.89 and p=.022. Because g expresses differences in standard-deviation units, it does not translate directly into pounds saved, orders fulfilled or hours returned to workers. Those outcomes still need to be measured in each organisation.
That average is not a forecast for a new installation. Differences between studies were near total: I² was 99.5%, with tau-squared of 7.18. The review's 95% prediction interval—the range intended to describe a comparable future setting—ran from -4.20 to 7.36. In plain language, the available literature is consistent with substantial harm in one context and exceptional benefit in another. Architecture, task, data, implementation and whether a study was simulated or field-based may all matter more than the pooled headline.[1]
Domain labels did not explain the uncertainty reliably
The authors tested whether retail, supply chain, e-commerce and inventory studies produced different effects. A common-effect calculation reported large differences, but the more appropriate random-effects comparison did not reach conventional statistical significance (Q=6.73, three degrees of freedom, p=.081). E-commerce contained only three studies, so its estimate was treated as descriptive rather than conclusive.
The search was conducted from March to June 2026 and covered Scopus, IEEE Xplore, Web of Science and ACM, plus Google Scholar, arXiv, SSRN, selected research repositories, four hand-searched journals and citation chains. Google Scholar screening was limited to the first 200 results for each query. The review allowed journal articles, conference papers and preprints from 2021 to 2026. That breadth helps find emerging work, but it also combines evidence with different review status and operational realism. The paper's study-level assessment explicitly distinguishes simulations from real-world evidence.[1]
The distinction that matters
Why the pooled gain is not a deployment forecast
Three statistics answer different questions, and the widest one matters most for a new setting.
Pooled average
Combine 19 reported effects
Large positive mean: g=1.58
Between-study variation
Test whether studies agree
Near-total heterogeneity: I²=99.5%
New setting
Estimate a future true effect
Prediction interval spans harm to benefit
A local pilot needs a failure measure, not only an upside target
For retailers, logistics teams and workers, the sensible implication is not that agentic AI usually delivers a very large gain. It is that published results do not yet support a portable expectation. A pilot should predefine the relevant denominator and comparator: orders, stock-outs, delivery time, labour hours, customer resolutions or safety failures under the existing process versus the proposed system. It should also record human intervention and downstream errors, not only tasks the agent completed.
Selective reporting is another warning. A limit meta-analysis reduced the estimated effect to 1.26 with p=.066, no longer conventionally significant. The review was funded by Tshwane University of Technology and the authors declared no conflict. Confidence would rise with preregistered, real-world field experiments, consistent outcomes, architecture-specific comparisons and publication of null or harmful results. Until then, the strongest evidence-backed claim is variability—not a general business case.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 11 min 27 sec · Background explainer
UNESCO Talks: Interview with 2024 Nobel laureate Professor Daron Acemoglu
YouTube loads only when you choose; playback starts when you press play.
Economist Daron Acemoglu discusses how choices about AI affect workers, inequality and society. Pair the interview with our comparison of task exposure, hiring and skills evidence.
Evidence check: An expert interview from 2024, offering a perspective on policy and work rather than a measured forecast of job losses.
Published 2024-10-14. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the AI and jobs guidePhoto: Dylan Gillis / Unsplash · Licence
Continue the story
Read next
Continue exploring ai agents: compare the finding and its evidence limits.
Can an AI agent run a neighbourhood shop? Four models fell short
In an unreviewed Indian simulation, the best model earned about one-eighth of a scripted shopkeeper's profit. The test exposes language and accounting failures, not outcomes in a live business.
Read the next reportExplore the full source and evidence record
What this means for people
- Workers and customers bear the consequences when a promising average is treated as a guaranteed local result. Pilots should measure correction work, errors and service outcomes alongside speed or cost.
Global context
The review was conducted in South Africa and synthesised studies across operational settings rather than estimating a country-specific effect. Labour practices, infrastructure, data quality and regulation can change both gains and harms, so organisations should validate locally.
What the evidence does not yet show
- The 19 studies differed profoundly, producing 99.5% heterogeneity and a prediction interval spanning negative and positive effects.
- The evidence base included journal articles, conference papers and preprints, and mixed simulations with real-world studies.
- Operational domains did not explain differences reliably under the random-effects model; the e-commerce subgroup contained only three studies.
- A selective-reporting adjustment reduced the pooled effect and removed conventional statistical significance.
What to watch next
- Preregistered field experiments with operational and worker outcomes
- Architecture-specific comparisons against existing workflows
- Publication of null and harmful results
- Common measures of human intervention, error recovery and downstream cost
Living evidence record
Impact record IAI-076ARZX
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
24 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Industrial Artificial Intelligence published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 24 October 2026
Research question, topics and regions
Whether agentic AI improves operational performance in retail and supply-chain settings
Topics: AI agents · Retail · Supply chains · Business evidence · Operational performance
Regions: South Africa · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.