Can AI-generated product carbon footprints be trusted? A 269-declaration test found a 33% median error and extreme outliers
A peer-reviewed framework tested two proprietary Watershed systems. Clear material descriptions mapped far better than vague ones, while modeled footprints showed a wide error range—evidence for targeted human review, not automatic assurance.
The short version
- What happened
- The authors combined standards review with structured work involving more than 60 participants from eight enterprises across seven sectors and two verification firms, then piloted criteria on two proprietary Watershed systems.
- Who it affects
- Sustainability teams may cover more products, but still need experts to review ambiguous, material or unstable estimates.
- What remains unproven
- Six authors are employed by Watershed and the case studies evaluate two proprietary Watershed systems; core metrics include internal evaluations.
Keep in mind
Six authors are employed by Watershed and the case studies evaluate two proprietary Watershed systems; core metrics include internal evaluations.
The 269 environmental declarations provide end-to-end totals, not validated values for every modeled component.
Expert-labeled mapping datasets used small annotator pools without formal inter-rater agreement measurement.
Standards-conforming modelling under product-category rules was outside scope, and results should not be treated as certification.
Source analysis · 1 linked source. Primary analysed source date: 5 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The answer: useful automation needs benchmarks, uncertainty flags and a human stop rule
A peer-reviewed methodology paper finds that AI can assist product carbon-footprint work, but the quality of an output depends strongly on input clarity and the intended use. In a proprietary mapping system, 91.2% of well-specified material descriptions matched an expert-accepted proxy, compared with 59.7% of vague descriptions such as internal product codes or generic labels. In a second system that generated cradle-to-gate supply-chain models, comparison with 269 environmental product declarations produced a median relative error of 33%. The authors' central proposal is not an accuracy badge. It is a framework for deciding what can be checked once at system level, what needs output-level evidence and which results must go to a qualified reviewer.
The distribution matters more than the median alone. The middle 80% of relative errors ran from 6% to 98%. Signed error had a median of minus 2.3%, yet its 10th and 90th percentiles ranged from minus 49% to plus 98%. Thirteen runs exceeded plus 200%, with a maximum overestimate of 4,494%, pulling the mean signed error to plus 48%. A near-zero median bias can therefore coexist with individual footprints that are badly wrong. For procurement screening across thousands of low-materiality items, that may support triage. For a reported product claim, supplier contract or regulated disclosure, the same tail risk may be unacceptable.
The paper is unusually explicit that the case study is not proof the tested systems satisfy the proposed criteria. It demonstrates whether the criteria can be applied and whether they expose weaknesses. That distinction is essential because six authors work for Watershed Technology, the systems are Watershed products and the work was conducted through employment or advisory activity there. A Tsinghua University researcher is the remaining co-author. The commercial connection does not invalidate the data, but it raises the value of independent replication, public benchmarks and transparent labeling protocols.[1]
How the evaluation criteria were built
The authors began with ISO 14040 and 14044, the Greenhouse Gas Protocol Product Standard and the World Business Council for Sustainable Development's PACT framework. A product advisory group involved more than 60 people in sustainability, procurement and life-cycle-assessment roles from eight enterprises across seven sectors. Participants used the systems with their own procurement data and existing life-cycle studies in standardized workshops and recorded working sessions. Two professional verification firms, one in Europe and one in North America, were consulted separately. Draft criteria were then refined through pilots on real procurement records and early system versions.
The framework divides the work into use cases because one universal threshold would be too lenient for complex modelling and too strict for simple mapping. Case one links an unstructured description such as a steel grade to a life-cycle-inventory proxy. Case two constructs a synthetic bill of materials and process flow for a product without a direct database match. Standards-conforming modelling under a particular product-category rule is explicitly outside scope. Criteria cover transparent logic, comparison with expert judgment or documented reference values, uncertainty characterization and enough documentation for a reviewer to challenge an output.
That separation also recognizes that these are hybrid systems, not bare language models. Fixed boundary rules, controlled lookup tables and deterministic unit conversions may be validated at system level. Stochastic mapping or parameter generation can vary between runs and needs output-level checking. The authors tested repeat-run variability as a warning signal: flagging items whose emission factors varied by more than a factor of two across runs sent 22% for review while catching 88% of severe deviations in the available labeled set. That is promising triage evidence, not a universal threshold; performance will depend on the production data and the definition of severe error.[1]
The distinction that matters
Where an AI-assisted footprint can fail
The framework separates deterministic controls from stochastic choices and routes ambiguous or high-impact outputs to review.
Input
Check description specificity
Vague records map less reliably
System
Benchmark repeatability and expert agreement
Aggregate performance evidence
Output
Expose sources, assumptions and flags
A reviewer can challenge the estimate
Use
Apply a materiality threshold
Screening and disclosure need different assurance
The benchmark reveals ambiguity, wide tails and incomplete indicators
For automated mapping, the fall from 91.2% expert-accepted results on non-vague inputs to 59.7% on vague inputs shows that better source data may matter more than a model upgrade. Repeat-run entropy also rose with ambiguity. A procurement team should therefore preserve the original description, request supplier detail where material and expose alternative mappings rather than allowing the system to present one polished estimate as fact. Expert-accepted does not necessarily mean a unique correct answer: multiple proxies can be defensible, and the expert labels themselves came from small pools without formal inter-annotator-agreement measurement.
For full modelling, 86% of the 269 cases fell within 80% of the environmental-declaration benchmark. Component checks found 75% to 99% within a factor-of-two bound, depending on the component, with proxy mappings, hotspots and direct emissions stronger than key input rates. But the declarations provided product totals, not validated component values. The paper therefore says the component-level checks cannot yet justify waiving review. A flag that appears to identify the weak part of a footprint still needs calibration against reference values for that part, not only against the eventual total.
Environmental product declarations are not a perfect gold standard either. They can differ because of boundaries, allocation, data vintages and compliant methodological choices. The authors cite evidence that declarations for similar products and expert teams using the same cotton data can diverge materially. That explains some difference but does not make every AI result acceptable. The framework appropriately makes fitness for purpose the decision: the residual error among unreviewed outputs must be compared with the materiality tolerance of the actual use. A screening estimate and a public comparative claim require different evidence.[1]
What companies, auditors and the public should require
A company using AI-assisted footprinting should publish the intended use, benchmark population, error distribution and review policy. Each material output should retain sources, assumptions, proxy choices and data-quality indicators so an auditor can reproduce the path from procurement record to result. Ambiguous inputs should trigger a request for better data or review, not silent completion. Systems should be tested after model, database or prompt changes because stochastic behavior and reference inventories can shift. The paper says anonymized benchmark results are available on reasonable request, but the central performance metrics include internal Watershed evaluations; open test sets would make cross-provider comparison more credible.
For sustainability teams, automation could expand coverage beyond a small set of flagship products and help identify supply-chain hotspots. It does not remove the need for life-cycle expertise. A 33% median relative error and extreme positive outliers may be manageable for prioritizing supplier questions but could misdirect spending or misstate a product claim if treated as an assured figure. Verification firms and regulators should ask whether an uncertainty indicator has been validated for the task, how many outputs escape review and what upper confidence bound remains on severe errors among those accepted automatically.
Confidence would change with independent replication across providers, sectors and countries; public, versioned benchmarks; formal label-agreement studies; and component-level reference data. Standards-conforming footprints under product-category rules also need separate evaluation. The authors disclose using an LLM to synthesize technical documentation into the manuscript and say they reviewed and take responsibility for the content. The practical conclusion is narrower than 'AI can calculate carbon': AI can help organize and model incomplete supply-chain information, but trustworthy use depends on input quality, transparent evidence and review matched to the consequence of being wrong.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring ai assurance: compare the finding and its evidence limits.
What does the UK’s ten-year AI-for-space roadmap commit to now? Less than its 2036 vision suggests
The UK Space Agency has set priorities for assurance, Earth-observation analytics, autonomous operations and sovereign navigation resilience. The 73-page roadmap is a consulted strategy, not a new funding award or proof that systems are deployed.
Read the next reportExplore the full source and evidence record
What this means for people
- Sustainability teams may cover more products, but still need experts to review ambiguous, material or unstable estimates.
- Suppliers can be affected when a guessed proxy changes purchasing priorities; source descriptions and challenges should remain visible.
- Consumers, investors and regulators need to know whether a footprint is a screening estimate or a standards-conforming assured claim.
Global context
The criteria draw on international carbon-accounting standards and consultation with enterprises and verification firms across regions, but the tested systems and much of the implementation context are US-based. Database coverage, supplier records, electricity mixes and disclosure rules differ internationally.
What the evidence does not yet show
- Six authors are employed by Watershed and the case studies evaluate two proprietary Watershed systems; core metrics include internal evaluations.
- The 269 environmental declarations provide end-to-end totals, not validated values for every modeled component.
- Expert-labeled mapping datasets used small annotator pools without formal inter-rater agreement measurement.
- Standards-conforming modelling under product-category rules was outside scope, and results should not be treated as certification.
What to watch next
- Independent cross-provider replication.
- Public versioned benchmarks and label-agreement studies.
- Validation of component-level uncertainty flags.
- Separate tests for standards-conforming public disclosures.
Living evidence record
Impact record IAI-1IHNSZ2
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Journal of Industrial Ecology published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
How AI-assisted systems for mapping procurement records and generating cradle-to-gate product carbon footprints should be benchmarked, documented and routed for human review
Topics: Carbon accounting · Life-cycle assessment · Supply chains · AI assurance
Regions: United States · China · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.