Can AI teams estimate the energy cost of every generated token?
Across 374 held-out executions on NVIDIA GH200 and GB300 systems, a memory-traffic model stayed within 3.73% on average—but only for a narrow set of dense-model decoding conditions.
The short version
- What happened
- The researchers tested Qwen2.5-7B, Qwen2.5-14B and Mistral-7B on NVIDIA GH200 and GB300 hardware, including one two-GPU configuration.
- Who it affects
- More transparent serving estimates could help organizations choose lower-energy configurations, but incomplete accounting can still understate the electricity and emissions associated with a service.
- What remains unproven
- The unreviewed work covers a small set of dense models and NVIDIA accelerators under controlled decoding conditions; it excludes several major system and lifecycle energy sources.
Memory traffic predicted decode energy better than shortcuts
Researchers from Texas Tech, the Texas Advanced Computing Center and NVIDIA propose a falsifiable way to estimate the GPU energy used for each generated token. Their model links energy to high-bandwidth-memory traffic, model size, sequence length and fixed overheads. It was fitted and tested across Qwen2.5 models with 7 and 14 billion parameters and Mistral-7B on GH200 and GB300 hardware, with five executions for each measured cell.[1]
Who it affects
More transparent serving estimates could help organizations choose lower-energy configurations, but incomplete accounting can still understate the electricity and emissions associated with a service.[1]
Keep in mind
The unreviewed work covers a small set of dense models and NVIDIA accelerators under controlled decoding conditions; it excludes several major system and lifecycle energy sources.
Source analysis · 1 linked source. Original source date: 20 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Memory traffic predicted decode energy better than shortcuts
Researchers from Texas Tech, the Texas Advanced Computing Center and NVIDIA propose a falsifiable way to estimate the GPU energy used for each generated token. Their model links energy to high-bandwidth-memory traffic, model size, sequence length and fixed overheads. It was fitted and tested across Qwen2.5 models with 7 and 14 billion parameters and Mistral-7B on GH200 and GB300 hardware, with five executions for each measured cell.
The complete model was tested on 374 held-out energy executions and 177 held-out memory profiles. Across held-out condition averages, predictions stayed within 3.73%; on the GB300 tests highlighted by the authors, the largest mean cell error was 2.14%. Models using only elapsed time or only bytes transferred exceeded 10% error in those comparisons, suggesting that both memory movement and fixed costs matter.[1]
The result is a component estimate, not a data-centre footprint
For developers and infrastructure teams, a validated per-token estimate could help compare serving configurations before running a full energy audit. It might also make energy reporting more specific than a single model-wide average. But the paper measures GPU-board energy during autoregressive decoding; it excludes host processors, networking, storage, cooling, training and embodied emissions.
The tested workload was dense BF16 inference with full-history decoding. Multi-node systems, pipeline parallelism, mixture-of-experts routing and other model families were not covered. One author lists NVIDIA, the hardware supplier, as an affiliation, and the preprint gives no separate conflict statement. Independent replication on additional hardware and end-to-end facility measurements would change the assessment.[1]
The distinction that matters
Where a per-token energy estimate begins and ends
The proposed law models one part of the serving stack.
Weights and cache
Estimate memory bytes moved
Captures sequence and model-size effects
GPU board
Combine traffic with fixed overhead
Predicts decode energy per token
Whole system
Add CPU, network and cooling
Not measured in this study
Evidence check
The unreviewed work covers a small set of dense models and NVIDIA accelerators under controlled decoding conditions; it excludes several major system and lifecycle energy sources.
Reporting basis: Source analysis. Original source date: 20 October 2026.
Continue the story
Read next
Continue exploring data centres: compare the finding and its evidence limits.
Can MENA connect AI data centres without straining power and water?
Only with site-level evidence and joined-up planning. A new World Economic Forum and SRMG Think report says the region can connect projects faster than mature markets, but its heat, water scarcity and volatile AI loads turn speed into a resilience test.
Read the next reportExplore the full source and evidence record
What this means for people
- More transparent serving estimates could help organizations choose lower-energy configurations, but incomplete accounting can still understate the electricity and emissions associated with a service.
Global context
The experiments used US research and company infrastructure. Electricity mix, cooling demand and operational emissions vary substantially by region and facility.
What the evidence does not yet show
- The unreviewed work covers a small set of dense models and NVIDIA accelerators under controlled decoding conditions; it excludes several major system and lifecycle energy sources.
What to watch next
- Independent hardware replication, tests on mixture-of-experts and multi-node serving, facility-level validation and reporting that separates GPU-board energy from total operational and lifecycle impacts.
Living evidence record
Impact record IAI-1BG5CRM
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
21 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 21 October 2026
Research question, topics and regions
Whether high-bandwidth-memory traffic can predict GPU-board energy per generated token during language-model decoding
Topics: AI energy · Inference · Tokens · Data centres · GPU memory
Regions: United States · North America · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.