Does a bigger medical LLM justify its energy use? Accuracy rose, but power costs rose faster
Across 13 open models and 3,773 medical question-answering samples, larger models were generally more accurate and more energy intensive. One fine-tuned 14-billion-parameter model reached 81.03% accuracy while consuming 0.089 kWh in the reported inference experiment—not a clinical deployment.
The short version
- What happened
- Researchers evaluated 13 open models from the Qwen, Gemma and Llama families on 3,773 questions from four medical benchmark sources, in zero-shot and fine-tuned settings.
- Who it affects
- Clinical and research teams can compare task accuracy with measured energy rather than treating model size as a proxy for quality.
- What remains unproven
- The work is a preprint and workshop paper, not a peer-reviewed clinical validation study.
Keep in mind
The work is a preprint and workshop paper, not a peer-reviewed clinical validation study.
All measurements used one A40 GPU, one software setup and no specialised inference optimisation.
The final comparison excludes fine-tuning energy and does not include pretraining, hardware manufacture or full service overheads.
Carbon estimates depend on a German 2025 grid factor and assumed power usage effectiveness.
Medical benchmark accuracy is not patient outcome, diagnostic safety, calibration or fairness.
Source analysis · 1 linked source. Primary analysed source date: 2 November 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The finding: bigger models answered more benchmark questions correctly, at a steeper energy cost
A study of 13 open language models found a recurring trade-off in medical question answering: larger models usually achieved higher accuracy, but the measured energy used during inference rose faster. The researchers tested models from the Qwen, Gemma and Llama families, ranging from 270 million to 14 billion parameters. Their evaluation contained 3,773 questions drawn from MedQA, MedMCQA, PubMedQA and medical subjects in MMLU. Each model was evaluated without task-specific training and after parameter-efficient LoRA fine-tuning.
The best headline result came from the fine-tuned Qwen3-14B model. It answered 81.03% of the benchmark items correctly and consumed 0.089 kilowatt-hours during the reported inference experiment, corresponding to an estimated 0.034 kilograms of carbon dioxide under the study's assumptions. Those are not hospital-wide energy figures, annual totals or emissions from building the model. They are measurements for a bounded run on a single Nvidia A40 GPU, with carbon calculated from a German 2025 grid factor and a data-centre power-usage multiplier.[1]
How the researchers measured the trade-off
The team selected related model families so that it could compare scale within a lineage instead of mixing unrelated architectures. Models answered the same four-option or closed-answer medical questions under controlled prompts. The zero-shot condition measured their off-the-shelf behaviour. The fine-tuned condition used LoRA, which adapts a smaller set of parameters rather than retraining every model weight. Accuracy was the share of benchmark answers marked correct; the work did not score explanation quality, calibration or the clinical consequence of a wrong answer.
Energy was logged with CodeCarbon, which sampled hardware consumption every 15 seconds. The researchers report direct inference electricity and then estimate carbon emissions using a grid-intensity value for Germany and an assumed power usage effectiveness factor for the supporting data centre. That makes the energy readings tied to the test hardware, while the carbon figures are even more context-dependent. The same computation could have a different emissions estimate in a region or time period with a different electricity mix.
The final family comparisons show why a single accuracy number is insufficient. Moving from the smallest to the largest tested Qwen model increased accuracy by about 64% while energy rose 187%. For Gemma2, the corresponding changes were 34% and 95%; for Gemma3, 134% and 146%; and for Llama3, 61% and 186%. These are relative changes within the authors' selected models and conditions. They do not mean that every parameter increase has a fixed energy penalty or that one family is universally more efficient.[1]
The distinction that matters
Relative changes within four tested model families
Larger models improved benchmark accuracy, while inference energy generally rose more sharply. Percentages compare the smallest and largest tested model in each family.
Qwen
+64% accuracy
+187% energy
Gemma2
+34% accuracy
+95% energy
Gemma3
+134% accuracy
+146% energy
Llama3
+61% accuracy
+186% energy
Fine-tuning helped accuracy, but its preparation energy was left outside the final comparison
LoRA fine-tuning generally improved accuracy and can make a smaller model competitive with a larger off-the-shelf model. That creates a practical option for organisations: choose a smaller model, adapt it to a defined task and potentially reduce energy per inference. Yet the accounting boundary matters. The paper's final energy-accuracy comparison focuses on inference and excludes the electricity used for fine-tuning. A service with a very large number of queries may spread that one-off adaptation cost across many uses; a short pilot may not.
The study also does not include the energy used to pretrain the underlying models, manufacture hardware, store datasets or operate the rest of a clinical application. Conversely, it does not count possible system savings if a model reduces other work. A responsible procurement decision would therefore measure the complete service: task-specific accuracy, uncertainty, throughput, hardware utilisation, adaptation cost, expected query volume and the local electricity mix.
For clinicians and patients, efficiency cannot replace validity. A model that saves electricity but gives unsafe advice is not an acceptable optimisation. These benchmark datasets test factual and exam-like medical question answering, not diagnosis at the bedside, communication with a patient or performance across demographic groups. The work is best read as a measurement study about model selection, not evidence that any tested model should be used for care.[1]
Evidence limits, funding and what would change the assessment
All runs used one A40 GPU and no specialised inference optimisation. Quantisation, batching, serving software and newer accelerators can change both speed and energy. The set of 13 models is useful but narrow, and benchmark accuracy can conceal different error patterns. Because the study reports aggregate accuracy, it does not tell a hospital which mistakes would harm a patient or whether a smaller model is reliable for a particular local population.
The authors disclose support from the European Union and the German state of North Rhine-Westphalia through the LLM4KMU project, identified as EFRE-20801074. They also disclose use of AI tools in preparing the work. The paper was submitted on 2 November 2026 and accepted to the Green-AI Workshop at CIKM 2026. That provides a venue for discussion but is not equivalent to a peer-reviewed journal or full conference paper.
Confidence would improve with replication on multiple hardware platforms and electricity grids, end-to-end accounting that includes fine-tuning, and clinical task evaluations with patient-relevant outcomes and error analysis. Until then, the study supports a modest operational lesson: do not assume the most accurate large model is the most efficient choice. Measure the smallest system that meets a validated task requirement, and state the accounting boundary. The preprint is current research, not a breaking event, so it appears as Latest rather than Breaking.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring medical ai: compare the finding and its evidence limits.
How much does MedGemma improve medical AI?
A peer-reviewed study reports gains over similarly sized base models across medical image questions, chest X-ray classification and simulated agent tasks. The evidence comes from benchmarks and small specialist reviews, not prospective clinical deployment or patient outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- Clinical and research teams can compare task accuracy with measured energy rather than treating model size as a proxy for quality.
- Patients should not interpret benchmark performance as evidence of safe diagnosis or treatment; no patients or clinical outcomes were evaluated.
- Smaller, adapted models may reduce repeated inference energy, but organisations need to include adaptation and service overheads in their own accounting.
Global context
AI energy use depends on where and how a model runs. Regions with carbon-intensive grids face different emissions from the same electricity demand, while organisations with scarce accelerators may prioritise throughput and memory. The study's open-model comparison is relevant beyond Germany, but its carbon values should not be exported unchanged.
What the evidence does not yet show
- The work is a preprint and workshop paper, not a peer-reviewed clinical validation study.
- All measurements used one A40 GPU, one software setup and no specialised inference optimisation.
- The final comparison excludes fine-tuning energy and does not include pretraining, hardware manufacture or full service overheads.
- Carbon estimates depend on a German 2025 grid factor and assumed power usage effectiveness.
- Medical benchmark accuracy is not patient outcome, diagnostic safety, calibration or fairness.
What to watch next
- Independent measurements on other GPUs, quantisation levels and serving stacks.
- End-to-end accounting that includes fine-tuning and expected lifetime query volume.
- Medical evaluations that analyse consequential errors, calibration and subgroup performance.
- Whether smaller adapted models retain their efficiency in live, monitored services.
Living evidence record
Impact record IAI-133M3TE
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Energy and accuracy in medical language models
Topics: Energy use · Medical AI · Open models · Fine-tuning · Green AI
Regions: Germany · European Union · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.