One LLM run can make a prompt look better than it is: repeated tests found randomness dominated some results
A peer-reviewed experiment across three causal-map tasks found that model-to-model and run-to-run variation could overwhelm controllable prompt choices. The result is a warning for researchers and teams that benchmark an AI workflow only once.
The short version
- What happened
- The researchers compared three algorithms for combining causal maps, using six model versions from the DeepSeek, Gemini and GPT families and five repeated runs for each experimental condition.
- Who it affects
- Researchers, auditors and workplace teams can make a poor tool look dependable if they report only one run. Repetition makes unstable cases visible before a model shapes a study result or operational decision.
- What remains unproven
- Three specialised causal-map combination algorithms
Keep in mind
Three specialised causal-map combination algorithms
Six model versions from three provider families in the reported factorial comparison
Five completed replications per condition, with some high-variance cells needing more
No human decision, factuality, productivity or safety outcome
Proprietary API behaviour may change after the recorded access window
Source analysis · 1 linked source. Primary analysed source date: 27 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The same controlled setup did not always give the same answer
A single successful LLM test can exaggerate how much a prompt, model setting or workflow contributed to the result. In a peer-reviewed experiment published on 27 October, researchers used LLMs to combine causal maps—directed graphs that represent possible relationships between factors. They varied prompt components and decoding approaches, repeated conditions five times and measured how much of the result could be attributed to choices the researcher controlled rather than residual variation between runs. That matters for consequential decisions in research, public policy and operational planning.
The study covered three combination algorithms and six model versions from the DeepSeek, Gemini and GPT families. Its factorial design tested prompt elements together instead of changing one element at a time. Across the experiment, model family and version mattered, prompt effects depended on the metric, and no decoding strategy was consistently best. For the most sequential algorithm, run-to-run randomness accounted for about 84% to almost 96% of variance for the two DeepSeek versions and about 73% to more than 81% for the two Gemini versions.[1]
Five repetitions were enough to reveal the problem—not always to estimate it precisely
The team ran 4,800 tests for its open-weight-model decoding comparison and thousands more across the wider factorial designs. Yet a confidence-interval check found that a small subset of high-variance conditions would need many more repetitions for a precise estimate. In one Algorithm 3 recall analysis, the median estimated total among varying conditions reached 7,684 runs for DeepSeek Chat v3.1 and 5,283 for GPT-4o. Those large budgets are estimates derived from the observed variance, not a universal minimum for evaluating an LLM.
The practical lesson is to match the replication plan to the decision. A research paper comparing prompts should preserve the exact model version, access dates, decoding settings and normalization choices, then report the spread of results. A workplace team testing whether an assistant is reliable should repeat realistic tasks, retain failures and examine whether a favourable average hides unstable cases. Turning temperature down to zero is not, by itself, proof of reproducibility.[1]
The distinction that matters
What the experiment separated
The factorial design asked which source of variation actually moved each performance measure.
Prompt
Examples, explanations and representations
Effects changed by model and metric
Decoding
Greedy, beam and contrastive strategies
No universally best method
Repeated run
Same condition executed again
Up to almost 96% of variance in one workflow
The finding is important, but its scope is deliberately narrow
The tasks joined causal maps about obesity and suicide prevention; they did not measure writing quality, factuality, coding success or decisions made by people using the output. The three algorithms also differed in how many sequential LLM steps they required, which helps explain why errors could cascade in the most variable workflow. Results from proprietary models may change as providers update systems behind an API.
The authors released their data and scripts, reported no specific funding and declared no competing interests. Stronger follow-up would reproduce the design on other tasks, use independently hosted model snapshots where possible and connect output variability to human decisions or downstream harms. For now, the paper supports a concrete research standard: a benchmark should describe a distribution, not just its most convenient run.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
Google DeepMind · 1 min 52 sec · Background explainer
Protein folding explained
YouTube loads only when you choose; playback starts when you press play.
A short introduction to why a protein’s shape matters. It provides background for understanding what AI structure prediction contributes to biological research.
Evidence check: This is the developer’s explanation. Predicting a structure does not by itself establish a treatment’s safety or benefit for patients.
Published 2020-11-30. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current researchContinue the story
Read next
Continue exploring large language models: compare the finding and its evidence limits.
Can AI design better optimisation rules?
A Nature Machine Intelligence paper reports that a structured LLM framework outperformed prior LLM approaches across 36 combinatorial-optimisation benchmarks and produced feasible algorithms for four new port-logistics problems. It remains an offline benchmark, not a live operations result.
Read the next reportExplore the full source and evidence record
What this means for people
- Researchers, auditors and workplace teams can make a poor tool look dependable if they report only one run. Repetition makes unstable cases visible before a model shapes a study result or operational decision.
Global context
The models came from US, Chinese and European providers, while the research team was based in the United States and France. The reproducibility problem travels across providers, but the exact variance depends on the task, model version and deployment environment.
What the evidence does not yet show
- Three specialised causal-map combination algorithms
- Six model versions from three provider families in the reported factorial comparison
- Five completed replications per condition, with some high-variance cells needing more
- No human decision, factuality, productivity or safety outcome
- Proprietary API behaviour may change after the recorded access window
What to watch next
- Independent replication on coding, writing and decision-support tasks
- Version-pinned or self-hosted model comparisons
- Prespecified repetition and uncertainty reporting in LLM benchmarks
- Links between output variance and consequential human decisions
Living evidence record
Impact record IAI-0W0J332
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
27 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what PLOS One published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 27 October 2026
Research question, topics and regions
How much prompt design, decoding, model choice and residual run-to-run variability shape LLM-assisted causal-map combination results
Topics: Research methods · Reproducibility · Large language models · Prompt evaluation · Conceptual modelling
Regions: United States · France · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.