The odds stayed the same—why did AI advice change?
Across seven models and 240 synthetic high-stakes scenarios, complementary gain and loss wording changed the recommendation category in 32.4% of complete comparisons. Almost all shifts involved uncertainty rather than a direct yes-to-no reversal.
The short version
- What happened
- FrameBench-HPS generated 2,880 prompts from 240 synthetic scenarios across healthcare, finance, education or career, and legal or consumer-rights domains.
- Who it affects
- People asking about treatment, money, education or legal remedies may receive more hesitant or forceful language depending on equivalent wording. Interfaces should show the underlying probability and avoid presenting one phrasing as neutral.
- What remains unproven
- This is an unreviewed synthetic benchmark, not an evaluation of real decisions, outcomes or professional advice.
Keep in mind
This is an unreviewed synthetic benchmark, not an evaluation of real decisions, outcomes or professional advice.
Scenarios used four probability levels and controlled templates; broader uncertainty and multi-turn conversations were not tested.
Human rating covered 140 pairs, and inter-rater agreement was moderate or lower across the three qualities.
The paper reports no separate funding or competing-interest statement.
Source analysis · 1 linked source. Original source date: 21 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Equivalent probabilities changed commitment
Researchers built 240 synthetic scenarios spanning healthcare, finance, education or career, and legal or consumer rights. Each appeared at 20%, 40%, 60% and 80% positive-outcome probabilities, with neutral, gain and mathematically complementary loss wording. Seven models returned a recommendation category, risk score, actionability and confidence, producing 2,880 prompts before repeated runs and robustness checks.
Among 6,644 complete model-scenario-probability groups, gain versus loss wording changed the recommendation category 2,153 times, or 32.41%. Yet 98.1% of those changes involved the uncertain category; direct yes-to-no reversals occurred in only 42 groups, 0.63%. Loss framing raised model-assigned risk by 0.285 points on average and made answers more directive, even though the underlying odds and scenario facts were unchanged.[1]
People noticed the tone shift, but decision quality was not tested
Human annotators rated 140 paired responses, balanced across domains, probabilities and models. Loss-framed answers scored 0.27 points higher for risk presentation and 0.32 higher for directiveness. Agreement was moderate at best: ordinal alpha was 0.572 for risk, 0.480 for directiveness and 0.419 for expressed certainty. A held-out paraphrase test covered 32 scenarios across three model families and found the effect persisted beyond fixed templates.
The scenarios are synthetic and use only four probability levels. The study measures consistency, not whether the recommendation is correct, beneficial or harmful. Multi-turn context, personalised risk and real professional decisions were outside scope. Rewriting both frames into a canonical probability reduced sensitivity, which suggests a practical control, but it needs prospective testing with domain experts and real users before it can be treated as a safety intervention.[1]
The distinction that matters
One probability, three presentations
The facts stay fixed while the sentence around the probability changes.
Gain
State the chance of the favourable outcome
Foregrounds success
Loss
State the complementary chance of failure
Foregrounds harm
Compare
Hold scenario and probability constant
Measures framing sensitivity
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring decision support: compare the finding and its evidence limits.
Do AI models favour one cancer guideline?
Four models produced 6,000 choices across 15 conflicts between US immunotherapy guidelines and usually selected NCCN. But 10 conflicts involved one lung-cancer setting, and removing them weakened or reversed the pattern for most models.
Read the next reportExplore the full source and evidence record
What this means for people
- People asking about treatment, money, education or legal remedies may receive more hesitant or forceful language depending on equivalent wording. Interfaces should show the underlying probability and avoid presenting one phrasing as neutral.
Global context
Gain-loss framing is a human cognitive effect with cross-cultural variation. The model results span commercial systems but not languages or local risk norms; translation and professional standards could change the pattern.
What the evidence does not yet show
- This is an unreviewed synthetic benchmark, not an evaluation of real decisions, outcomes or professional advice.
- Scenarios used four probability levels and controlled templates; broader uncertainty and multi-turn conversations were not tested.
- Human rating covered 140 pairs, and inter-rater agreement was moderate or lower across the three qualities.
- The paper reports no separate funding or competing-interest statement.
What to watch next
- Pre-registered real-user studies
- Multilingual and culturally varied replications
- Tests of decision quality and downstream behaviour
- Independent evaluation of probability canonicalisation in deployed systems
Living evidence record
Impact record IAI-08L4N5F
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
22 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 22 October 2026
Research question, topics and regions
Whether equivalent gain and loss statements change language-model recommendations in synthetic high-stakes decisions
Topics: Decision support · Framing effects · Uncertainty · Healthcare · Finance · Legal advice
Regions: China · Hong Kong · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.