Ten LLMs faced complex colorectal-cancer decisions; none established a stable clinical lead
Across 1,080 primary responses to synthetic cases, full concordance ranged from 49.1% to 63.0%. Stricter scoring, counterfactual changes and confident wrong answers exposed why a fluent plan is not clinical validation.
The short version
- What happened
- Researchers tested ten LLMs three times on 36 primary synthetic scenarios, producing 1,080 responses. Fully concordant plans ranged from 49.1% to 63.0%, with no statistically robust model effect.
- Who it affects
- For patients and oncology teams, an apparently polished treatment sequence can hide an omitted condition or a failure to react when one decisive fact changes. The study supports supervised testing of decision support, not direct use of a general-purpose model for treatment choices.
- What remains unproven
- Synthetic vignettes
Keep in mind
Synthetic vignettes
36 independent primary scenarios
Ten prompt-development cases remained in the benchmark
One zero-shot prompt
No retrieval or multimodal record
Models tested at one September 2026 time point
Consensus-derived reference retains judgment
No clinician-plus-LLM arm
No patient outcomes
Source analysis · 1 linked source. Primary analysed source date: 28 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
A benchmark built around linked treatment decisions
Colorectal cancer that has spread to the peritoneum forces teams to connect several difficult choices: referral, systemic therapy, restaging, surgery, intraperitoneal treatment and surveillance. The researchers translated a 2026 expert-consensus pathway into 48 synthetic vignettes: 36 primary scenarios and 12 counterfactual cases arranged as six pairs where one decisive clinical feature changed.
Ten LLMs answered every vignette three times using one frozen zero-shot prompt. Two blinded colorectal surgeons validated the reference classifications. Across 1,080 primary responses, 56.4% were fully concordant under the structured scoring rule, 10.8% partial and 32.8% discordant. Model-level full concordance ran from 49.1% to 63.0%, but the statistical tests did not support a reliable ranking.[1]
Small wording changes were not the main problem
The result weakened when the evaluation asked whether the whole recommendation was semantically complete: full concordance fell to 45.5%. Counterfactual tests were also unstable. No model responded in the expected direction for more than three of six paired changes in a majority of runs, even when the altered fact should have changed management.
Confidence did not reliably protect the reader. Twenty-eight per cent of semantically discordant answers expressed certainty of at least 90. The 21 prespecified clinically consequential flags came from only five independent scenarios, so they are warning patterns rather than 21 observed harms. No real patient record, multidisciplinary team, clinician-plus-model workflow or patient outcome was tested.[1]
The distinction that matters
Why one benchmark score was not enough
The study checked agreement, response to a changed clinical fact and the relationship between confidence and correctness.
Concordance
Compare plans with an expert-consensus pathway
49.1%–63.0% full by model
Counterfactual
Change one decision-critical feature
No stable majority success beyond 3 of 6 pairs
Certainty
Compare stated confidence with correctness
28% of discordant answers still claimed at least 90 certainty
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring large language models: compare the finding and its evidence limits.
Did an LLM help reviewers find cancer-treatment skin toxicity?
In a 160-patient retrospective study, LLM-assisted reviewers extracted skin toxicities from clinical notes more accurately and consistently than unassisted reviewers. It did not test real-time diagnosis, treatment decisions or patient outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- For patients and oncology teams, an apparently polished treatment sequence can hide an omitted condition or a failure to react when one decisive fact changes. The study supports supervised testing of decision support, not direct use of a general-purpose model for treatment choices.
Global context
The models were accessed through provider endpoints and the clinical reference came from an international consensus pathway, but treatment access and practice vary. A benchmark in one disease pathway does not establish performance across oncology or health systems.
What the evidence does not yet show
- Synthetic vignettes
- 36 independent primary scenarios
- Ten prompt-development cases remained in the benchmark
- One zero-shot prompt
- No retrieval or multimodal record
- Models tested at one September 2026 time point
- Consensus-derived reference retains judgment
- No clinician-plus-LLM arm
- No patient outcomes
What to watch next
- Prospective clinician-plus-LLM trials
- Patient-relevant outcomes
- Guideline-grounded and retrieval systems
- Version-stability testing
- More counterfactual scenarios
- Privacy and audit trails in clinical deployment
Living evidence record
Impact record IAI-1TWHCKS
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
28 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Medicine published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 28 October 2026
Research question, topics and regions
How reliably ten contemporary LLMs reproduce an expert-consensus pathway for colorectal cancer with peritoneal metastases
Topics: Colorectal cancer · Clinical decision support · Large language models · Oncology · Benchmarking · Human oversight
Regions: China · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.