A pathology AI answered almost as well without the slide—was it reading the image?
In an unreviewed controlled evaluation, SlideChat scored 74.82% with a pathology slide and 74.41% without it on 1,227 multiple-choice items. The result exposes benchmark shortcuts, not clinical performance.
The short version
3 key points
- What changed
- SlideChat answered 74.82% of 1,227 multiple-choice pathology questions correctly with the slide and 74.41% after the slide was removed; the difference was statistically indistinguishable from zero.
- Why it matters
- Pathologists and clinical teams should ask whether an answer changes when the image changes, not accept a benchmark score as proof that the model inspected tissue.
- What to watch
- Independent replication on external, leakage-controlled slides and free-text tasks
The benchmark answer barely depended on the slide
An unreviewed evaluation found that one pathology vision-language assistant could keep nearly all of its multiple-choice accuracy after researchers removed the pathology slide. On 1,227 re-encoded SlideBench-VQA items, SlideChat scored 74.82% with the slide and 74.41% without it. A mismatched slide also produced no significant gap. Its underlying text model scored 43.52%, close to the benchmark's 40.7% most-frequent-letter baseline, suggesting that instruction-tuning taught answer-pattern shortcuts that did not require visual evidence.[1]
Who it affects
Pathologists and clinical teams should ask whether an answer changes when the image changes, not accept a benchmark score as proof that the model inspected tissue.[1]
Keep in mind
The manuscript is an unreviewed preprint using retrospective public data and released checkpoints.
The slide-grounding probe covered 1,227 multiple-choice items and two assistants; free-text reporting and other systems may behave differently.
Cross-model feature comparisons are confounded by architecture, data and aggregation differences, and no prospective clinician study was performed.
Source analysis · 1 linked source. Original source date: 19 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
The benchmark answer barely depended on the slide
An unreviewed evaluation found that one pathology vision-language assistant could keep nearly all of its multiple-choice accuracy after researchers removed the pathology slide. On 1,227 re-encoded SlideBench-VQA items, SlideChat scored 74.82% with the slide and 74.41% without it. A mismatched slide also produced no significant gap. Its underlying text model scored 43.52%, close to the benchmark's 40.7% most-frequent-letter baseline, suggesting that instruction-tuning taught answer-pattern shortcuts that did not require visual evidence.
The researchers ran four controlled probes using public checkpoints and data while varying one input at a time. They also shifted answer positions, substituted 'none of the above' and paraphrased questions. Paraphrasing did not materially change average accuracy, but 8–23% of individual answers flipped. That distinction matters: a stable headline score can conceal instability on the same case.[1]
Language added some signal—but not a clinical validation
Language was not useless. Across 1,633 TCGA patients—394 progression-free and 345 overall-survival events—adding a frozen embedding of stage-redacted report text to a slide embedding increased pooled C-index by 0.019 and 0.016 respectively. In lung and renal cancer subtyping, however, zero-shot prompting was statistically indistinguishable from a supervised linear probe when features and pooling were held fixed.
These experiments test benchmark construction and representation quality, not care. The data are retrospective and largely TCGA-derived; the grounding test uses multiple-choice questions, only two slide-level assistants were probed, and free-text behaviour may differ. Several contributors disclosed employment and possible stock holdings in Merck; the accessible manuscript does not state a separate funding source. Prospective multisite studies with pathologist readers, calibrated abstention and tissue-linked citations would be needed before the systems could support clinical decisions.[1]
The distinction that matters
A simple test of whether the model used the slide
The researchers kept the question and answer choices fixed while changing the visual evidence.
Correct slide
Give SlideChat the intended pathology image
74.82% correct
No slide
Remove the image but keep the question
74.41% correct
Interpretation
Compare the two conditions
No measurable grounding gap
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Evidence check
The manuscript is an unreviewed preprint using retrospective public data and released checkpoints.
The slide-grounding probe covered 1,227 multiple-choice items and two assistants; free-text reporting and other systems may behave differently.
Cross-model feature comparisons are confounded by architecture, data and aggregation differences, and no prospective clinician study was performed.
Reporting basis: Source analysis. Original source date: 19 October 2026.
Continue the story
Read next
Also about vision-language models
How much does MedGemma improve medical AI?
A peer-reviewed study reports gains over similarly sized base models across medical image questions, chest X-ray classification and simulated agent tasks. The evidence comes from benchmarks and small specialist reviews, not prospective clinical deployment or patient outcomes.
Read the 8 min reportAlso about clinical ai
An Alzheimer’s MRI model scored 94.5%. On new data, it fell to 61.6%
For a patient, the important test is whether a model works beyond familiar images. This classifier’s accuracy dropped sharply on an external dataset, and it remains a research system rather than a clinical diagnostic tool.
Read the 6 min reportExplore the full source and evidence record
What this means for people
- Pathologists and clinical teams should ask whether an answer changes when the image changes, not accept a benchmark score as proof that the model inspected tissue.
- Patients should not read the result as evidence that pathology AI is ready for diagnosis; no patient-care workflow, prospective outcome or safety endpoint was tested.
Global context
Pathology AI benchmarks often combine public images, generated questions and models exposed to related training material. These controlled ablations show why clinical evaluation needs counterfactual image tests and answer stability alongside accuracy.
What the evidence does not yet show
- The manuscript is an unreviewed preprint using retrospective public data and released checkpoints.
- The slide-grounding probe covered 1,227 multiple-choice items and two assistants; free-text reporting and other systems may behave differently.
- Cross-model feature comparisons are confounded by architecture, data and aggregation differences, and no prospective clinician study was performed.
What to watch next
- Independent replication on external, leakage-controlled slides and free-text tasks
- Prospective reader studies measuring whether the tool improves pathologist accuracy or time
- Routine reporting of image-removal, wrong-image, paraphrase, calibration and abstention tests
Living evidence record
Impact record IAI-1O12ET6
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
20 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 20 October 2026
Research question, topics and regions
Whether language adds measurable value to pathology vision-language systems and whether slide-level answers are grounded in the supplied image
Topics: Computational pathology · Vision-language models · Clinical AI · Benchmarking · Model grounding
Regions: United States · Global health research
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.