Can commercial multimodal AI read breast-cancer pathology slides? A 427-case study found systematic unsupported calls
Three commercial models received the same overview plus 16 tissue tiles per cancer. None reached agreement suitable for autonomous extraction; one model called lymphovascular invasion in 26 packets where blinded reviewers found none.
The short version
- What happened
- The retrospective single-centre study compared Claude Sonnet 4.6, Gemini 2.5 Pro and GPT-5.5 with a pathology-report reference across 427 invasive breast cancers, using the same overview, 16 H&E tiles and prompt per case.
- Who it affects
- Patients were not exposed to the tested outputs, but an unsupported invasion or grade could be consequential if it entered care.
- What remains unproven
- This was a retrospective single-centre concordance study, not a prospective diagnostic trial or patient-outcome study.
Keep in mind
This was a retrospective single-centre concordance study, not a prospective diagnostic trial or patient-outcome study.
Models received 16 selected tiles rather than whole-slide navigation, full assays or complete clinical context.
Only three named commercial model versions and one standardized prompting protocol were evaluated.
The primary report reference and human re-reads can disagree; the study does not establish a perfect gold standard for every feature.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The answer: none of the three models was reliable enough to extract pathology features alone
A peer-reviewed South Korean study of 427 invasive breast cancers found that three commercial multimodal language models made systematic, model-specific errors when asked to read clinically relevant features from pathology images. Each model received the same case overview, 16 hematoxylin-and-eosin tissue tiles and the same structured prompt. The comparison covered Claude Sonnet 4.6, Gemini 2.5 Pro and GPT-5.5. None achieved agreement that the researchers considered sufficient for autonomous extraction of any tested feature. The study addresses a tempting shortcut—sending slide images to a general-purpose model where specialist access is limited—but its results argue for controlled evaluation and pathologist oversight rather than substitution.
The clearest safety signal came from a blinded review of the actual image packets. Among 110 assessable packets, reviewing pathologists saw no lymphovascular invasion. Claude and GPT-5.5 also reported none, while Gemini called lymphovascular invasion in 26 packets, with none supported by the supplied images. Perineural invasion was visible in one packet. Five packets contained no invasive carcinoma at all, yet across the three models a Nottingham grade was returned in 14 of the 15 model-case combinations. These are not merely small deviations around a continuous score; they include confident-looking outputs for evidence absent from the input.
That does not mean every apparently wrong answer was invented in the ordinary sense. A model may reproduce patterns associated with the wording, case mix or prompt even when the relevant feature is not visible in the selected tiles. The study could not separate learned priors from prompt-model interaction. For a clinician, however, the operational distinction is secondary: an unsupported finding entering a report can change staging, treatment discussion or further testing. A usable system must indicate when evidence is absent, not fill a structured field because the format appears to demand an answer.[1]
A paired design exposed both model error and the limits of the input packet
The primary reference was the board-certified pathologist's clinical report. The researchers compared raw agreement and chance-corrected agreement for features such as histological type and grade, then characterized the direction of each model's mistakes. To avoid treating the original report as infallible, 111 cases were selected for a second layer of review: two pathologists, blinded to the report, re-read the submitted slide, and the exact tile packets given to the models were reviewed separately. That created slide-level and packet-level references and a within-study benchmark for how much human readers disagreed.
Human agreement was not perfect. Quadratic-weighted kappa for grade between two readers on submitted slides was 0.51, a moderate value. Agreement between the models and the report for grade ranged from 0.27 to 0.42, while the second pathologist's agreement with the report was 0.36. The overlap is a warning against presenting pathology as a trivial ground-truth extraction task. Yet it does not rescue the models: the paper's conclusion rests on multiple features, systematic directions of error and cases in which the relevant tissue or finding was demonstrably absent from the image packet.
The input protocol also matters. Sixteen static tiles cannot reproduce a pathologist's navigation of a whole-slide image, selection of representative tumour and integration of immunohistochemistry, clinical history and gross findings. The negative-control analysis illustrates that boundary. Against immunohistochemistry-defined receptor classes, sensitivity for HER2-positive disease was at most 3.6% across the three models—what the authors expected when H&E images alone do not provide the required receptor measurement. A model that appears fluent about a receptor class is not thereby reading the missing assay.[1]
The distinction that matters
Why the 16-tile test is informative but not a clinical deployment
The study held the packet and prompt constant to compare models, then asked pathologists what the packet actually contained.
Cases
Select 427 invasive cancers
Single-centre retrospective cohort
Input
Provide overview plus 16 H&E tiles
Standardized but incomplete view
Reference
Compare reports and blinded re-reads
Human disagreement and packet limits visible
Use
Require tumour-aware, supervised testing
Autonomous extraction unsupported
A high overall percentage concealed clinically important misses
Claude had the highest raw concordance for histological type at 87.4%. That headline number looked reassuring until the researchers examined class-specific sensitivity: the model classified every lobular and micropapillary carcinoma as invasive breast carcinoma of no special type. The ranking then reversed. Rare but important classes contribute little to overall accuracy, so a model can look strong by repeatedly choosing the dominant category. Clinical evaluation needs per-class sensitivity, confusion matrices and predefined minimum performance for findings that alter management, not only a weighted average across the case mix.
The researchers ran limited secondary analyses using prompt variants, different reasoning effort and an alternative higher-tier model from the same vendor. These did not remove the model-specific patterns. Because the analyses were limited and performed only for individual models, they do not establish that prompting never helps or that later model versions will behave identically. They do show why a hospital cannot inherit a safety claim from a general benchmark. A change in model, interface, prompt, compression or image sampling can change the error distribution and should trigger version-specific validation.
The practical use case is also narrower than remote diagnosis. Commercial services may lower hardware barriers, but uploading pathology images introduces data protection, cross-border processing, service continuity and vendor-change questions. The paper used anonymized retrospective material under ethics approval; it did not evaluate those operational controls. A deployment would need locked model versions, audit logs, pathologist confirmation, clear handling of abstention and missing evidence, and monitoring for class-specific drift. It should also define whether the model drafts, checks or prioritizes rather than leaving an ambiguous promise to 'assist'.[1]
What the study can and cannot change in clinical practice
For patients, the immediate message is not that AI has entered routine breast pathology at the study hospital. This was a retrospective concordance experiment and no output guided care. For pathologists, it provides concrete failure cases for procurement and validation: unsupported vascular invasion, grading when invasive tumour is absent and collapse of uncommon types into the majority class. For health systems with workforce shortages, it is a reason to invest in digital-slide infrastructure and specialist networks while testing AI as a supervised component, not to replace scarce expertise with a consumer-facing interface.
The study is single-centre and reflects one hospital's case mix, slide preparation and reporting conventions. It tested three named model versions through a standardized protocol at one point in time. Selected tiles may disadvantage models designed for whole-slide analysis, while the general-purpose models were not trained or cleared as dedicated pathology devices. The work reports concordance, not diagnostic accuracy against a prospective consensus panel, downstream treatment decisions, turnaround time or patient outcomes. Its funding came from South Korea's National Research Foundation; the authors declared no competing interests.
Confidence would change with external, multi-centre studies using whole-slide images, prespecified tumour-aware sampling and independent pathologist consensus. Trials should compare a supervised workflow with current practice, record when the system abstains and test whether errors are caught before a report is signed. Subgroup analyses need to cover uncommon histological types and variation in scanners, staining and tissue preparation. Until that evidence exists, the finding is direct and consequential: fluent multimodal output from a commercial model is not a pathology report, and absence of visible evidence must be treated as a stop condition rather than an invitation to complete the form.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring breast cancer: compare the finding and its evidence limits.
Can AI make breast-cancer notes easier to understand? Comprehension improved—but clinicians found 82 errors
In a peer-reviewed mixed-methods study, 200 healthy women over 40 answered more action questions and reported greater confidence when clinical notes included GPT-4-generated simplifications, definitions and summaries. Yet three clinician reviewers found 82 errors across the generated material, including 28 judged potentially harmful or misleading, while only seven participants with breast-cancer histories tested the interface with their own notes.
Read the next reportExplore the full source and evidence record
What this means for people
- Patients were not exposed to the tested outputs, but an unsupported invasion or grade could be consequential if it entered care.
- Pathologists need systems that show evidence, abstain when tissue is missing and preserve final professional review.
- Health services facing specialist shortages should test supervised workflows and digital infrastructure rather than treating a commercial interface as autonomous expertise.
Global context
The work was conducted in South Korea and speaks to global interest in remote pathology where specialists are scarce. Case mix, slide preparation, regulation, data residency and access to confirmatory assays differ across health systems, so local prospective validation is essential.
What the evidence does not yet show
- This was a retrospective single-centre concordance study, not a prospective diagnostic trial or patient-outcome study.
- Models received 16 selected tiles rather than whole-slide navigation, full assays or complete clinical context.
- Only three named commercial model versions and one standardized prompting protocol were evaluated.
- The primary report reference and human re-reads can disagree; the study does not establish a perfect gold standard for every feature.
What to watch next
- Whole-slide, multi-centre external validation.
- Prospective pathologist-supervised workflow trials.
- Locked-version testing after model or interface updates.
- Performance on uncommon types and missing-evidence abstention.
Living evidence record
Impact record IAI-156YCIM
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Breast Cancer Research published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
Whether general-purpose commercial multimodal language models can extract clinically relevant breast-cancer pathology features from standardized H&E image packets
Topics: Pathology · Multimodal models · Breast cancer · Clinical safety
Regions: South Korea · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.