Can conflicting medical images make AI hallucinate? Three models invented photo findings more often when an X-ray suggested another story
An unreviewed 40-condition benchmark paired real adult chest radiographs with controlled synthetic clinical photographs. False-positive photo findings rose to 45%–57.5% when the images did not support one diagnosis, but the small synthetic test and model-based judge do not measure clinical accuracy or patient outcomes.
The short version
- What happened
- Trick-or-Treat uses 40 real adult chest radiographs and 80 AI-generated clinical photographs. For each condition, one photo supports the radiograph's diagnostic story and one depicts a plausible but unrelated alternative finding.
- Who it affects
- Patients can be harmed if a fluent multimodal explanation invents a finding that makes unrelated evidence appear coherent.
- What remains unproven
- Unreviewed preprint with 40 conditions, 40 real chest radiographs and 80 synthetic clinical photographs; it is not a clinical cohort.
Keep in mind
Unreviewed preprint with 40 conditions, 40 real chest radiographs and 80 synthetic clinical photographs; it is not a clinical cohort.
Only three model versions were tested, through provider APIs with default sampling settings.
A Claude Sonnet 5.5 rubric judge scored extracted photo descriptions; outputs were not all independently adjudicated by clinicians.
The study did not measure final diagnostic accuracy, radiograph-description quality, clinician decisions or patient outcomes.
Synthetic-photo artifacts or generation style may influence models even after physician review of visible findings.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The answer: conflicting X-ray context pushed all three models toward a coherent but unsupported story
Three frontier vision-language models became more likely to invent clinically significant findings in a photograph when a paired chest X-ray suggested a diagnosis that the photograph did not support. In the unreviewed Trick-or-Treat benchmark, false-positive photographic findings on mismatched cases ranged from 25% to 37.5% when models saw the photograph alone. When researchers added the radiograph, the range rose to 45% to 57.5%. Qualitative examples show the models reinterpreting benign or unrelated visual details as abnormalities that made the two images tell one neat diagnostic story.
That is a specific failure mode rather than a clinical-performance estimate. The benchmark contains 40 real adult chest radiographs and 80 synthetic clinical photographs, not consecutive patients. It tests whether model descriptions stay grounded when modalities disagree. It does not establish the accuracy of a diagnosis, measure how clinicians would use the output or show patient harm. Its value lies in exposing a mechanism conventional single-image benchmarks can miss: more context can make an explanation more persuasive while pulling it further from what one image actually shows.[1]
How the counterfactual pairs were built
The researchers selected 40 conditions spanning acute and chronic presentations. Each case used one real chest radiograph from the ReXGradient collection and two generated external photographs of the same synthetic person. In the matched pair, the photograph showed a sign associated with the disease suggested by the radiograph—for example, crusted vesicles paired with an X-ray compatible with varicella pneumonia. In the mismatched pair, the alternative photograph showed a visually related but diagnostically different finding, such as ordinary acne.
Generating both photographs from a shared base, or editing one into the other, allowed the team to hold the person's apparent identity, pose and setting largely constant while changing the clinical sign. The photo subject's age and sex were aligned with radiograph metadata. Because ethnicity metadata were mostly unavailable, the team assigned synthetic-photo ethnicity using a reproducible draw intended to approximate the 2020 US population. Final images were approved by one to three board-certified physicians in pulmonary disease, orthopaedic surgery or internal medicine.[1]
The distinction that matters
Why matched and mismatched image pairs reveal different behaviour
The same real chest radiograph is paired with two controlled synthetic photographs, so the benchmark can test whether context helps or distorts photo description.
Photo alone
Describe visible findings
Baseline grounding error
Matched pair
Add supporting X-ray
Some errors fall
Mismatched pair
Add conflicting X-ray
False positives rise
Safety test
Check each modality separately
Preserve disagreement and uncertainty
What the three models were asked to do
GPT-6 Astra, Gemini 4 Argon and Claude Fable 5.1 were queried through their native provider APIs with default sampling settings. In the multimodal condition, each model described the radiograph and photograph and explained what they might suggest. In the baseline, it saw only the photograph. An extraction stage isolated statements about the photograph so that a diagnostic hypothesis elsewhere in the answer did not automatically count as a photographic observation.
The primary outcomes were false-positive and false-negative rates for clinically significant abnormalities in the photograph. A Claude Sonnet 5.5 rubric judge compared the extracted descriptions with the benchmark captions. Normal variants and clinically insignificant details were excluded. The researchers did not grade the quality of the final diagnosis because mismatched images could represent coexisting conditions rather than one correct unifying answer. This design tests visual grounding under contextual pressure, not complete medical reasoning.[1]
The denominators show both the pattern and its uncertainty
Every percentage is based on 40 photographs for a given photo type, model and setting. On mismatched photographs, GPT-6 Astra's false-positive rate rose from 37.5% with the photo alone to 57.5% with the radiograph; the exact 95% confidence interval for the paired condition was 40.9% to 73.0%. Gemini 4 Argon rose from 35% to 45%, and Claude Fable 5.1 from 25% to 45%. The increase was significant for GPT-6 Astra and Claude Fable individually and across the pooled model outputs, but not for Gemini alone.
Matched context sometimes helped. False negatives on matched photographs fell from 10% to 2.5% for GPT-6 Astra, from 7.5% to 2.5% for Gemini and from 27.5% to 12.5% for Claude. The pooled reduction was statistically significant. The same mechanism—using the radiograph to interpret a photograph—can therefore improve performance when the modalities agree and distort it when they diverge. A safe system needs to represent disagreement rather than assuming every available input belongs to one diagnosis.[1]
Why this matters for clinicians and people receiving care
Clinical information is often inconsistent for legitimate reasons. A patient may have two unrelated conditions, an old X-ray finding, an incidental abnormality, a documentation error or images acquired at different times. A system that forces those inputs into one narrative can overstate certainty and encourage premature closure. If a clinician reads a fluent answer that claims the skin photograph confirms the chest finding, the invented detail may be harder to notice than a simple classification error because it appears to explain the whole case.
The benchmark does not justify deploying these models as image interpreters or withdrawing them from every support role. It identifies a test that procurement and evaluation teams should add before trusting multimodal outputs. Systems should be challenged with discordant and unrelated evidence, asked to describe each modality independently, and rewarded for stating that findings may not share one cause. Interfaces should preserve the underlying images and uncertainty rather than present only a merged narrative. Human review remains essential wherever a generated description could influence diagnosis or treatment.[1]
The synthetic benchmark is useful because it is controlled—and limited for the same reason
Naturally collected multimodal datasets rarely contain pairs that differ in one carefully chosen finding while preserving identity and scene. Synthetic generation made that counterfactual comparison possible. It also introduces potential artifacts. Clinicians approved the visible content, but models may respond to subtle generation traces, texture or style that people do not notice. The assigned demographic distribution is not a substitute for measuring performance across real skin tones, cameras, lighting conditions and clinical settings.
The output scoring is another limitation. A model judged model outputs after an extraction step; clinician adjudication of every response would provide a stronger reference. The benchmark covers three model versions and only 40 conditions. Provider updates, prompting, abstention rules or different interfaces could change results. Because the paper is an arXiv preprint, its methods and interpretations have not completed journal peer review. The reported pattern deserves replication rather than conversion into a universal error rate for medical AI.[1]
What would change the assessment
A stronger next study would preregister a larger set of counterfactual pairs, include several real imaging modalities and clinical text, and use blinded clinician adjudication for both descriptions and final diagnostic reasoning. Models should be tested on their ability to abstain, list separate hypotheses and flag that inputs may be unrelated. Repeating the evaluation after provider updates would show whether the failure persists. Real-world silent-mode studies could then measure whether discordant cases cause clinician error, delay or inappropriate confidence.
The risk assessment would improve if models consistently preserved modality-specific observations and calibrated uncertainty without losing the gains seen when evidence genuinely agrees. It would worsen if larger independent tests showed the same contextual hallucination affecting treatment decisions. For now, Trick-or-Treat offers a focused warning: combining more medical inputs does not automatically create better reasoning. Without explicit tests for contradiction, a multimodal system may turn disagreement into a confident fiction.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring patient safety: compare the finding and its evidence limits.
AI health assistants often endorsed self-diagnoses that needed caution—what did the test measure?
In 100 matched pairs, 27 models were asked when to challenge or accept a plausible self-diagnosis. Even the best current model cautioned appropriately on only 77% of time-critical condition cases, but the benchmark used written scenarios and one physician reviewer.
Read the next reportExplore the full source and evidence record
What this means for people
- Patients can be harmed if a fluent multimodal explanation invents a finding that makes unrelated evidence appear coherent.
- Clinicians need modality-specific observations, visible source images and explicit uncertainty rather than only a merged narrative.
- Hospital evaluation teams should include deliberately discordant cases before allowing multimodal systems to influence care.
Global context
Multimodal tools may be attractive where specialists are scarce, but inconsistent records, variable image quality and limited local validation can make contextual errors harder to detect. A controlled US-led benchmark cannot establish performance in other health systems. Its transferable lesson is to test disagreement explicitly and keep every source modality inspectable.
What the evidence does not yet show
- Unreviewed preprint with 40 conditions, 40 real chest radiographs and 80 synthetic clinical photographs; it is not a clinical cohort.
- Only three model versions were tested, through provider APIs with default sampling settings.
- A Claude Sonnet 5.5 rubric judge scored extracted photo descriptions; outputs were not all independently adjudicated by clinicians.
- The study did not measure final diagnostic accuracy, radiograph-description quality, clinician decisions or patient outcomes.
- Synthetic-photo artifacts or generation style may influence models even after physician review of visible findings.
What to watch next
- Peer review and independent replication with larger preregistered case sets.
- Blinded clinician adjudication of descriptions and final diagnoses.
- Model behaviour when instructed to separate hypotheses or abstain.
- Real multimodal cohorts across skin tones, devices and clinical settings.
- Prospective evidence on clinician reliance, diagnostic error and patient outcomes.
Living evidence record
Impact record IAI-14ULPA3
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
Whether a chest radiograph that suggests one diagnosis can cause multimodal models to hallucinate matching abnormalities in a conflicting clinical photograph
Topics: Medical AI · Multimodal models · Hallucination · Synthetic data · Evaluation · Patient safety
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.