Can a skin-lesion AI survive the move from dermoscopes to smartphone photos?
A four-class model’s macro-F1 fell from 74.27% internally to 54.92% on 1,376 unseen smartphone images. Its safety gate retained only 11.5% of external cases, exposing the cost of reliable abstention.
The short version
3 key points
- 1.The researchers developed the model on 11,259 eligible dermoscopy images and kept 1,376 smartphone-acquired PAD-UFES-20 images completely outside training and threshold setting.
- 2.Across five seeds, macro-F1 fell from 74.27% internally to 54.92% externally; melanoma F1 dropped from 58.21% to 27.38%.
- 3.A source-fitted reliability gate retained 32.37% of internal cases but only 11.50% externally, while accepted-case accuracy remained above 90%.
External images exposed a large reliability gap
A skin-lesion classifier that looked reasonably strong on dermoscopy images lost substantial performance when moved to unseen smartphone photographs. In a peer-reviewed four-class study, mean macro-F1 fell from 74.27% on the source-domain test sets to 54.92% on 1,376 PAD-UFES-20 images. The result matters because a model used outside specialist imaging must survive changes in cameras, lighting, backgrounds and patient mix—not just repeat a laboratory benchmark.
Researchers developed the system on 11,259 eligible images from HAM10000 and Derm7pt, covering basal cell carcinoma, melanoma, melanocytic naevus and a harmonised benign-keratosis category. They separated data by lesion or clinical case and reserved distinct partitions for training, model selection, calibration, reliability thresholds and internal testing. The smartphone dataset and a separate Stanford DDI audit did not influence training or threshold choice.[1]
The average concealed a sharp melanoma failure
Performance did not fall evenly. Melanoma F1 decreased from 58.21% internally to 27.38% externally, while naevus F1 fell from 89.65% to 60.80%. Basal-cell-carcinoma F1 was broadly preserved, but its sensitivity dropped from 78.88% to 64.66%. Macro-AUROC declined much less—from 0.926 to 0.880—showing why a ranking metric alone can look reassuring while thresholded clinical classifications deteriorate.
The external shift was not simply ‘dermoscope versus phone’. Nearly 70% of source images were naevi, whereas 61.4% of the smartphone cohort were basal cell carcinomas. Devices, clinical setting, dataset construction and diagnosis prevalence all changed together. The authors correctly describe a compound transfer test, not an isolated measurement of camera effects.
Their selective-decision gate adds another useful warning. It retained 32.37% of internal cases, with 97.86% accepted-case accuracy, but only 11.50% externally, with 92.97% accuracy. Refusing uncertain cases protected performance, yet almost nine in ten external images were not retained for an unrestricted prediction. A system that is safer because it abstains frequently may still create a large specialist-review workload.[1]
The distinction that matters
What changed when the model left its source domain
The external test changed acquisition and diagnosis mix while keeping the trained model and thresholds fixed.
Source test
Dermoscopy-derived internal evaluation
Macro-F1 74.27%; 32.37% retained
External test
1,376 unseen smartphone images
Macro-F1 54.92%; 11.50% retained
Clinical gap
Defer uncertain cases to specialists
Workload and patient outcomes were not tested
A heatmap is not a clinical explanation
For dermatology teams, the practical lesson is to validate the whole decision pathway: who captures the image, which lesions enter the system, when it abstains and how quickly a clinician can review the remainder. The paper’s Grad-CAM and Integrated Gradients audit describes explanation stability, but it does not show that highlighted pixels correspond to clinically causal features or improve a clinician’s decision.
The evidence check remains retrospective and dataset-based. No clinician used the tool prospectively, no patient outcome changed, and performance reflects four harmonised categories rather than the full range of skin disease. The model also relied on clinical metadata, whose availability and quality may differ in practice. The paper reports performance across five random seeds, but that repeated training does not substitute for testing a locked system in a real clinical pathway. Confidence would rise with prospective multi-device studies, prespecified skin-tone subgroup recruitment, comparison against clinicians, measured referral workload and patient-level outcomes. Until then, the study supports specialist-supervised triage and explicit abstention—not autonomous smartphone diagnosis.[1]
Evidence check
Retrospective public datasets, four harmonised diagnoses, compound acquisition and prevalence shift, incomplete subgroup evidence and no prospective clinician or patient outcomes.
Reporting basis: Source analysis. Original source date: 16 October 2026.
Explore the full source and evidence record
What this means for people
- People photographing a suspicious lesion need a tool that recognises when its answer is unreliable; frequent abstention also means services must have capacity for timely specialist review.
Global context
Smartphone capture could widen access beyond specialist dermoscopy, particularly where dermatology capacity is limited, but heterogeneous devices and populations make external validation more—not less—important.
What the evidence does not yet show
- Retrospective public datasets, four harmonised diagnoses, compound acquisition and prevalence shift, incomplete subgroup evidence and no prospective clinician or patient outcomes.
What to watch next
- Prospective multi-device trials, skin-tone subgroup performance, specialist workload after abstention, clinician comparisons and evidence that triage changes time to diagnosis or treatment.
Living evidence record
Impact record IAI-1GWVMS5
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
16 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Diagnostics published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 16 October 2026
Research question, topics and regions
Cross-device reliability of multi-class skin-lesion AI
Topics: Skin cancer · Smartphone imaging · Domain shift · Clinical safety · Uncertainty
Regions: Brazil · Iraq · Hungary · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Healthcare, culture & human impact · Background explainer
Published 2026-07-01 · 44 min 28 sec
Is this the AI future we want? | The Royal Society
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
Can an AI agent safely summarise years of myeloma care?
A retrospective study found that a skill-guided agent agreed with expert answers on 80.1% of 469 questions from 100 patient records. It helped most on very long records, but most system errors were judged clinically significant and no live-care benefit was tested.
8 min · 1 source
Health & Life Sciences
How should AI diet advice be tested before people rely on it?
A peer-reviewed Perspective proposes four evidence gates for AI-generated menus, from checking nutrient arithmetic to prospective outcomes. It is a testable framework—not a validated standard—and its thresholds still need calibration and independent use.
7 min · 1 source
Health & Life Sciences
Did web search make GPT-5 a better diagnostician?
Not reliably. In a peer-reviewed benchmark of 161 image-based clinical cases, the strongest reasoning configuration reached 46.0% diagnostic accuracy. Adding web search cut the simplest configuration from 38.5% to 26.7%, while agent and deep-research modes also trailed that baseline.
11 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.