A melanoma model reached 95.7% sensitivity—but its test split was not patient-independent
The hybrid model missed 29 of 679 melanoma images in a locked test set. Related images may still cross partitions, and no independent cohort was tested.
The short version
- What happened
- The study used 25,331 ISIC 2019 images in a fixed 70:15:15 image-level split; the locked test set contained 3,801 images, including 679 melanoma images.
- Who it affects
- A person with a suspicious lesion needs a pathway that performs reliably across clinics, devices and skin tones. This study measured images from one public collection and did not test referral decisions or patient outcomes.
- What remains unproven
- 25,331 images from ISIC 2019
Keep in mind
25,331 images from ISIC 2019
3,801-image test set
Image-level split
Patient and lesion independence unverified
No external cohort
No calibration or subgroup analysis
No clinical outcomes
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
A strong image result with a clinically important miss count
A hybrid image model correctly identified 650 of 679 melanoma images in its held-out test set, according to a peer-reviewed Frontiers in Artificial Intelligence paper published on 29 October. That corresponds to 95.73% sensitivity, while 29 melanoma images were missed. The model also produced 26 false positives among 3,122 non-melanoma images, with reported specificity of 99.17%, balanced accuracy of 97.45% and ROC-AUC of 0.98.
Those figures describe classification of dermoscopic images, not diagnosis of people. The negative class pooled seven other ISIC categories, including basal-cell carcinoma, squamous-cell carcinoma and actinic keratosis. It therefore means non-melanoma, not benign. A false negative could delay further assessment, while a false positive could prompt avoidable referral or biopsy; the paper did not test either consequence in clinical care.[1]
The test set was locked, but independence is unresolved
Researchers used all 25,331 images in the ISIC 2019 training collection and made a fixed stratified image-level split: 17,731 for training, 3,799 for validation and 3,801 for final testing. Training-only augmentation and validation-only model selection reduce obvious test leakage. The system combined fixed U-Net lesion localization, EfficientNet-B0 for local boundary features, a Swin Transformer for global context and a feature-optimization block before binary classification.
The key qualification is the unit of splitting. The retained experiment did not provide a verifiable patient or lesion identifier, and the authors could not complete a duplicate or near-duplicate audit. Images of the same lesion or patient may therefore have crossed partitions, making the test task easier than a genuinely independent clinical evaluation. HAM10000 was not used as external evidence because it overlaps with ISIC 2019.[1]
The distinction that matters
What the locked test set actually contained
The confusion matrix exposes both the headline performance and the errors hidden by accuracy.
Melanoma images
679 in the test set
650 found; 29 missed
Other images
3,122 in the test set
3,096 correct; 26 false alarms
Independence
Image-level split
Patient/lesion separation unverified
What would make the result clinically persuasive
The paper provides no independent external validation, source-diagnosis subgroup errors, probability calibration, precision–recall curve, patient-level confidence intervals or formal case-level comparison with another model. Grad-CAM++ visualizations were qualitative and were not assessed by dermatologists. The fixed U-Net preprocessing component also lacked retained training provenance and independent segmentation benchmarking. The authors reported no funding and no commercial or financial conflicts.
A stronger assessment would test a frozen model on a non-overlapping cohort from different clinics, cameras and skin-tone groups, using patient-grouped partitions and clinician-confirmed outcomes. Calibration and decision-curve analysis should show whether scores support a safe referral threshold. Until then, the model is a promising benchmark pipeline, not evidence that it can screen patients or reduce missed melanoma in practice.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring medical imaging: compare the finding and its evidence limits.
Can endoscopy AI travel beyond still images?
A new classifier reached about 94% accuracy in cross-validation and 96% on an aligned external image subset, then narrowly outscored five gastroenterologists on 200 still images. Both datasets came from Norway, and the work did not test live video, workflow or patient outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- A person with a suspicious lesion needs a pathway that performs reliably across clinics, devices and skin tones. This study measured images from one public collection and did not test referral decisions or patient outcomes.
Global context
The authors are based in India, while the public ISIC collection aggregates images from multiple sources. Deployment in any country would require local, patient-level external validation across devices, skin tones and clinical pathways.
What the evidence does not yet show
- 25,331 images from ISIC 2019
- 3,801-image test set
- Image-level split
- Patient and lesion independence unverified
- No external cohort
- No calibration or subgroup analysis
- No clinical outcomes
What to watch next
- Patient-grouped validation
- Independent hospital cohorts
- Skin-tone subgroup results
- Calibration and referral thresholds
- Dermatologist comparison
- Prospective patient outcomes
Living evidence record
Impact record IAI-11647EJ
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
29 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Artificial Intelligence published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 29 October 2026
Research question, topics and regions
Hybrid CNN–Transformer classification of melanoma in ISIC 2019 dermoscopy images
Topics: Melanoma · Dermoscopy · Medical imaging · Computer vision · Validation
Regions: India · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.