Can an Alzheimer’s MRI model travel between datasets?
Only partly. A model scored 94.5% on its internal slice-level test but 61.6% on 292 external images, with mild-dementia recall falling to 11%. The public training set lacked patient identifiers, so even the internal split was not truly patient-independent.
Editorial responsibility: The Impact of AI Editorial Desk · Report a factual concern
At a glance
- 1Internal accuracy was 94.5% on a balanced 292-slice test set, but fell to 61.6% on a separate balanced 292-image collection without fine-tuning.
- 2External recall for mild dementia was 10.96%, compared with 98.63% internally; very-mild recall fell from 97.26% to 56.16%.
- 3The novel KAN head was statistically indistinguishable from a conventional MLP head in the matched comparison, while pretraining and data use drove most of the performance.
Research topic
How a four-class Alzheimer's MRI model performs internally and on an independent dataset, and whether its novel classifier head adds measurable value
The answer: the headline accuracy did not travel
A peer-reviewed engineering study reports 94.5% accuracy for a four-stage Alzheimer’s MRI classifier on its internal test set. When the same model was applied without retraining to 292 images from an independent collection, accuracy fell to 61.6% and macro-F1 to 0.565. The 33-point gap is the result clinicians and procurement teams should notice: a high score within one image distribution did not translate into dependable staging elsewhere.
The researchers deserve credit for reporting the external test and for stating that the model is not ready for clinical use. The work shows partial transfer, especially for visually distinctive moderate dementia, but performance on the clinically difficult intermediate categories deteriorated sharply. It is a methods study using public image collections, not a prospective diagnostic trial, and it provides no evidence that using the model improves diagnosis, treatment or patient outcomes.[1]
The Impact Brief · Free
Follow the evidence in health & life sciences.
Get a five-minute weekday briefing on what changed, why it matters and where the evidence comes from. Choose the topics you care about.
How the model and data were assembled
The system combines an ImageNet-pretrained ConvNeXt-Tiny image backbone with a Kolmogorov–Arnold Network, or KAN, classification head. The head maps a 768-dimensional feature vector to four labels: non-demented, very mild, mild and moderate dementia. Training used mild image transformations, MixUp, CutMix, label smoothing, AdamW optimisation and test-time augmentation.
The publicly distributed OASIS slice collection contained 86,437 axial images: 67,222 labelled non-demented, 13,725 very mild, 5,002 mild and 488 moderate. The researchers capped each class at 1,500 slices, held out balanced validation and test sets before oversampling and replicated only the remaining training images. That avoids the obvious error of duplicating a test image into training and gives each class equal weight in the 292-slice internal test set.[1]
Patient-level leakage remains unresolved
The more important dependence problem could not be eliminated. One three-dimensional MRI scan produces many neighbouring two-dimensional slices, but the public slice collection does not retain per-slice patient identifiers. The study could check that files were not bit-for-bit duplicates across partitions, yet it could not stop slices from the same person appearing in both training and test sets. Similar anatomy across adjacent slices can make internal performance look better than generalisation to a new patient.
Calling the internal protocol leakage-controlled therefore needs a precise qualification: it controls duplication caused by oversampling, not shared-patient leakage. The authors acknowledge this and name a patient-level split as the most important next step. They also used one fixed split rather than repeated hold-outs or cross-validation, so the stability of the 94.5% estimate across alternative partitions is unknown.[1]
The external errors concentrate where staging matters
The independent dataset used the same four labels and contributed 73 images per class. Moderate dementia remained perfectly recalled at 73 of 73 images, and non-demented recall was 79.45%, close to the internal 82.19%. Mild-dementia recall, however, collapsed from 98.63% internally to 10.96% externally. Very-mild recall fell from 97.26% to 56.16%.
Many mild images shifted into the neighbouring very-mild or moderate categories. That pattern is consistent with acquisition differences, contrast, slice selection and label construction moving the decision boundary. It is not enough to say the external set is harder: a model proposed for staging needs trustworthy discrimination at the earlier and intermediate stages, where care decisions are sensitive and visible structural differences are subtle.[1]
The novel head was not the source of a clear gain
In a matched shortened training protocol, the KAN and conventional multilayer-perceptron heads were statistically indistinguishable on accuracy (McNemar p=0.52). The MLP led on several threshold-dependent metrics, while KAN led on ROC-AUC. The KAN head had about nine times as many head parameters, though it increased the full model by only 2.84% and added no material runtime in the reported GPU test.
The paper's ablation instead identifies pretrained features and expanded data use as the principal contributors. Its training-partition resampling also outperformed the tested focal-loss, class-balanced-loss and feature-space SMOTE approaches. Those results are useful engineering evidence, but they weaken any simple claim that the new KAN component produced the headline performance.[1]
What clinicians and health systems should take from it
Clinicians should not use this model to stage a patient. The labels came from public image repositories, the unit of evaluation was an image slice rather than a patient pathway, and the study did not compare predictions with a blinded multi-reader clinical reference. Saliency maps concentrated on plausible brain regions, but no quantitative test established that those maps were faithful explanations or that clinicians could act on them safely.
For health systems, the external decline is a practical demonstration of why local validation must precede deployment. Testing should preserve patient separation, use scans from every intended scanner and population, report calibration and clinically defined errors, and evaluate the entire workflow—including radiologist review, uncertainty handling and the consequences of a wrong stage—not only per-image accuracy.[1]
What would change the assessment
The next evidence should use subject-linked volumetric MRI with patient-level partitions, repeated or cross-validated estimates and multi-site cohorts such as ADNI under an appropriate data-use agreement. A locked model should be evaluated prospectively against clinical diagnoses and independent readers, with age, sex, scanner, comorbidity and disease-spectrum breakdowns. Domain adaptation must be trained separately from the external test it is intended to pass.
Until then, the most important contribution is not a deployable Alzheimer’s classifier. It is a transparent example of how internal success can shrink under distribution shift and how a novel architectural component may add less than disciplined data handling. The external test makes the paper more useful—and the clinical claim much narrower—than the 94.5% headline alone would suggest.[1]
What this means for people
- Patients should not receive a dementia stage from this model; it has not been tested as a clinical diagnostic system.
- Clinicians need patient-level, multi-site validation and error analysis at early and intermediate stages before considering decision support.
- Health-system buyers should treat external performance and workflow safety as primary evidence, not an internal accuracy headline.
Global context
The study used publicly distributed MRI slices and an independent Kaggle collection rather than a geographically defined clinical cohort. Scanner protocols, populations, diagnostic practice and prevalence differ across sites, so external performance cannot be assumed elsewhere.
What the evidence does not yet show
- The OASIS slice collection lacked subject identifiers, preventing a patient-independent split.
- The internal result came from one fixed balanced split of 292 slices, not repeated patient-level validation.
- External validation used 292 images from a Kaggle collection rather than a clinically governed multi-site cohort.
- The unit was an image slice, not a patient or complete diagnostic episode.
- No prospective workflow, clinician comparison, calibration assessment or patient outcome was tested.
What to watch next
- Subject-level evaluation using volumetric scans and repeated partitions
- Locked multi-site validation on ADNI or equivalent clinically governed cohorts
- Per-patient calibration and error rates for very mild and mild disease
- Prospective reader studies measuring diagnostic decisions and patient consequences
Living evidence record
Impact record IAI-0U1SS4A
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
13 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Artificial Intelligence published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 13 October 2026
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Continue the story
Related reporting
Health & Life Sciences
Can brain MRI reveal more than BMI?
A peer-reviewed study trained a deep-learning model on 45,702 MRI scans from six cohorts. Brain-derived features tracked BMI and separated several disease groups better than BMI alone—but the disease analysis stayed inside UK Biobank and cannot establish cause, diagnosis or clinical benefit.
9 min · 2 sources
Health & Life Sciences
Can this MRI model draw brain-tumour boundaries reliably?
A peer-reviewed multimodal segmentation model was developed on 2,422 public MRI volumes and externally tested on 125 cases. Accuracy remained useful but fell outside the development data; no prospective clinical workflow, radiologist comparison or patient-outcome test was performed.
8 min · 2 sources
Health & Life Sciences
Can AI support breast-ultrasound decisions across countries?
A peer-reviewed South Korean-led study validated an interpretable retrieval-augmented system across 8,311 images from 11 cohorts in seven countries and tested assistance with four readers. The retrospective evidence is encouraging, but it is not a prospective screening trial or proof of better patient outcomes.
8 min · 3 sources
Reader commentary
Add evidence, experience or a question
No account is required. Reader notes are published after a brief civility, relevance and safety check; disagreement is welcome.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.