Can AI find children likely to miss a measles vaccine?
Models trained on 13,196 vaccine-eligible children in Uganda and Kenya identified many recorded defaulters, but the advanced ensembles barely beat simpler tuned models, thresholds were chosen on the test set, and the study did not test live records, outreach or vaccination uptake.
The short version
3 key points
- 1.The analysis included 6,793 vaccine-eligible infants in Uganda and 6,403 in Kenya; recorded defaulting prevalence was 22.1% and 26.3%, respectively.
- 2.The selected ensemble reached AUC 0.8847 and F1 0.7111 in Kenya, and AUC 0.8862 and F1 0.6756 in Uganda, but F1 gains over the best tuned base models were only 0.0054 and 0.0059.
- 3.The authors optimized classification thresholds on the test set and did not perform temporal, geographic or prospective external validation. The model has not shown that targeted reminders or outreach increase vaccination.
The answer: the models separate risk groups, but the extra AI complexity adds little
The Uganda–Kenya study shows that tabular machine-learning models can recover substantial signal about which children were recorded as missing a measles vaccination. In the held-out test sets, the selected Kenya ensemble identified 256 of 337 defaulters and the selected Uganda ensemble identified 226 of 301. Their receiver-operating-characteristic areas were about 0.885 and 0.886. Those are useful development results for prioritising further validation; they are not evidence that a health service has safely identified today's children or prevented an outbreak.
The headline architectural gain is much smaller. In Kenya, the advanced ensemble's F1 score exceeded the best tuned base model by 0.0054; in Uganda the gain was 0.0059 and not statistically significant. The confidence intervals for both F1 differences crossed zero. The paper itself concludes that additional computational complexity did not provide a meaningful predictive advantage in Uganda. A simpler, cheaper and easier-to-audit model may therefore be the better operational comparator.[1]
What data went into the models
The researchers used nationally collected Demographic and Health Survey data from 2016–2022. From 8,928 infants in the Uganda dataset, 6,793 met the study's measles-vaccination eligibility rules; Kenya contributed 6,403 eligible infants. The recorded outcome classified a child as immunised when a date or mark appeared on the card or vaccination was reported by the mother, while 'no' or 'do not know' became the defaulter class. That outcome therefore mixes documentation and caregiver recall with actual receipt.
Candidate predictors covered the child's age, earlier vaccines and vitamin A, the respondent's education and age at first birth, travel time to water, wealth, altitude and current or previous district. Feature filtering retained 11 variables after removing highly correlated or statistically weak candidates. The models were trained separately by country, which avoids pretending that one set of associations must transfer unchanged across Uganda and Kenya, but it also means that performance in one survey wave cannot be assumed in a later registry or district.[1]
How the modelling pipeline can make performance look stronger
The paper split training and test data, applied synthetic minority oversampling only after the split, and used five-fold stratified cross-validation with grid-search tuning. Those are valuable protections against some forms of leakage. However, it then selected the classification threshold that maximised F1 using the test set. Once a test set influences the decision boundary, it no longer provides a fully untouched estimate of future performance. A nested validation set or preregistered threshold followed by a fresh external test would give a cleaner estimate.
The ensemble comparisons also involve many models, hyperparameters and thresholds. Reported AUCs describe ranking ability, while the operational decision depends on calibration, threshold and the relative costs of false negatives and false positives. The paper reports precision, recall, F1 and confusion matrices, but not a prospective calibration study or decision-curve analysis showing net benefit at a health service's real capacity. A model that flags too many families could overwhelm outreach teams or repeatedly contact already vaccinated children.[1]
The errors are large enough to matter
In Kenya's 1,281-person test set, the selected soft-voting model produced 256 true positives, 81 false negatives, 817 true negatives and 127 false positives. In Uganda's 1,359-person test set, the selected stacked model produced 226 true positives, 75 false negatives, 916 true negatives and 142 false positives. Put plainly, roughly one child in four recorded as a defaulter was missed, while more than one hundred vaccinated children in each country were flagged.
Those trade-offs may still be acceptable for a low-cost reminder, but not automatically. A false positive can consume staff time, expose sensitive family data or erode trust if records are wrong; a false negative means a child who may need support receives none. The correct threshold depends on intervention cost, vaccine access, data quality and whether the flag triggers a text, a home visit or a clinical decision. The study does not compare those pathways or measure the consequences of its errors.[1]
What the predictors mean—and do not mean
Child age and recent vitamin A were influential in both countries. Education, water-source travel time and migration-related district variables mattered more in Kenya; earlier polio and pentavalent vaccination variables were prominent in Uganda. These may help describe where service barriers and incomplete schedules cluster. They are not causal estimates. SHAP values explain how a fitted model used its available variables; they do not prove that changing education, water access or vitamin A administration would cause measles-vaccine completion.
Several influential variables are also close to the immunisation process itself. Prior vaccine receipt can be a strong predictor because it records engagement with the same service pathway, while age affects whether there has been time to receive or record the dose. That may be operationally useful, but it is less surprising than discovering a new mechanism. Before using district or socioeconomic variables for targeting, programmes need fairness checks, community involvement and a rule against denying care to people assigned a low score.[1]
What would justify deployment
The next test should lock the model, preprocessing and threshold, then evaluate them on newer registry data from districts and facilities not represented in development. It should report calibration and error rates by region, urbanity, wealth, education and documentation quality. Because survey variables may not exist in routine systems, researchers must also show that the required fields are available, timely, lawful and accurate in DHIS2 or an immunisation registry without importing data the service cannot maintain.
The decisive evidence would be a prospective trial comparing usual outreach with model-assisted prioritisation. Outcomes should include verified vaccination, missed eligible children, duplicate contacts, staff workload, cost per additional vaccination, complaints and equity. The study received no external funding and declared no conflicts, but its prototype has not undergone such a test. Until then, the models are promising retrospective triage research—not proof that an AI system improves immunisation coverage or prevents measles.[1]
What this means for people
- A risk flag could help overstretched vaccination teams prioritise reminders or outreach, but it should never become a reason to withhold access from a child scored as low risk.
- Families may be falsely contacted because survey reports and cards are incomplete; programmes need a simple correction route and strict limits on how household data are used.
- About one quarter of recorded defaulters in each selected test result were missed, so a model cannot replace universal access, stock management or routine follow-up.
Global context
The study is valuable because it develops and compares models using large datasets from two East African countries rather than assuming evidence from a high-income health system transfers. Yet DHS surveys are periodic research datasets, not live vaccination registries. Other countries have different schedules, data systems, mobility patterns and service constraints, so each implementation would need local data mapping, validation and community governance.
What the evidence does not yet show
- The outcome combines card records, caregiver report and 'do not know', so some apparent defaulting may be documentation error.
- Classification thresholds were optimized on the test set, weakening its independence as a final performance estimate.
- The ensemble gains over simpler tuned base models were very small; Uganda's difference was not statistically significant.
- There was no temporal, geographic or prospective external validation and no live registry integration.
- The study measured prediction, not vaccination uptake, outbreak prevention, cost, workload, fairness or family experience.
What to watch next
- Locked-model validation on newer districts and routine registry fields
- Calibration and subgroup errors, not only AUC and F1
- Prospective trials measuring verified additional vaccinations and staff workload
- Comparison with transparent logistic or rules-based prioritisation that may be easier to operate
Living evidence record
Impact record IAI-1WEPMHM
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
14 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what JMIR AI published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 14 October 2026
Research question, topics and regions
Whether machine-learning ensembles can predict measles-vaccination defaulting from demographic, socioeconomic, community and prior-immunisation variables in Uganda and Kenya
Topics: Measles vaccination · Public health · Predictive models · DHS data · Health equity · Model evaluation
Regions: Uganda · Kenya · Africa · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
An Alzheimer’s MRI model scored 94.5%. On new data, it fell to 61.6%
For a patient, the important test is whether a model works beyond familiar images. This classifier’s accuracy dropped sharply on an external dataset, and it remains a research system rather than a clinical diagnostic tool.
7 min · 1 source
Health & Life Sciences
Is neonatal respiratory AI ready for routine care?
A peer-reviewed scoping review mapped 35 studies using AI and digital tools to predict, image or monitor newborn breathing problems. The evidence spans promising prototypes, but heterogeneity, small samples, limited external validation and few clinical-impact tests keep routine use unproven.
9 min · 2 sources
Health & Life Sciences
New evidence programme centres locally led AI-health trials across Africa and Asia
APHRC and J-PAL are helping deliver the $60 million EVAH initiative to fund rigorous, locally led evaluations of AI in healthcare across Africa, South Asia and Southeast Asia.
2 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.