Did AI find cultural health differences—or just data leakage and sample-size noise?
A peer-reviewed reanalysis of 482 Indian and 70 Japanese students found a perfect obesity classifier collapsed to chance after BMI was removed. The study is a methods warning, not a national comparison.
The short version
3 key points
- 1.The final convenience samples contained 482 students from one Indian university and 70 from one Japanese university, after eligibility and exclusion rules.
- 2.Removing BMI from a model that defined obesity using BMI dropped AUC from 1.00 to 0.50–0.57; stress prediction reached AUC 0.49–0.71.
- 3.Every Japanese feature-importance value fell within the range generated by repeatedly shrinking the Indian sample to the same size.
The perfect obesity model had been given the answer
A machine-learning model appeared to classify student obesity perfectly until researchers removed the variable used to define obesity. In a peer-reviewed reanalysis of university cohorts in India and Japan, discrimination fell from an area under the curve of 1.00 to 0.50–0.57 after body mass index was excluded as a predictor. That is essentially chance performance and a textbook example of target leakage.
The dataset came from surveys conducted in June–August 2024. Of 1,238 initial responses, age and health exclusions left 482 students at one Indian university and 70 at one Japanese university. The teams tested penalised logistic regression, random forests and gradient-boosted trees with held-out evaluation, permutation importance and bootstrap intervals. The paper is best read as an audit of methods, not a map of two national cultures.[1]
Unequal samples can manufacture a cultural story
The researchers also revisited claims that different lifestyle features mattered in the two countries. When they repeatedly reduced the Indian cohort to match the Japanese sample of 70, every feature-importance value observed in Japan fell within the range produced by sample-size variation alone. That does not prove the cohorts are identical; it means this dataset cannot distinguish a cultural signal from instability created by the eightfold size difference.
The only remaining modelled outcome was elevated self-rated stress. Held-out AUC ranged from 0.49 to 0.71, from chance to modest discrimination. The stress measure was a single 0–10 item rather than the validated ten-item scale described in the original plan. Diabetes and hypertension could not support modelling after the stated exclusions were correctly applied: eligible positive cases disappeared, and hypertension was not asked in Japan.
Those details show why an attractive cross-country narrative can survive several layers of weak evidence. A model may fit circular labels, an importance ranking may move with sample size, and inconsistent survey instruments may be mistaken for population difference. None of those problems is repaired by choosing a more complex algorithm.[1]
The distinction that matters
Three checks changed the interpretation
Each check asks whether the model learned a health signal or a property of the dataset.
Leakage
Remove BMI from the obesity predictors
AUC falls from 1.00 to 0.50–0.57
Validation
Score models on held-out data
Stress prediction is modest at best
Sample size
Shrink India to n=70 repeatedly
The apparent country gap fits sampling noise
A defensible comparison starts before model training
For digital-health teams, the practical checklist is concrete: list every variable used to construct the outcome, remove those variables from the predictors, reserve a held-out test set, report uncertainty and repeat the comparison with matched sample sizes. Cross-cultural work should also test whether an instrument means the same thing in each setting before comparing model explanations.
The evidence check cuts both ways. This reanalysis is transparent and includes reproducible code, but it still relies on self-reported measures and convenience samples from one institution per country. The two samples are not representative of India or Japan, and the study performed no formal measurement-invariance test. Confidence in a cultural difference would require harmonised, validated measures; larger multi-site probability samples; preregistered outcomes; external validation; and stable effects after sample-size matching. The study’s contribution is therefore corrective: it shows how much apparent insight can disappear when the leakage is removed.[1]
Evidence check
Self-reported data, one university per country, n=70 in Japan, inconsistent instruments and no national sampling or formal measurement-invariance test constrain the conclusions.
Reporting basis: Source analysis. Original source date: 16 October 2026.
Explore the full source and evidence record
What this means for people
- Students and patients can be misrepresented when unstable model explanations are labelled as cultural traits; careful validation helps prevent weak evidence from shaping screening or policy.
Global context
Unequal and unrepresentative datasets are common in cross-country health AI. The same leakage and sample-size checks apply far beyond India and Japan.
What the evidence does not yet show
- Self-reported data, one university per country, n=70 in Japan, inconsistent instruments and no national sampling or formal measurement-invariance test constrain the conclusions.
What to watch next
- Prospective multi-site studies with harmonised validated measures, balanced recruitment, preregistered analysis, external validation and uncertainty around feature-importance comparisons.
Living evidence record
Impact record IAI-0R69H3R
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
16 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Digital Health published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 16 October 2026
Research question, topics and regions
Leakage-free reanalysis of cross-cultural lifestyle-health models
Topics: Digital health · Machine learning · Data leakage · Cross-cultural research · Reproducibility
Regions: India · Japan · Asia
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Healthcare, culture & human impact · Background explainer
Published 2026-07-01 · 44 min 28 sec
Is this the AI future we want? | The Royal Society
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
Can a routine blood test triage gastric-cancer risk?
A six-variable model reached an AUC of 0.831 in 735 later patients at one Chinese hospital. It could become a low-cost prompt for endoscopy, but it has no external or prospective validation and age and sex may explain part of its performance.
7 min · 1 source
Health & Life Sciences
Can routine lab results distinguish two stroke types?
A ten-biomarker model reached an AUC of 0.902 in 495 later patients at the same Chinese hospital. The result is promising decision-support research, but unknown blood-draw timing, no stroke mimics and no external hospital mean it cannot replace emergency brain imaging.
6 min · 1 source
Health & Life Sciences
Can AI identify a medicine safely outside the lab?
A review of 91 studies finds rapid progress in pill detection, classification and imprint reading, but little external, prospective or clinical validation. High benchmark accuracy is not evidence that an app prevents medication harm.
7 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.