Did data leakage make a Ghana kidney-screening AI look stronger?
A 155-patient preprint found 83.9% accuracy when synthetic cases were created before the train-test split, but 61.5% on 39 genuinely unseen patients. The live tool uses models fitted under the optimistic protocol.
The short version
- What happened
- The retrospective study used records from 155 patients at one Ghanaian hospital: 43 had a recorded CKD level and 112 did not.
- Who it affects
- A low-cost blood-count screen could be valuable where confirmatory testing is harder to access. But an inflated score can falsely reassure patients, create avoidable referrals or persuade clinicians that a model has travelled further than the evidence supports.
- What remains unproven
- Unreviewed retrospective preprint from one Ghanaian hospital.
Keep in mind
Unreviewed retrospective preprint from one Ghanaian hospital.
Only 155 patients and 43 CKD cases; the cleaner holdout contained 39 people and 11 cases.
The outcome was a recorded hospital CKD category, without paired eGFR, creatinine or urine-albumin verification and without confirmation of chronicity.
The deployed models were fitted under the leaky preprocessing protocol; the paper does not report prospective clinical outcomes from website use.
No separate funding or competing-interest statement was identified in the paper.
Source analysis · 1 linked source. Primary analysed source date: 31 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 5 min The evidence, methods, implications and limitations
One preprocessing choice moved the result by 22.4 points
A Ghanaian research team has measured how much a common evaluation mistake can flatter a clinical AI score. Its unreviewed study used 155 de-identified records from Ho Technical University Hospital: 43 people had a recorded chronic-kidney-disease level and 112 did not. Ten predictors—age, cardiovascular disease and eight full-blood-count measurements—fed 11 algorithms, six resampling strategies and 13 ensembles.
When the team used SMOTE to create synthetic CKD records before dividing the data, random forest and extra trees each reached 83.9% accuracy on a 56-row test partition. Their ROC AUCs were 0.925 and 0.939. But synthetic examples can be partial copies of people who later appear in the test set. In the cleaner protocol, oversampling occurred only inside training folds. A final ensemble then achieved 61.5% accuracy and ROC AUC 0.604 on 39 real, unseen patients. Its accuracy was also below the 71.8% obtained by always predicting the majority class, although its class-balanced accuracy was 0.622.[1]
The live screening tool uses the optimistic models
The paper is unusually explicit about the consequence: the two models serving NephroScan GH were fitted after SMOTE had been applied to all 155 patients. That created 69 synthetic CKD rows before the split. Among the resulting 56 test rows, accuracy was 93.3% for 15 synthetic rows and 80.5% for 41 real rows—but even the real rows could have synthetic neighbours derived from them in training. The authors therefore call the evaluation optimistic by construction.
For a person using the website, this distinction matters more than the headline AUC. In the leakage-controlled holdout, the model found seven of 11 CKD cases and missed four; it also flagged 11 of 28 people without CKD. The site reportedly averages two forests, explains their contributions and directs users to Ghanaian laboratories and renal centres. It also tells users to obtain estimated glomerular filtration rate and urine albumin-to-creatinine ratio tests. Those safeguards reduce the chance that a score is mistaken for a diagnosis, but they do not validate the deployed model.[1]
The distinction that matters
Same records, different test boundary
The model inputs stayed similar; what changed was whether synthetic cases could carry information across the train-test boundary.
Optimistic protocol
SMOTE before the split; 224 rows including 69 synthetic CKD rows
83.9% accuracy; ROC AUC 0.925–0.939
Cleaner protocol
SMOTE only inside training folds; 39 real people held out
61.5% accuracy; ROC AUC 0.604
Clinical confirmation
eGFR and urine albumin-to-creatinine testing
Still required; the model does not diagnose CKD
A larger prospective study must test the clinical question
The conservative result is itself highly uncertain. The 39-person holdout contained only 11 CKD cases; its ROC AUC confidence interval ran from 0.371 to 0.833, spanning values compatible with poor or useful discrimination. The records came from one hospital, and the outcome was its recorded CKD level rather than a protocol-verified diagnosis. The researchers lacked paired creatinine, eGFR or urine-albumin results and could not confirm the three-month chronicity criterion. Age was the only variable that separated groups after correction for multiple comparisons.
This is therefore stronger evidence about evaluation design than about kidney screening. It shows, on a concrete African dataset, why synthetic oversampling must stay inside the training process and why a live demonstration is not clinical validation. What would change the assessment is prospective, multicentre testing with prespecified thresholds, verified CKD labels, calibration, subgroup results and patient follow-up. Until then, the practical use is to prompt standard testing—not to rule kidney disease in or out.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
Can seizure-warning AI survive a clean patient split?
Five EEG pipelines fell close to the 0.5 AUC reference when tested on unseen patients. Randomly splitting overlapping windows made one baseline look much stronger, showing why a promising benchmark is not yet a usable seizure alarm.
Read the next reportExplore the full source and evidence record
What this means for people
- A low-cost blood-count screen could be valuable where confirmatory testing is harder to access. But an inflated score can falsely reassure patients, create avoidable referrals or persuade clinicians that a model has travelled further than the evidence supports.
Global context
The study addresses a real access problem: full blood counts are more widely available than kidney-function tests in many lower-resource settings. That makes careful local validation more important, not less. Models trained at one hospital can encode its age mix, disease definitions, laboratory practice and referral patterns.
What the evidence does not yet show
- Unreviewed retrospective preprint from one Ghanaian hospital.
- Only 155 patients and 43 CKD cases; the cleaner holdout contained 39 people and 11 cases.
- The outcome was a recorded hospital CKD category, without paired eGFR, creatinine or urine-albumin verification and without confirmation of chronicity.
- The deployed models were fitted under the leaky preprocessing protocol; the paper does not report prospective clinical outcomes from website use.
- No separate funding or competing-interest statement was identified in the paper.
What to watch next
- Peer review and independent reproduction of both protocols
- Replacement or revalidation of the models serving the live tool
- Prospective multicentre recruitment across Ghana
- Protocol-verified eGFR, urine-albumin and chronicity labels
- Calibration and performance by age, sex and disease stage
- Whether referrals improve timely diagnosis without excessive false alarms
Living evidence record
Impact record IAI-0HAYWC2
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
1 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what medRxiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 1 November 2026
Research question, topics and regions
Evaluation leakage in machine-learning screening for chronic kidney disease
Topics: Clinical AI · Kidney disease · Screening · Data leakage · Model validation · Preprint
Regions: Africa · Ghana
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.