Can seizure-warning AI survive a clean patient split?
Five EEG pipelines fell close to the 0.5 AUC reference when tested on unseen patients. Randomly splitting overlapping windows made one baseline look much stronger, showing why a promising benchmark is not yet a usable seizure alarm.
The short version
- What happened
- Five fixed pipelines were evaluated on 17,424 CHB-MIT EEG windows from 22 independent pediatric subject groups using 20 train-validation-test partitions that held out whole people.
- Who it affects
- A seizure warning must catch enough events without exhausting a person with false alarms. Inflated window-level scores can move a model towards trials or procurement before anyone knows whether its thresholds transfer to a new patient or remain useful over time.
- What remains unproven
- Unreviewed, non-preregistered retrospective analysis of public datasets.
Keep in mind
Unreviewed, non-preregistered retrospective analysis of public datasets.
The chronological control was post hoc and evaluated only PSD+LDA, not the four deep models.
CHB-MIT contained 22 independent pediatric groups; external testing covered 13 Siena participants and only the spectral baseline.
The study measured selected EEG windows, not continuous alarms, seizure-level sensitivity, warning time or patient outcomes.
AUC 0.5 is a reference point, not proof that a model is statistically equivalent to chance in every population.
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 5 min The evidence, methods, implications and limitations
Holding out whole people changed the headline result
A New Zealand preprint found that five seizure-prediction pipelines that can look promising under familiar evaluation schemes performed close to the 0.5 ROC reference when tested on people they had never seen. The researchers used 17,424 twenty-second EEG windows from the CHB-MIT database. Its 23 case folders represented 22 independent pediatric subject groups; 69 of 182 annotated seizures contributed preictal windows. For each of 20 fixed partitions, 14 groups trained the models, four set checkpoints and thresholds, and four were held out for testing.
The five pipelines ranged from a 90-feature spectral classifier to an EEG transformer with more than 850,000 parameters. Mean pooled test AUC ranged only from 0.527 to 0.553. Giving each of the 22 subject groups equal weight moved the range to 0.502–0.523. A single pipeline's standard deviation across partitions was 0.06–0.08—larger than the 0.026 spread between model means. The study therefore does not support ranking one architecture above another for cold-start use on an unseen patient.[1]
Random windows shared more information than a future alarm would have
The team traced every window back to its recording and seizure. In all ten cases eligible for the original index-split audit, at least one seizure contributed windows to more than one partition and raw EEG samples overlapped across a boundary. A stricter control kept complete preictal blocks together and ordered them in time. Across 11 eligible cases and 11,175 windows, the fixed PSD-plus-LDA baseline averaged AUC 0.572 under chronological blocks, compared with 0.815 when the same windows and class counts were randomly allocated. Random allocation scored higher in every case.
Threshold transfer exposed another practical gap. A validation rule designed to label at most 1.67% of interictal windows positive produced mean test false-positive proportions of 1.81–3.13% on new people and exceeded its target in six to ten of 20 partitions, depending on the model. At those strict thresholds, only 2.33–5.56% of preictal windows were detected. A zero-shot external check of the spectral baseline on 5,657 windows from 13 Siena participants produced pooled AUC 0.480. The four deep models were not tested externally.[1]
The distinction that matters
The same model, three different questions
Changing the test boundary changes what a reported score means, even when the pipeline stays fixed.
Unseen people
Hold out four of 22 subject groups
Five pipelines: mean pooled AUC 0.527–0.553
Within one person
Keep seizure blocks chronological
PSD+LDA mean AUC 0.572 across 11 cases
Random windows
Shuffle the same windows and class counts
PSD+LDA mean AUC 0.815, with related time information mixed
Window classification is still several steps from a warning device
The paper is a retrospective computational audit, not a prospective test of continuous alarms. Sampled-window false positives are not false alarms per hour, and the analysis did not measure seizure-level sensitivity, time in warning, alarm persistence or how a patient would act. The chronological control was added after the original results were inspected, was not preregistered and used only the simple spectral baseline. CHB-MIT is a pediatric dataset collected at one centre, while Siena is a separate small public dataset; neither represents the full range of people living with epilepsy.
For patients, the result is a warning about evidence quality rather than proof that seizure forecasting is impossible. Developers should keep people, seizure episodes and overlapping raw samples on one side of the train-test boundary, freeze thresholds before external testing and report continuous-alarm outcomes. Regulators and hospitals should ask what information crossed each split before treating an AUC as clinically meaningful. Auckland University of Technology supported the work; the authors declared no competing interests and released code, partitions, window manifests and predictions for comparison.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
Did data leakage make a Ghana kidney-screening AI look stronger?
A 155-patient preprint found 83.9% accuracy when synthetic cases were created before the train-test split, but 61.5% on 39 genuinely unseen patients. The live tool uses models fitted under the optimistic protocol.
Read the next reportExplore the full source and evidence record
What this means for people
- A seizure warning must catch enough events without exhausting a person with false alarms. Inflated window-level scores can move a model towards trials or procurement before anyone knows whether its thresholds transfer to a new patient or remain useful over time.
Global context
Public EEG datasets make reproducible research possible, but small, centre-specific cohorts encourage repeated reuse and fragile comparisons. Common released partitions and external prospective recordings would help teams compare models on the same clinical question instead of on whichever split produces the strongest score.
What the evidence does not yet show
- Unreviewed, non-preregistered retrospective analysis of public datasets.
- The chronological control was post hoc and evaluated only PSD+LDA, not the four deep models.
- CHB-MIT contained 22 independent pediatric groups; external testing covered 13 Siena participants and only the spectral baseline.
- The study measured selected EEG windows, not continuous alarms, seizure-level sensitivity, warning time or patient outcomes.
- AUC 0.5 is a reference point, not proof that a model is statistically equivalent to chance in every population.
What to watch next
- Peer review and independent reproduction using the released manifests
- External testing of all five pipelines
- Causal preprocessing that never looks into future EEG
- Seizure-level sensitivity and false alarms per monitoring hour
- Prospective performance after patient-specific calibration
- Patient-reported usefulness and burden
Living evidence record
Impact record IAI-0YCETA2
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
1 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what medRxiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 1 November 2026
Research question, topics and regions
Evaluation design for EEG seizure-prediction models
Topics: Clinical AI · Seizure prediction · EEG · Data leakage · External validation · Preprint
Regions: Oceania · New Zealand · North America · Europe
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.