Can an ICU AI model survive a hospital change?
Three mortality models lost reliable decision behaviour when moved between MIMIC and eICU. One random forest made no positive predictions after transfer, showing why a respectable AUC cannot substitute for local calibration and threshold testing.
The short version
3 key points
- 1.The primary analysis used 110 MIMIC ICU episodes with 33 deaths and 1,324 eICU episodes with 117 deaths, restricting inputs to shared variables observed during the first 24 hours.
- 2.When trained on eICU and tested on MIMIC, AUCs were 0.598 to 0.642. At the fixed 0.5 threshold, the random forest predicted no deaths and gradient boosting detected only two of 33 deaths.
- 3.The study is a retrospective stress test, not a deployed clinical trial. Its small, selected MIMIC demo cohort and non-representative eICU subset limit the size of any general claim.
The answer: ranking patients is not the same as making safe decisions
An ICU model can retain some ability to rank higher-risk patients and still fail at the point where a score becomes an alert. In a new cross-dataset study, a random forest trained on eICU and tested on MIMIC produced an area under the receiver-operating curve of 0.642, modestly above chance. Yet at the conventional probability threshold of 0.5 it labelled every one of 110 ICU episodes as a survivor, missing all 33 deaths. Gradient boosting detected two deaths and missed 31. A single discrimination score would not reveal that operational collapse.
The practical finding is not that one algorithm is uniquely unsafe. All three tested approaches—logistic regression, random forest and gradient boosting—had weak transportable discrimination, poor or unstable calibration, or damaging threshold behaviour when moved between the datasets. Hospitals therefore need to test the entire decision pathway on local, later patients: input definitions, missingness, probability calibration, alert threshold, false positives, false negatives and the action that follows. Importing a published model and retaining its old threshold is not evidence of safety.[1]
What the researchers compared
Researchers in Ecuador used two established US critical-care resources. MIMIC-III is a single-centre database from Beth Israel Deaconess Medical Center, while eICU combines records from multiple hospitals. The primary cohort retained ICU episodes observable for at least the first 24 hours and used only clinical variables sufficiently represented in both sources. It contained 110 MIMIC ICU episodes from 79 patients and 105 admissions, including 33 deaths, and 1,324 eICU episodes from 1,124 patients and 1,277 hospital-system stays, including 117 deaths.
The unit of analysis was the ICU episode and the outcome was in-hospital mortality. The team constructed 35 pre-encoding predictors from 11 shared vital-sign and laboratory families plus age and sex. Temperature was excluded because 91.8% of eICU values were missing under the corrected 24-hour definition. The design deliberately removed endpoint-derived and source-specific fields, including eICU predicted mortality and APACHE score, to reduce leakage and make the two domains more comparable. Logistic regression, random forest and gradient boosting were then evaluated internally and in both transfer directions.[1]
What happened when eICU models met MIMIC patients
With eICU as the training source and MIMIC as the test target, AUC was 0.598 for logistic regression, 0.642 for random forest and 0.613 for gradient boosting. Logistic regression detected 18 of 33 deaths but also flagged 33 of 77 surviving episodes, producing sensitivity of 0.545 and specificity of 0.571. Random forest made no positive predictions. Gradient boosting produced only three positive alerts: two were deaths and one was a survivor, for sensitivity of 0.061. None of these operating points is a credible clinical alert system.
The uncertainty was substantial. Patient-cluster bootstrap intervals placed random forest AUC between 0.503 and 0.765, while logistic regression ranged from 0.459 to 0.720 and gradient boosting from 0.461 to 0.755. The random forest's zero-sensitivity interval was mechanically fixed at zero because resampling an all-negative set of predictions cannot create a positive decision; it does not prove the population sensitivity is known with certainty. That distinction is important whenever a seemingly precise interval describes a degenerate model output.[1]
Reverse transfer did not solve the problem
Training on the much smaller MIMIC cohort and testing on eICU also produced weak results. Logistic regression detected 44 of 117 deaths but generated 422 false positives, giving precision of 0.094. Random forest detected only four deaths, and gradient boosting detected 31; their respective precision values were 0.103 and 0.109. A clinician facing those alerts would either miss most deaths or review a large volume of false alarms, depending on the model.
Event prevalence differed sharply: 30.0% in the selected MIMIC landmark cohort and 8.8% in eICU. Measurements, hospital mix and missingness also differed. Those shifts change the meaning of a probability and the balance of errors. Even nominally shared variables can be collected with different devices, timing and clinical protocols. Structural column matching is therefore not clinical harmonisation, and recalibration alone cannot repair a model whose ranking performance is weak or whose target population is poorly represented.[1]
A warning that also appeared before transfer
The random forest's threshold failure was not created solely by moving hospitals. In internal eICU out-of-fold evaluation, it achieved an AUC of 0.694 but sensitivity of 0.009 at the same 0.5 threshold. In a separate eICU holdout it had 91.2% accuracy and an AUC of 0.765 while making no positive predictions. Because deaths were uncommon, an all-survivor policy looked accurate. That is a textbook example of why accuracy can conceal failure on the outcome that matters.
For clinicians, safety teams and procurement staff, the lesson is concrete. Ask for the confusion matrix at the proposed operating threshold, not only AUC. Compare sensitivity, specificity, precision, calibration, average precision and workload against the local event rate. Predefine what action an alert should trigger and the acceptable number of missed events and false alarms. If a vendor changes a threshold after seeing external test results, that site is no longer an untouched validation set and a fresh evaluation is needed.[1]
The study is a useful stress test, not a deployment trial
The MIMIC side used the small MIMIC-III demo population, which was selected from patients who eventually died and is not representative of a general ICU population. Its 110 landmark episodes came from only 79 patients. The eICU data were an available local subset rather than a hospital-stratified sample, and hospital representation was not used for inference. The analysis was retrospective, used public US datasets and did not place scores in front of clinicians or measure care, alarm burden, treatment changes or patient outcomes.
The authors accounted for repeated episodes with patient-grouped validation and cluster bootstrap intervals, but the reported transfer intervals condition on the fitted source model and do not include uncertainty from refitting it. The study did not define a clinical-utility threshold or calculate net benefit. It received no financial support and the authors declared no commercial conflict. They also disclosed using generative AI for language refinement, grammar and structure while stating that the study design, processing, analysis and interpretation were their own.[1]
What would change the assessment
Confidence would rise with a locked model evaluated on complete, representative datasets from hospitals that played no role in development. Variables should be harmonised by clinical meaning and timing, not merely by name. Thresholds need to be chosen in advance from source validation or explicit utility criteria, then tested without adjustment. Results should be reported by hospital, patient group and missingness pattern, with calibration and decision curves alongside discrimination.
The decisive evidence would be prospective and workflow-based: does the alert reach the right clinician, improve an action, reduce missed deterioration and avoid harmful alarm fatigue compared with ordinary care? Independent monitoring should detect drift in prevalence, inputs and error rates after launch, with a safe fallback when the model is outside its validated domain. Until that evidence exists, the paper supports a governance rule rather than a clinical product: model transfer is a new test, and a respectable AUC is not permission to deploy.[1]
What this means for people
- Patients can be harmed by both silent false negatives and alarm-driven overtreatment when a model's threshold is moved unchanged to another hospital.
- Clinicians need local confusion matrices, calibration and workload estimates at the actual alert threshold—not a headline AUC alone.
- Public-service buyers should require untouched external validation and prospective workflow evidence before treating a transported model as clinically ready.
Global context
The authors were based in Ecuador, but the patient records came from US critical-care datasets. Both sources are widely used for research, yet they do not represent all hospitals, countries, contemporary care pathways or data systems. Transfer to another health service requires a new local evaluation.
What the evidence does not yet show
- Retrospective analysis of public US critical-care data rather than prospective clinical use
- Only 110 MIMIC episodes from the selected demo cohort in the primary analysis
- The eICU extract was an available subset rather than a hospital-stratified representative sample
- Transfer uncertainty intervals did not include uncertainty from refitting the source model
- No prespecified clinical utility threshold, net-benefit analysis or patient outcome evaluation
What to watch next
- Locked external validation on complete, representative hospital datasets
- Threshold selection based on clinical utility before external testing
- Prospective effects on missed deterioration, false alarms and clinician workload
- Performance and calibration by hospital and patient subgroup
- Ongoing monitoring for prevalence, measurement and workflow drift
Living evidence record
Impact record IAI-1MZIFDO
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
15 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Frontiers in Digital Health published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 15 October 2026
Research question, topics and regions
Whether ICU mortality models retain discrimination, calibration and clinically usable decisions when transferred between MIMIC and eICU data
Topics: Clinical AI · Dataset shift · Intensive care · External validation · Calibration · Patient safety
Regions: United States · Ecuador · Americas
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
AI basics · Background explainer
Published 2018-11-28 · 2 min 31 sec
What is artificial intelligence? | The Royal Society
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
AI Risks & Safety
Do AI models favour one cancer guideline?
Four models produced 6,000 choices across 15 conflicts between US immunotherapy guidelines and usually selected NCCN. But 10 conflicts involved one lung-cancer setting, and removing them weakened or reversed the pattern for most models.
7 min · 2 sources
AI Risks & Safety
Can hidden image text mislead dental AI?
A peer-reviewed German stress test found that adversarial text placed inside 270 dental radiographs could flip four vision-language models from an abnormal to a normal finding. OCR sanitisation sharply reduced the measured attacks, but the experiment used a permissive prompt, a pathology-heavy benchmark and no live clinical system.
7 min · 3 sources
AI Risks & Safety
Can an AI worm replicate across smart devices yet?
Anthropic made one large open-weight model spread a smaller model across a deliberately vulnerable eight-node test network. Small models could configure unfamiliar hardware but could not complete the whole chain; the team projects, rather than demonstrates, that the gap may close this year.
8 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.