Can a delirium-risk model transfer without retraining? Three in four alerts were false positives
A model deployed at a second Mount Sinai hospital kept an AUROC of 0.91 and was associated with more detected delirium. Yet its positive predictive value was only 26%, and most admissions were labelled from billing codes rather than a uniform bedside assessment.
The short version
- What happened
- The model scored 9,020 admissions at Mount Sinai Morningside and achieved AUROC 0.91, sensitivity 92% and specificity 84% at its operational threshold without local retraining.
- Who it affects
- Delirium teams may use the score to prioritise assessments, but roughly three in four high-risk classifications did not match the study's delirium label.
- What remains unproven
- Both hospitals belonged to one health system and shared EHR infrastructure, workflows and data definitions.
Keep in mind
Both hospitals belonged to one health system and shared EHR infrastructure, workflows and data definitions.
Only 2,649 of 9,020 external-site admissions received CAM; the remainder used diagnosis codes as the reference label.
The model helped select patients for CAM, creating verification and workflow-incorporation bias.
The pre-model and post-model cohorts came from different years and were not randomised.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct finding: discrimination travelled, but precision did not become clinically comfortable
A JAMA Network Open study reports that a delirium-risk model trained and developed at Mount Sinai Hospital retained an area under the receiver operating characteristic curve of 0.91 after deployment at Mount Sinai Morningside, without retraining on the second hospital's data. At the external site's prespecified threshold of 0.50, sensitivity was 92% and specificity 84%. The result matters because clinical models commonly deteriorate when moved into a different population or workflow, and local rebuilding requires data engineering, specialist staff and continuing maintenance that many hospitals cannot supply.
The operational weakness appears in the positive predictive value: 25.9%, reported as 26% in the paper. Put plainly, about one in four admissions labelled high risk met the study's delirium outcome, while about three in four did not. The negative predictive value was 99%, but that figure is aided by the low measured prevalence of 5.7% at Morningside. AUROC describes ranking across thresholds; it does not say how much work a chosen threshold creates. A service deciding whether to deploy this model must therefore budget for false alerts and trained assessment capacity, not stop at the 0.91 headline.[1]
The denominator: 33,095 admissions across two hospitals, with different labels inside the external cohort
The quality-improvement analysis covered 33,095 non-intensive-care admissions involving people aged 60 or older. During the contemporaneous deployment period from 1 December 2024 to 31 December 2025, the model scored 9,020 admissions at Morningside and 21,186 at the development hospital. The patient groups differed materially: Morningside had an older median age, more medical rather than surgical admissions, a higher recorded dementia prevalence and different racial and ethnic composition. Those differences make the transfer test more informative than reusing a held-out sample from one hospital, even though both sites remain inside one health system.
The reference standard was not uniform. At Morningside, 2,649 of 9,020 admissions had a Confusion Assessment Method assessment, while the remaining 6,371 were classified using delirium-related ICD-10-CM codes. CAM is a structured bedside assessment; a diagnosis code is created through a different clinical and administrative process and can miss undocumented cases. The model itself prioritised high-risk patients for CAM assessment, so the people receiving the stronger reference test were selected partly by the prediction under evaluation. This differential verification can make performance look better or worse in ways that a simple confusion matrix does not resolve.[1]
The distinction that matters
How the result changes from ranking to real work
Good discrimination did not remove the need for bedside confirmation or service-capacity choices.
External site
Score 9,020 admissions
AUROC 0.91 without retraining
High-risk threshold
Prioritise CAM assessments
92% sensitivity and 84% specificity
Alert burden
Compare alerts with delirium labels
26% positive predictive value
Workflow yield
Compare monthly CAM detection
9.23% before and 19.59% after
Evidence boundary
Inspect design and reference standard
Historical comparison and selective testing
Detection increased after deployment, but the before-and-after comparison is not a causal trial
The mean monthly rate of CAM-positive delirium among assessed patients increased from 9.23% before the model to 19.59% after deployment at Morningside. An adjusted logistic model estimated 1.63 times the odds of delirium detection after deployment, with a 95% confidence interval from 1.37 to 1.94. That is an important workflow result: a small delirium service used the score to focus daily assessments, and a larger share of those assessments found delirium. It indicates that risk ranking may help scarce specialist attention reach patients more likely to need it.
It does not establish that the model doubled true hospital-wide detection. The pre-model cohort covered October 2018 to September 2019, whereas the post-model cohort covered December 2024 to December 2025. Patients, documentation, staffing and clinical practice could have changed across that long interval. The denominator also changed: the earlier group included 2,889 admissions that all received CAM, while only 2,649 of 9,020 post-deployment admissions received CAM. The outcome was yield among selected assessments, not a uniformly measured delirium incidence across every admission. A randomised or stepped-wedge deployment would offer stronger causal evidence.[1]
This is within-system transportability, not independent health-system validation
The two hospitals serve different communities, but they share Mount Sinai's organisation, Epic infrastructure, data definitions, documentation practices and custom workflow tools. The development team also operated the model and evaluated its transfer. Those common elements reduce the technical shocks that often break a model elsewhere. The paper appropriately describes factors that may have preserved performance, including the heterogeneous medical and surgical development population and the shared electronic record. The evidence supports transport within this system; it cannot yet support a claim that another hospital can install the model unchanged and expect the same results.
Even the operating thresholds were site-specific: 0.50 at Morningside and 0.55 at Mount Sinai Hospital. At the development site, AUROC was 0.94, sensitivity 84%, specificity 89% and positive predictive value 19%. Different thresholds reflected local staffing and screening capacity, demonstrating that deployment is partly an operations decision rather than a property of the model alone. An organisation with fewer assessors might need a higher threshold and miss more delirium; one prioritising sensitivity might accept more false alerts. Decision-curve analysis, calibration and explicit capacity modelling would help leaders choose that trade-off more transparently.[1]
What it means for nurses, doctors, patients and hospital leaders
For the delirium team, the score is best understood as a queueing tool: it identifies whom to assess first, not who definitively has delirium. Nurses and doctors still need to recognise an acute change, complete an appropriate assessment, search for causes and act on the result. A false positive can consume scarce time and contribute to alert fatigue; a false negative can delay recognition in an older patient whose confusion may otherwise be misattributed. The visible score should therefore be paired with clear escalation routes and should never prevent assessment when clinical concern conflicts with the model.
For patients and families, earlier recognition can matter because delirium is distressing and often signals an acute medical problem. Yet this study did not randomise care or report patient-centred outcomes from the external deployment, such as duration of delirium, falls, discharge destination, mortality or family experience. It evaluated discrimination and detection yield. Hospital boards should insist on local monitoring of both benefit and burden: assessment volume, time to review, false-alert distribution, missed cases, medication use and outcomes, stratified across demographic and clinical groups. The authors reported no conflicts of interest, and the institutional review board granted a consent waiver for minimal-risk record use.[1]
What evidence would change this assessment
Confidence would rise with prospective validation in geographically and organisationally independent health systems using a frozen model, predefined thresholds and a uniform reference assessment applied to a representative sample regardless of risk score. Calibration plots, decision curves and subgroup error rates should accompany AUROC. A stepped-wedge or cluster-randomised implementation could test whether the prioritisation workflow improves timely recognition and patient outcomes without overwhelming staff. Reporting the number needed to assess per additional confirmed case would make the operational value easier for clinical services to judge.
The assessment would weaken if performance falls outside the shared Epic environment, if false positives concentrate in particular groups, or if detection gains disappear when every patient receives the same reference test. It would strengthen if independent teams reproduce the result and show improved outcomes at sustainable workload. For now, the study is consequential peer-reviewed evidence of a live model travelling between hospitals in one system, with unusually clear numbers on its false-positive burden. It is not Breaking: it reports a completed quality-improvement evaluation, not an urgent safety alert, regulatory action or newly unfolding clinical incident.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
An EHR model transferred across hospitals—but rare-disease alerts were mostly false
TRACER was pretrained on 4.5 million patient records and tested in more than two million patients from two held-out health systems. At 95% specificity it found useful risk signals, but fewer than 1% of pancreatic-cancer alerts preceded a diagnosis.
Read the next reportExplore the full source and evidence record
What this means for people
- Delirium teams may use the score to prioritise assessments, but roughly three in four high-risk classifications did not match the study's delirium label.
- Clinicians still need bedside assessment and must be able to override the score when a patient's condition raises concern.
- Patients and families may benefit from earlier recognition, but this external study did not establish better patient outcomes.
Global context
Clinical AI often fails after transfer because populations, recording practices and workflows differ. This study advances the evidence by evaluating a live deployment rather than only a retrospective test set, but its shared health-system context remains unusually favourable. The central lesson travels more broadly: decision-makers need threshold-specific workload, label quality and patient outcomes alongside AUROC before calling a model portable.
What the evidence does not yet show
- Both hospitals belonged to one health system and shared EHR infrastructure, workflows and data definitions.
- Only 2,649 of 9,020 external-site admissions received CAM; the remainder used diagnosis codes as the reference label.
- The model helped select patients for CAM, creating verification and workflow-incorporation bias.
- The pre-model and post-model cohorts came from different years and were not randomised.
What to watch next
- Independent health-system validation.
- Uniform CAM assessment in a representative sample.
- Subgroup calibration and false-positive burden.
- Patient outcomes and staff workload in a prospective design.
Living evidence record
Impact record IAI-0SNOY0R
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what JAMA Network Open published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Whether a live machine-learning delirium risk model can preserve useful performance at a second hospital without retraining, and what its workflow effects and false-positive burden mean for clinical use
Topics: Clinical AI · Delirium · External validation · Electronic health records · Alert fatigue · Older people
Regions: United States · North America
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.