An EHR model transferred across hospitals—but rare-disease alerts were mostly false
TRACER was pretrained on 4.5 million patient records and tested in more than two million patients from two held-out health systems. At 95% specificity it found useful risk signals, but fewer than 1% of pancreatic-cancer alerts preceded a diagnosis.
The short version
- What happened
- TRACER was pretrained on 4,514,400 records from four University of California health systems; disease-specific survival heads used another 501,307 records, and two other UC systems with more than two million patients were held out for external evaluation.
- Who it affects
- Clinicians could receive a manageable-looking number of alerts while still reviewing many patients who will not receive the predicted diagnosis. Patients need evidence that follow-up tests and earlier action improve health without avoidable anxiety or harm.
- What remains unproven
- Preprint not yet peer reviewed; retrospective evaluation within one university health network, despite independently operated sites.
Keep in mind
Preprint not yet peer reviewed; retrospective evaluation within one university health network, despite independently operated sites.
External cohorts exceeded two million patients in aggregate, but endpoint-specific eligibility and outcome frequencies varied.
Non-UC care, free text, imaging, pathology and genomics were absent; recorded diagnosis was the outcome.
Thresholds were estimated separately in each cohort, so transport of a fixed operational threshold remains untested.
The authors declared no competing interests; the paper did not list a separate funding statement.
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
External testing exposed the cost of a useful-looking score
A health-record model retained much of its ability to rank future disease risk after moving to two hospitals excluded from development, but clinically relevant alert performance varied sharply. Researchers pretrained TRACER on 4,514,400 patient sequences from four University of California systems and trained survival heads on a separate 501,307. They then evaluated 16 diagnoses at one, two and five years in more than two million patients from two other UC systems.
At thresholds targeting 95% specificity, one-year heart-failure alerts captured 33.6%–44.2% of future diagnoses and selected about 5.2%–5.7% of records. Chronic-kidney-disease alerts were 8.0–8.5 times as likely as an unselected record to precede diagnosis. Yet pancreatic-cancer alerts had positive predictive values of just 0.29%–0.80%: most flagged patients were not diagnosed within a year.[1]
A screening candidate, not a clinical decision rule
The study's strength is that it reports sensitivity, positive predictive value and case yield rather than relying only on AUROC. It also preserves natural disease frequencies and accounts for incomplete follow-up. The trade-off is concrete: raising specificity from 90% to 99% reduced one-year heart-failure sensitivity from 63.7% to 15.1% in the development-domain evaluation.
TRACER used coded diagnoses, procedures, medicines and laboratory results, not notes, imaging or genomics. Care outside UC was missing, diagnosis codes mark documentation rather than biological onset, and short-horizon predictions may detect an investigation already under way. Each hospital's alert threshold was estimated retrospectively. Prospective trials must test fixed thresholds, follow-up workload, equity, costs and patient outcomes before deployment.[1]
The distinction that matters
One threshold, very different clinical meaning
At nominal 95% specificity, base rates determined how often an alert preceded a diagnosis.
Heart failure
5.2%–5.7% of records flagged
33.6%–44.2% of future cases identified
Kidney disease
One-year risk alert
8.0–8.5× enrichment over baseline
Pancreatic cancer
Similar share of records flagged
Only 0.29%–0.80% positive predictive value
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
Can an ICU forecast model see kidney injury earlier?
A model trained on 13,277 ICU admissions forecast eight physiological targets and improved retrospective discrimination for 45 moderate-to-severe kidney-injury events in an external cohort. That small event count and untested workflow effects keep it far from clinical deployment.
Read the next reportExplore the full source and evidence record
What this means for people
- Clinicians could receive a manageable-looking number of alerts while still reviewing many patients who will not receive the predicted diagnosis. Patients need evidence that follow-up tests and earlier action improve health without avoidable anxiety or harm.
Global context
EHR models can look portable while their alert yield changes with prevalence, recording practice and referral patterns. Health systems outside California need local prospective evaluation rather than assuming that a stable AUROC means a stable workload or benefit.
What the evidence does not yet show
- Preprint not yet peer reviewed; retrospective evaluation within one university health network, despite independently operated sites.
- External cohorts exceeded two million patients in aggregate, but endpoint-specific eligibility and outcome frequencies varied.
- Non-UC care, free text, imaging, pathology and genomics were absent; recorded diagnosis was the outcome.
- Thresholds were estimated separately in each cohort, so transport of a fixed operational threshold remains untested.
- The authors declared no competing interests; the paper did not list a separate funding statement.
What to watch next
- Peer review
- Prospective fixed-threshold trials
- Patient-level benefits and harms
- Alert workload and follow-up costs
- Subgroup calibration and equity
- Validation outside the UC network
Living evidence record
Impact record IAI-11F7S72
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
31 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 31 October 2026
Research question, topics and regions
Whether an EHR foundation model can retain useful disease-screening performance in health systems excluded from development
Topics: Clinical AI · Electronic health records · Disease screening · External validation · Alert burden
Regions: North America · United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.