Can AI catch specialist referrals that primary care forgot to send? Singapore's system found more—but missed half
Across seven public clinics, a weekly AI safety check found 225 clinician-verified referral failures over 20 weeks. Its 48.8% sensitivity means it cannot be the only safeguard.
The short version
- What happened
- The MR-AI system scanned roughly 25,000 consultation notes each week across seven Singapore primary-care clinics for clear referral intent without a matching electronic order.
- Who it affects
- A missed referral can leave a patient waiting for an appointment that was never ordered. This system gave clinics another chance to find and repair some of those failures, but it missed about half of the reference cases and did not demonstrate faster treatment or better health outcomes.
- What remains unproven
- Prospective quasi-experimental study
Keep in mind
Prospective quasi-experimental study
Seven clinics in one Singapore health system
20-week intervention
About 25,000 notes scanned weekly
428 flags and 225 clinician-verified cases
Single-week 20,369-note accuracy sample
43 reference missed referrals
48.8% sensitivity
99.9% specificity
Human reference standard
No patient outcome or cost-effectiveness result
True incidence unknown
Proprietary system and prompts
Source analysis · 1 linked source. Primary analysed source date: 30 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 5 min The evidence, methods, implications and limitations
A weekly scan found referral failures before patients had to chase them
An AI-supported safety check substantially increased detection of specialist referrals that clinicians intended but did not order. Across seven public primary-care clinics in western Singapore, MR-AI scanned about 25,000 consultation notes each week. Over a 20-week intervention it raised 428 possible failures; senior clinicians verified 225. Patients separately reported 31, taking the total detected during the period to 256.
The peer-reviewed study compared that period with two matched 20-week baselines, each of which recorded 18 patient-reported missed referrals. An interrupted time-series model estimated an immediate 4.7 percentage-point increase in the weekly proportion of clinic visits with a detected missed referral (95% confidence interval 2.4 to 6.9 points; p<0.001), about seven times the pre-implementation proportion. This measures detection, not proof that the underlying failure rate rose or that patients had better health outcomes.[1]
High specificity kept the review list short, but sensitivity was moderate
MR-AI first used rules to filter notes, then a locally deployed Llama 3.1 8B model classified whether the text expressed definite referral intent without a corresponding order. In a blinded accuracy exercise covering 20,369 notes from one week, seven reviewers established 43 missed referrals. The AI found 21 and missed 22: sensitivity was 48.8% (95% CI 33.3% to 64.5%). It correctly left almost all negative notes unflagged, with 99.9% specificity, and 21 of its 34 flags were confirmed, a positive predictive value of 61.8%.
That operating point was deliberately conservative to limit alert fatigue. Manual review of the same week took about 41 more staff-hours than AI processing plus verification, according to recorded spreadsheet activity and system logs. Across the full intervention, 225 of 428 AI flags were confirmed, a 52.6% verified-flag rate. Clinicians still had to inspect every flag, contact patients, correct the referral and record the incident; the model did not complete those safety tasks.[1]
The distinction that matters
What the blinded accuracy week showed
Seven reviewers assessed 20,369 notes without recorded referral orders. Their adjudicated decisions formed the study's reference standard.
Reference missed referrals
43 cases
21 found by MR-AI; 22 missed
Sensitivity
48.8%
The safety net detected about half of reference cases
Specificity
99.9%
Only 13 of 20,326 reference negatives were flagged
AI flags
34 cases
21 confirmed; PPV 61.8%
The system is a backstop, not evidence that referral care improved
For patients, an earlier warning could shorten the period in which an intended specialist appointment simply does not exist. For clinic teams, scanning can focus scarce review time on a short list rather than every note. But the study did not measure waiting time, diagnosis, treatment, harm avoided or cost-effectiveness. Only 123 of the 225 clinician-verified cases were entered in the operational incident system, and the clinical urgency of many referrals could not be determined retrospectively.
This was an observational evaluation in one Singapore health system. Patient reporting and retrospective screening were both incomplete, the true incidence is unknown, and the reference standard depended on human interpretation. The one-week accuracy sample had a 0.21% prevalence, which makes overall accuracy and negative predictive value look especially high. Seasonal or behavioural changes could also affect the time-series result. Larger prospective studies should test patient outcomes, costs and performance across clinical settings. Until then, MR-AI is evidence for an additional safety net—not for removing ordering controls, documentation standards or human oversight. The authors reported no specific funding or competing interests, although the institutional system and prompts are proprietary.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
Can chatbots match urologists on cancer decisions?
In 100 synthetic cancer cases, three leading chatbots were less often fully successful than two early-career urologists and made safety-critical errors far more often. Repeated answers also changed substantially.
Read the next reportExplore the full source and evidence record
What this means for people
- A missed referral can leave a patient waiting for an appointment that was never ordered. This system gave clinics another chance to find and repair some of those failures, but it missed about half of the reference cases and did not demonstrate faster treatment or better health outcomes.
Global context
Missed referrals occur in health systems beyond Singapore, but referral language, electronic records, specialty access and safety-reporting practices vary. Replication should measure performance and patient consequences in the local workflow rather than transfer this low-prevalence accuracy result unchanged.
What the evidence does not yet show
- Prospective quasi-experimental study
- Seven clinics in one Singapore health system
- 20-week intervention
- About 25,000 notes scanned weekly
- 428 flags and 225 clinician-verified cases
- Single-week 20,369-note accuracy sample
- 43 reference missed referrals
- 48.8% sensitivity
- 99.9% specificity
- Human reference standard
- No patient outcome or cost-effectiveness result
- True incidence unknown
- Proprietary system and prompts
What to watch next
- Prospective patient-outcome evaluation
- Cost-effectiveness
- Performance across other health systems and languages
- Sensitivity improvements without alert overload
- Real-time referral-order checks
- Complete incident logging
- Monitoring by specialty and urgency
- Independent evaluation of the proprietary workflow
Living evidence record
Impact record IAI-0K6MKFI
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
30 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what npj Digital Medicine published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 30 October 2026
Research question, topics and regions
AI surveillance for missed specialist referrals
Topics: Primary care · Patient safety · Specialist referrals · Clinical AI · Large language models · Workflow evaluation
Regions: Singapore · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.