An AI agent sped cancer-toxicity review—but prospective performance was lower
A Boston system reduced staff review time by 2.71 minutes per note in a 17-person crossover study. Silent prospective detection F1 was 0.72–0.79, below the 0.92 retrospective result.
The short version
- What happened
- Researchers tested six immune-related adverse-event categories in 263 retrospective test notes, 884 prospectively processed notes and a randomized crossover study involving 17 clinical research staff.
- Who it affects
- A patient receiving immunotherapy can develop serious side effects whose details are scattered through notes. Faster review may help research and safety monitoring, but the system has not shown that it improves diagnosis, treatment or outcomes.
- What remains unproven
- Boston academic centres
Keep in mind
Boston academic centres
263 retrospective test notes
884 silent prospective notes
17-person crossover study
No change to patient care
No patient outcomes
External validation still needed
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
Faster review, with a smaller real-time score
An agentic language-model system helped clinical research staff review cancer-immunotherapy toxicity notes faster and more consistently, according to a peer-reviewed Nature Cancer study published on 29 October. In a randomized crossover study, 17 staff members each reviewed 20 notes with and without assistance. The assisted condition reduced annotation time by 2.71 minutes per note, increased the odds of an exact match with the reference label by 45% (95% CI 1.01–2.08), and raised agreement between reviewers.
The technical result weakened when the system moved from a retrospective test to silent real-time use. Current-event detection reached a macro F1 of 0.92 on the retrospective evaluation, while binary F1 across three months of prospective processing ranged from 0.72 to 0.79. F1 combines precision and recall; it does not say whether every clinically dangerous event was caught. Multi-class severity grading was harder still, with retrospective F1 of 0.66.[1]
Three phases tested different questions
The team assembled notes from adults treated with immune checkpoint inhibitors at Boston academic hospitals between 2015 and 2024. It focused on six adverse events—pneumonitis, hepatitis, myocarditis, dermatitis, thyroiditis and colitis—and asked the system to extract whether an event was current or past, its grade, clinician-attributed cause and certainty. Phase one used 289 notes from 100 encounters: 26 development notes and 263 test notes.
Phase two processed 884 notes prospectively in silent mode from May to July 2025. The first 102 were dual-annotated; the remainder received one human annotation. Phase three randomized the order of assisted and manual review for 17 coordinators and nurses. That design measures annotation work, not clinical diagnosis: no prediction changed treatment, triggered an alert or demonstrated fewer hospital admissions, missed toxicities or deaths.[1]
The distinction that matters
Each phase answered a different question
The study moved from technical testing to silent deployment and then to staff workflow.
Retrospective
263 held-out notes
Current-event F1 0.92
Silent prospective
884 notes over three months
Binary F1 0.72–0.79
Crossover
17 staff; 20 notes each
2.71 minutes saved per note
Human verification remains part of the system
The best configuration used repeated model runs and consensus, costing about US$0.021 per note. Removing that self-consistency step approximately halved cost but reduced detection F1 from 0.92 to 0.78. A 779-term rules baseline also performed reasonably for simple binary detection, a reminder that the more complex agent must justify its extra cost through reliable extraction of grade, attribution and supporting evidence—not merely by finding keywords.
The study was funded by NIH/NCI and other research funders; several authors disclosed healthcare, pharmaceutical or AI-company relationships. External hospitals, other record systems and outcome-focused trials remain necessary. The assessment would change if independent teams reproduce the prospective results, measure event-specific misses and show that human-verified alerts improve timely care without adding harmful false alarms or automation bias.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring immunotherapy: compare the finding and its evidence limits.
Did an LLM help reviewers find cancer-treatment skin toxicity?
In a 160-patient retrospective study, LLM-assisted reviewers extracted skin toxicities from clinical notes more accurately and consistently than unassisted reviewers. It did not test real-time diagnosis, treatment decisions or patient outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- A patient receiving immunotherapy can develop serious side effects whose details are scattered through notes. Faster review may help research and safety monitoring, but the system has not shown that it improves diagnosis, treatment or outcomes.
Global context
The work was conducted in US academic oncology settings. Documentation styles, staffing, adverse-event prevalence and record systems vary across countries and hospitals, so transfer requires local evaluation and governance.
What the evidence does not yet show
- Boston academic centres
- 263 retrospective test notes
- 884 silent prospective notes
- 17-person crossover study
- No change to patient care
- No patient outcomes
- External validation still needed
What to watch next
- Independent external validation
- Event-specific sensitivity
- Prospective clinical alerts
- Automation-bias monitoring
- Patient-safety outcomes
- Performance across record systems
Living evidence record
Impact record IAI-0WQYJSX
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
29 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Nature Cancer published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 29 October 2026
Research question, topics and regions
Agentic extraction and human review of immune-related adverse events
Topics: Cancer · Immunotherapy · Adverse events · Clinical records · AI agents · Human oversight
Regions: United States · North America · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.