Did an LLM help reviewers find cancer-treatment skin toxicity?
In a 160-patient retrospective study, LLM-assisted reviewers extracted skin toxicities from clinical notes more accurately and consistently than unassisted reviewers. It did not test real-time diagnosis, treatment decisions or patient outcomes.
The short version
3 key points
- 1.The retrospective sample contained 160 immune-checkpoint-inhibitor recipients treated at Massachusetts General Hospital, Brigham and Women’s Hospital or Dana-Farber Cancer Institute.
- 2.Assisted reviewers reached F1 0.88 versus 0.77 without assistance, with sensitivity 0.91 versus 0.79, specificity 0.92 versus 0.87 and Cohen’s kappa 0.82 versus 0.50.
- 3.The experiment measured research chart abstraction, not live clinical diagnosis: one specialist supplied the reference labels, the model was tuned on the 100-case benchmark and no treatment or patient outcome was tested.
The answer: assistance improved a narrow retrospective review task
An LLM-supported workflow helped non-dermatologist reviewers identify skin toxicities linked to cancer immunotherapy more accurately and consistently in a small retrospective study. Compared with unassisted manual review, the assisted group achieved an F1 score of 0.88 rather than 0.77. Agreement between reviewers also rose from a Cohen’s kappa of 0.50 to 0.82, while average self-reported review time was about one-third shorter.
Those results matter because immune checkpoint inhibitors can trigger inflammatory adverse events, and evidence may be scattered across lengthy clinical notes. Researchers building cohorts or safety registries spend substantial time deciding whether a recorded rash or other skin problem was plausibly treatment related. A tool that retrieves the relevant passages and leaves a person to accept, modify or reject its recommendation could reduce that abstraction burden.
The study did not show that the system diagnoses patients, alerts clinicians in time to prevent harm or improves treatment. It reconstructed events from existing records for research review. The correct reading is therefore specific: in this sample and workflow, people given LLM-generated evidence performed a binary chart-classification task better than a separate unassisted pair. That is promising evidence for human-supervised data extraction, not a clinical deployment claim.[1]
What the researchers tested in 160 patient records
The researchers randomly selected 160 people from a clinical database of patients who received immune checkpoint inhibitors at Massachusetts General Hospital, Brigham and Women’s Hospital or Dana-Farber Cancer Institute. The notes covered the period from two weeks before treatment began through two years and 90 days after the first dose. All three institutions belong to the same Boston-area academic ecosystem, so the study is multi-hospital but not a geographically independent validation.
One board-certified supportive oncodermatologist reviewed the records and supplied the reference classification for whether each patient had a cutaneous immune-related adverse event. That expert adjudication gives the study a clinically informed target, but a single adjudicator cannot reveal how often specialists would disagree. It also means the reported accuracy is agreement with one expert-derived reference, not proof that every label represents an objectively certain event.
A 100-patient holdout set was first used to compare four language models in a zero-shot classification task. GPT-4o produced sensitivity of 0.82 and specificity of 0.88, ahead of Llama 3.1-70B, Gemma-2-27B and Qwen-2.5-72B on the balance reported. The researchers then iteratively refined prompts over ten rounds on those same 100 cases before building the final assisted workflow. That reuse is important: it can make the benchmark fit the observed sample better than a fully untouched test would.[1]
The system retrieved passages, proposed a label and invited correction
The final workflow combined dense and sparse retrieval, including BM25 keyword matching, to find potentially relevant text in a longitudinal record. A synthesiser agent assembled the evidence and a critic agent checked the proposed classification. Reviewers saw a binary recommendation together with supporting excerpts rather than an unexplained score. The model ran through GPT-4o on a HIPAA-compliant Azure environment inside Mass General Brigham’s secure infrastructure.
This design addresses two common problems in clinical language-model work. First, retrieval narrows attention to the notes most likely to contain an event, which can make a long record manageable. Second, showing the underlying excerpts gives a reviewer something concrete to inspect. It does not guarantee factuality: an irrelevant passage can still be retrieved, and a persuasive summary can still be wrong. The ability to reject or edit the output is therefore part of the intervention, not an optional safety decoration.
Four non-dermatologist reviewers were trained by the specialist and divided into two pairs. One pair reviewed records manually; the other used the LLM interface and could accept, modify or reject its recommendation. Because different people populated the two conditions, the comparison combines the effect of the tool with any differences between reviewers. A crossover design in which the same reviewers used both workflows on separate, randomised cases would have separated those effects more cleanly.[1]
Accuracy and agreement improved, but the denominator is modest
The assisted reviewers achieved sensitivity of 0.91, specificity of 0.92, precision of 0.86 and F1 of 0.88. The unassisted pair achieved sensitivity of 0.79, specificity of 0.87, precision of 0.76 and F1 of 0.77. Raw agreement within the assisted pair was 92.6%, compared with 76.5% for manual review; chance-adjusted agreement rose to kappa 0.82 from 0.50, with a reported p value below 0.001.
These measures capture different costs. Higher sensitivity means fewer specialist-labelled toxicities were missed; higher specificity means fewer records without the target event were wrongly marked positive. Precision tells researchers how many flagged cases were true according to the reference. F1 summarises the balance between recall and precision, while kappa asks how consistently the two reviewers reached the same answer beyond expected chance agreement.
The paper’s peer-reviewed abstract describes review time as approximately one-third shorter. That is a correction worth preserving because the earlier manuscript version described the saving as roughly half. Time was self-reported rather than recorded automatically, and a modest number of reviewers completed a specialised task after study training. The finding supports a prospective timed evaluation; it does not yet establish staffing savings in an operational cancer service.[1]
The most defensible use is research abstraction, not bedside diagnosis
For clinical researchers, the practical opportunity is faster construction of adverse-event datasets. Better extraction could support safety surveillance, retrospective studies and quality review when ordinary billing codes miss events or fail to capture why clinicians thought a symptom was treatment related. Human review remains valuable because the classification depends on chronology, alternative explanations and context that may not be stated in one sentence.
For clinicians and patients, the boundary must remain visible. The model was not evaluated as an alert during treatment, did not compare itself with dermatologists making live diagnoses and did not measure whether anyone received earlier care. A missed toxicity could delay assessment; a false positive could lead to anxiety, referrals or treatment disruption. Before any clinical use, a locked system would need prospective testing at the point where an alert changes work, with clear escalation rules and patient-important outcomes.
The study also used protected records and a commercial model in a secure institutional environment. Other hospitals may have different note styles, terminology, patient populations, documentation gaps and cloud arrangements. Reproducing the workflow requires not only prompts but compatible data access, privacy controls, model-version tracking, logging and a workforce able to review the evidence. Accuracy in one research pipeline does not make those organisational requirements disappear.[1]
Evidence limits and what would change the assessment
This is a peer-reviewed open-access Brief Communication with a clearly described clinical question, comparator and reference process. It reports both autonomous-model and human-assisted performance, and the assisted people outperformed GPT-4o acting alone. Funding came from research foundations and US public grants; the corresponding author disclosed advisory or consulting relationships and honoraria from several pharmaceutical and biotechnology companies, stated as financial competing interests.
The evidence remains preliminary. The total sample was 160 people; the prompt was refined on the 100-case benchmark; one specialist supplied the reference labels; and only four non-dermatologist reviewers took part. The hospitals belong to one health system and region. Review time was self-reported. There was no prospective workflow, no independent external cohort, no cost analysis and no test of whether the system changed diagnoses, treatment or outcomes.
Confidence would rise with a preregistered, locked-model study at independent hospitals, multiple specialist adjudicators, automatically recorded review time and a crossover or randomised reader design. Clinical use would need a prospective trial that measures missed and false alerts, time to specialist assessment, treatment changes, patient harm and performance across demographic and cancer subgroups. Until then, the result supports a carefully supervised research tool, not autonomous toxicity surveillance.[1]
What this means for people
- Researchers and registry staff may spend less time searching long cancer records, but they still need authority and training to reject the model’s recommendation.
- Clinicians should not treat the reported accuracy as evidence for real-time diagnosis or changing immunotherapy; neither use was tested.
- Patients would need transparent governance, privacy protection and evidence that alerts improve timely care without creating harmful false positives before clinical deployment.
Global context
The records came from three Boston-area academic hospitals within one US health system and the implementation used GPT-4o through a secure Azure environment. Note structure, immunotherapy pathways, privacy law, specialist access and cloud infrastructure differ internationally, so the reported figures are not transferable performance guarantees.
What the evidence does not yet show
- The retrospective sample contained 160 patients from three hospitals in one regional academic system.
- One oncodermatologist supplied the reference labels, so specialist disagreement was not measured.
- The prompt was refined in ten iterations on the same 100-case benchmark used for model comparison.
- Four non-dermatologist reviewers were divided between conditions rather than each testing both workflows.
- No prospective alerting, diagnosis, treatment decision, cost or patient outcome was evaluated.
What to watch next
- Locked-model validation in independent hospitals and health systems
- Multiple-specialist adjudication and subgroup performance with confidence intervals
- Randomised or crossover reader studies with automatically measured time
- Prospective evidence on alerts, treatment decisions, harm and patient outcomes
Living evidence record
Impact record IAI-0MDW7R4
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
15 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what npj Precision Oncology published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 15 October 2026
Research question, topics and regions
Whether a retrieval-augmented, multi-agent LLM can help non-dermatologist reviewers identify cutaneous immune-related adverse events in oncology notes
Topics: Cancer care · Immunotherapy · Adverse events · Large language models · Clinical records · Human oversight
Regions: United States · Global oncology research
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Healthcare, culture & human impact · Background explainer
Published 2026-07-01 · 44 min 28 sec
Is this the AI future we want? | The Royal Society
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
Did clinicians prefer AI discharge summaries after long hospital stays?
In a retrospective 60-case comparison, 12 physicians usually preferred GPT-5.2 summaries and annotated fewer omissions. Reviewers knew which summary was AI-written, one hospital supplied the records, and no patient outcome or time saving was tested.
7 min · 3 sources
Health & Life Sciences
Can chatbots match urologists on cancer decisions?
In 100 synthetic cancer cases, three leading chatbots were less often fully successful than two early-career urologists and made safety-critical errors far more often. Repeated answers also changed substantially.
7 min · 1 source
Health & Life Sciences
Will patients wait for a clinician to check AI medical advice?
In a survey of 1,185 US adults, 63.1% preferred waiting up to 48 hours for clinician-reviewed medical portal answers, even as majorities accepted five-minute AI responses for logistical tasks. The study tested hypothetical choices, not real behaviour or safety.
7 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.