Can AI make breast-cancer notes easier to understand? Comprehension improved—but clinicians found 82 errors
In a peer-reviewed mixed-methods study, 200 healthy women over 40 answered more action questions and reported greater confidence when clinical notes included GPT-4-generated simplifications, definitions and summaries. Yet three clinician reviewers found 82 errors across the generated material, including 28 judged potentially harmful or misleading, while only seven participants with breast-cancer histories tested the interface with their own notes.
The short version
- What happened
- Two hundred healthy women over 40 completed a randomised, within-person note-reading study; the seven people with breast-cancer histories contributed qualitative interviews and notes rather than the main comprehension denominator.
- Who it affects
- Patients may understand next steps better when jargon and key actions are explained beside the original note.
- What remains unproven
- The main comprehension sample comprised 200 healthy women over 40, not patients currently receiving breast-cancer care.
Keep in mind
The main comprehension sample comprised 200 healthy women over 40, not patients currently receiving breast-cancer care.
Only twelve notes were used, including four clinician-written synthetic notes and eight donated real notes.
The study measured short-term questions and perceptions, not treatment decisions, clinical outcomes or live-clinic workload.
Seven interview participants cannot establish population-wide acceptability, and the sample had limited demographic diversity.
The GPT-5.5 analysis was limited and did not provide a comparable, independently adjudicated safety rate.
Source analysis · 1 linked source. Primary analysed source date: 5 November 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The answer: the interface helped readers in a controlled study, but the generated text was not reliably safe
A peer-reviewed Scientific Reports study published on 5 November found that language-model additions to breast-cancer notes improved both objective question scores and readers' confidence in a controlled web experiment. Two hundred healthy women aged over 40 read three notes with different levels of assistance: the original note alone; selected help comprising a simplified version and definitions; or a fuller package adding frequently asked questions, extracted key information and a to-do list. Every participant experienced all three assistance levels in random order, while assignment to real or synthetic notes was between participants.
With all augmentations, the mean action-understanding score rose from 0.537 without help to 0.605, a statistically significant difference reported at p=0.02. Self-rated understanding increased from 1.45 to 2.25 on the study's three-point scale, and confidence about next steps from 2.17 to 2.95, both at p<0.001. For real donated notes alone, action understanding rose from 0.451 to 0.541; for synthetic notes the change from 0.622 to 0.669 was not statistically significant.
The safety result pulls in the opposite direction. Three clinician researchers reviewed GPT-4-generated outputs for twelve notes and identified 82 errors. Twenty-eight, or 34%, were classified as potentially harmful and misleading. Most errors occurred in outputs derived from real donated notes, while only one appeared in augmentations of synthetic notes. The study therefore supports a narrow conclusion: carefully presented AI assistance can make notes easier to use in a test, but generated explanations still require clinical review and should not replace the original record or professional advice.[1]
What the tool generated and how it was tested
The researchers used GPT-4-0314 to produce five kinds of augmentation. A simplification rewrote the note in plain English while retaining diagnoses and medicines. Definitions explained detected medical terms. A frequently-asked-questions panel generated seven questions and answers. A key-information panel extracted medicines and appointments, while a to-do list attempted to identify actions the patient should take before the next visit. The interface always allowed readers to toggle back to the original note, an important safeguard when the added text was incomplete or wrong.
The note corpus was small. One clinician wrote four synthetic notes, and seven participants with a history of breast cancer donated eight de-identified real notes dated between 2007 and 2023. The resulting twelve notes averaged 322 words. The 200-person survey drew four real and four synthetic notes from that pool, randomly showing each participant three. After each note, readers answered four multiple-choice action questions, then rated their understanding and confidence. The paper reports corrected confidence intervals and statistical tests, but did not measure whether readers would make safer treatment decisions or experience better health outcomes.
The qualitative component was separate. Seven people with a history of breast cancer used the interface with their own notes and joined video interviews. They all had prior patient-portal experience and adequate health literacy, and all but one already agreed that AI could benefit medicine. Participants described definitions and integrated explanations as convenient and less overwhelming, but worried that an augmentation could omit nuance, make an assumption or create misplaced confidence. Their experiences give useful design insight, not a representative estimate of acceptability among all cancer patients.[1]
The distinction that matters
Three samples answer three different questions
The study combines a survey, interviews and clinician review. Their denominators should not be merged into one claim.
200 healthy women
Read notes under three interface conditions
Objective question scores and self-rated confidence
7 breast-cancer survivors
Used the interface with donated notes and joined interviews
Detailed experiences, not population prevalence
12 notes
Three clinicians reviewed GPT-4 augmentations
82 identified errors, including 28 potentially harmful or misleading
Limited GPT-5.5 check
One clinician and an LLM judge examined updated outputs
Fewer old errors but new errors and judge disagreement
Why the error denominator changes the interpretation
The paper's 82 errors are not an error rate over 200 people. They came from clinician review of generated augmentation sets across twelve notes, with multiple augmentation types per note. The study does not provide a single clinically validated denominator such as errors per 1,000 statements, nor does it estimate how often a patient would act on an error. The count nevertheless shows that apparently helpful text can contain many small assumptions and a consequential subset of misleading content.
Definitions were a frequent source of problems. A term may have several clinical meanings, while the model inferred one from incomplete context. Other errors involved omissions, phrasing or invented details. The concentration in real notes is especially important: clean synthetic examples can understate the ambiguity, shorthand and missing context found in actual documentation. Because readers could still inspect the source note, the experiment also tested an augmented interface rather than a standalone AI summary. Removing that comparison route could change both comprehension and risk.
An exploratory update with GPT-5.5 suggested fewer errors, but the comparison was limited and should not be treated as a new head-to-head safety trial. A clinician initially identified about four problems, which expanded to ten after reviewing suggestions from an LLM judge. The automated judge flagged 97 possible issues, including 33 it called harmful, but the clinician agreed with only six of those missed on the initial pass; it also missed clinician-found errors. New errors appeared even when older GPT-4 problems were resolved. Model updates therefore do not remove the need for human review and fresh validation.[1]
What the numbers mean for patients and health services
The objective gain was real within the experiment but modest in absolute terms: about 6.8 percentage points on the action questions when all augmentations were compared with control. Self-rated gains were larger, which can be useful if confidence encourages people to engage with their records, but confidence can also outrun correctness. The study did not show that a reader could detect every hallucination, recognise a missing qualification or know when the original note itself contained an error.
Simplification and definitions were the most popular features. Eighty-three and a half per cent of survey participants said they would use simplification again, and 72.5% chose definitions. The selected two-feature package slightly outperformed the larger bundle on some measures, supporting a design lesson: adding more generated panels can introduce cognitive load and more opportunities for misunderstanding. A health service considering such a tool should test each feature separately rather than assuming a larger interface is automatically more helpful.
Operational deployment would also require privacy, governance and escalation controls beyond this study. Clinical notes contain protected information. A service would need an approved processing environment, logging, version control, clinician ownership of corrections and a clear way for patients to report errors. Patients should be told which text is generated, retain access to the untouched source note and receive guidance on questions that require the care team. The paper did not evaluate those processes in a live clinic.[1]
What remains unproven—and what would change the assessment
The 200-person experiment did not recruit people currently navigating breast-cancer treatment. Participants were healthy women over 40 recruited through Prolific, many with adequate health literacy and favourable views of medical AI. Most participants were white and non-Hispanic, Latino or Spanish origin. The seven people with breast-cancer histories were highly selected, comfortable sharing notes and experienced with patient portals. Language, disability, literacy, access and cultural differences remain under-tested.
The evidence would strengthen with a preregistered multi-site trial involving a larger and more diverse patient population, many more note types and a human-written plain-language comparator. Researchers should report statement-level clinical error rates, severity adjudication, disagreement between reviewers and how often patients notice or act on errors. Outcomes should include question quality, clinician workload, anxiety, follow-up adherence, privacy incidents and whether the tool narrows or widens understanding gaps.
For now, the study offers a useful product-design warning rather than permission for autonomous deployment. AI can help people navigate difficult notes when the original remains visible and the generated layer is treated as fallible. The same interface can also produce plausible misinformation. Services should evaluate the exact model, prompts, notes and presentation they intend to use, repeat testing after model changes, and keep clinicians responsible for patient-facing clinical meaning.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring clinical notes: compare the finding and its evidence limits.
Can AI turn rare-disease notes into reliable treatment evidence?
A 108-patient Rett-syndrome study extracted treatment-related features from 26,763 notes, but only one of 18 targets had a full ground truth—and documentation is not clinical improvement.
Read the next reportExplore the full source and evidence record
What this means for people
- Patients may understand next steps better when jargon and key actions are explained beside the original note.
- Potentially harmful or misleading generated text means the tool cannot safely substitute for the record or the care team.
- Clinicians and service leaders need version-specific monitoring because model upgrades can fix some errors while introducing others.
Global context
The study was conducted in the United States and in English. Open-note rules, portal access, privacy requirements and clinical terminology vary internationally. Translation quality and culturally appropriate explanations require separate testing rather than assuming an English-language tool transfers safely.
What the evidence does not yet show
- The main comprehension sample comprised 200 healthy women over 40, not patients currently receiving breast-cancer care.
- Only twelve notes were used, including four clinician-written synthetic notes and eight donated real notes.
- The study measured short-term questions and perceptions, not treatment decisions, clinical outcomes or live-clinic workload.
- Seven interview participants cannot establish population-wide acceptability, and the sample had limited demographic diversity.
- The GPT-5.5 analysis was limited and did not provide a comparable, independently adjudicated safety rate.
What to watch next
- Patient trials during active care with diverse languages, literacy levels and cancer pathways.
- Statement-level error denominators and independent multi-clinician severity adjudication.
- Comparisons with human-written explanations and simpler non-generative definitions.
- Whether improved comprehension changes adherence, anxiety, clinician messages or health outcomes.
- Safe update procedures when the underlying model, prompts or clinical documentation system changes.
Living evidence record
Impact record IAI-0MHR7GS
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Scientific Reports published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Patient-facing LLM augmentation of breast-cancer clinical notes
Topics: Breast cancer · Clinical notes · Patient understanding · Large language models · Clinical safety
Regions: United States
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.