Can AI turn rare-disease notes into reliable treatment evidence?
A 108-patient Rett-syndrome study extracted treatment-related features from 26,763 notes, but only one of 18 targets had a full ground truth—and documentation is not clinical improvement.
The short version
3 key points
- 1.The single-centre study analysed 26,763 notes from 108 patients with sustained trofinetide exposure over 30 months.
- 2.On 183 expert-labelled patient-timepoint cells for severe diarrhoea, the selected prompt reached 84.2% accuracy and F1 0.674 versus a regular-expression baseline.
- 3.The other 17 phenotypes had only six expert spot checks each, so their rates are exploratory; the system measured documented presence, not improvement or causation.
The system organised notes; it did not prove a drug worked
Researchers used two large language models in agreement to turn 26,763 clinical notes from 108 people with Rett syndrome into a timeline of 18 treatment-related features. The pipeline could help rare-disease teams search narrative records that are too labour-intensive to review repeatedly. Its output, however, is whether a feature was documented at a visit—not whether the patient improved, worsened or changed because of trofinetide.
The unreviewed study came from Texas Children's Hospital and followed patients with confirmed sustained exposure for up to 30 months. It tracked four adverse effects, 13 features of Rett syndrome and medication refusal. Notes first passed a keyword filter; two model calls then classified the same material, and the pipeline required consensus before producing a patient-by-timepoint matrix.[1]
Only severe diarrhoea had a full reference set
Prompt design was developed on severe diarrhoea, the treatment's best-known adverse effect. The reported comparison used 183 expert-labelled cells from 56 patients and included 47 positives. The selected prompt scored 84.2% accuracy and F1 0.674, 47 percentage points above regular-expression matching. But it was selected and scored on the same cells, so the authors call that accuracy an in-sample upper bound rather than independent validation.
For the other 17 targets, experts reviewed only six calls per phenotype—three positive and three negative—and agreed with 92 of 102. That 90.2% spot-check agreement cannot establish sensitivity or prevalence. Later follow-up also thinned sharply: just 13 evaluable cells remained at 24 months and six at 30 months. Clinical notes missed information carried by phone or patient portals, and time-window placement explained 12 of the 29 diarrhoea discrepancies. The four-person clinical team also resolved labels together rather than annotating independently, so the reliability of the reference standard itself could not be measured.[1]
The distinction that matters
What the pipeline can and cannot conclude
The same note moves through filtering and two model calls before becoming a timeline entry.
Find
Keyword-filter 26,763 notes
Candidate passages for 18 features
Classify
Require two LLMs to agree
Documented presence at each timepoint
Clinical meaning
Compare with severity and external evidence
Still needed to infer improvement or harm
Why clinicians still need a separate evidence step
The tool could make chart review more scalable, especially when a small specialist service needs to locate side effects or follow-up discussions. It cannot attribute trajectories to the medicine rather than concurrent care or the underlying condition. Thirteen core features were recorded as present in 92–100% of patients at every visit, illustrating why presence alone cannot reveal treatment response.
The work had public and charitable funding. Two authors disclose consulting relationships, including with trofinetide maker Acadia Pharmaceuticals; the paper does not say those funders or companies controlled the analysis. The underlying notes cannot be shared because of privacy rules, although prompts, keyword lists and aggregate results are available. A credible next step needs independent annotations for every phenotype, external hospitals, severity or direction-of-change measures and prospective tests of whether clinicians find problems faster without missing them. Until then, this is a record-structuring study, not evidence of clinical benefit.[1]
Evidence check
The study is an unreviewed, single-centre retrospective analysis. Only severe diarrhoea had a full expert reference set, and the pipeline measured documented presence rather than direction of change, severity or causation.
Reporting basis: Source analysis. Original source date: 15 October 2026.
Explore the full source and evidence record
What this means for people
- Families and specialist teams could gain a faster way to find treatment-related information scattered through years of notes.
- Patients could be misled if a documentation trend is presented as improvement, deterioration or a drug effect without clinical adjudication.
Global context
The cohort came from one US children's hospital. Rett care, note structure, language and access to trofinetide differ across health systems, so the extraction method requires local validation before wider use.
What the evidence does not yet show
- The study is an unreviewed, single-centre retrospective analysis. Only severe diarrhoea had a full expert reference set, and the pipeline measured documented presence rather than direction of change, severity or causation.
What to watch next
- Multi-centre validation with independently double-annotated phenotypes, explicit severity measures and tests of clinician decisions and missed events.
Living evidence record
Impact record IAI-0A7KQ9Y
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
16 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Palacios et al. (arXiv) published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 16 October 2026
Research question, topics and regions
Dual-LLM extraction of trofinetide treatment phenotypes from clinical notes
Topics: Rett syndrome · Clinical notes · Treatment monitoring · Large language models · Real-world evidence
Regions: United States
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Healthcare, culture & human impact · Background explainer
Published 2026-07-01 · 44 min 28 sec
Is this the AI future we want? | The Royal Society
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
Did an LLM help reviewers find cancer-treatment skin toxicity?
In a 160-patient retrospective study, LLM-assisted reviewers extracted skin toxicities from clinical notes more accurately and consistently than unassisted reviewers. It did not test real-time diagnosis, treatment decisions or patient outcomes.
9 min · 1 source
Health & Life Sciences
Can EHR-note AI help distinguish epilepsy from PNES?
EpiScreen separated epilepsy from psychogenic non-epileptic seizures across 13,633 records from two US datasets and improved accuracy in an eight-clinician simulation. It remains retrospective decision support, not a substitute for neurological assessment or video-EEG.
8 min · 2 sources
Health & Life Sciences
Did clinicians prefer AI discharge summaries after long hospital stays?
In a retrospective 60-case comparison, 12 physicians usually preferred GPT-5.2 summaries and annotated fewer omissions. Reviewers knew which summary was AI-written, one hospital supplied the records, and no patient outcome or time saving was tested.
7 min · 3 sources
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.