Can an AI agent prepare clinical data safely? THReD succeeded often—but open-ended work remained harder
A synthetic benchmark based on clinical-data preparation reported 92.7% first-attempt task completion for one agent configuration across 141 tasks. Performance fell without documentation, and the study did not let the model handle real patient records or test clinical outcomes.
The short version
- What happened
- THReD contains 141 data-preparation tasks derived from 100 templates and seven categories. With repeats and four documentation settings, the reported evaluation covered 2,550 agent episodes.
- Who it affects
- Clinical researchers could automate repeatable preparation steps, but domain experts still need to approve cohort and outcome definitions that can change results.
- What remains unproven
- The paper is a preprint and has not been peer reviewed.
Keep in mind
The paper is a preprint and has not been peer reviewed.
One agent configuration was tested on synthetic tables designed by the study team.
The authors created the prompts, gold answers and automated checks, which can favour the benchmark's own assumptions.
Real clinical records were used for local tool validation, not given to the agent; no clinical outcomes were tested.
Eight injection tasks are too few to establish security, and one did not meet the pre-specified resistance criterion.
Source analysis · 2 linked sources. Primary analysed source date: 2 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct answer: the agent performed well on a bounded synthetic benchmark, not on live patient data
THReD is an attempt to test a less visible part of clinical AI: the work required before a model or statistician can analyse health records. Researchers routinely have to discover fields, reconcile coding systems, define cohorts, handle dates and missing values, and produce an auditable dataset. The benchmark assembled 141 tasks derived from 100 templates across seven categories. Repeated runs and four documentation conditions produced 2,550 evaluation episodes for one deployed agent configuration using Claude Haiku 5.5.
The agent completed 92.7% of tasks on its first attempt, with a reported 95% confidence interval of 88.1% to 96.4%. Allowing up to three conversational turns raised completion to 95.6%, and the paper reports 93.6% pass-at-five reliability across repeat trials. Those figures show that a tool-using model can execute many specified transformations in a controlled environment. They do not show that it can independently decide what a clinical study should measure, safely access a hospital system or produce a trustworthy research conclusion.[1][2]
What was tested and why documentation mattered
The tasks covered schema discovery, coding and label reconciliation, cohort construction, temporal alignment, data-quality checks, feature engineering and export. Synthetic tables were calibrated to patterns in HealthShare Exchange data, allowing the authors to create known answers without exposing patient records to the agent. A simulated researcher asked for a transformation, the agent selected tools or code, and rule-based checks compared the result with a gold answer.
Four context settings varied how much reference material the agent received. With the intended documentation, examples and tool descriptions, performance was high. In the no-context condition, success fell to 58%. That difference is operationally important. It suggests that the surrounding data dictionary, interface contract and examples are part of the system, not optional prose added after deployment. A hospital with inconsistent field definitions should not expect a general model to infer local meaning reliably.
The most open-ended goal-level tasks were harder, with 80.6% completion. These tasks required the agent to work from a research objective rather than follow a tightly specified operation. That gap matches a familiar human distinction: converting a clear rule into code is different from deciding which rule is scientifically justified. Clinical governance becomes more important, not less, as instructions move from exact transformations to broad analytical goals.[1][2]
The distinction that matters
What the headline percentages do—and do not—measure
THReD separates a synthetic agent benchmark from real-record validation of the underlying tools.
2,550 episodes
Synthetic agent-task evaluations
Known answers, no patient records sent to model
92.7%
First-attempt task completion
One agent configuration in the benchmark
58%
Success without reference context
Documentation materially affected performance
Real records
Local tool and pipeline validation
Not an agent test on patient data
The tool layer was checked on real records, but the model was kept away from them
The authors separately validated parts of the tool layer against real electronic health records from two chronic-disease cohorts. The tools reconstructed patient classes at roughly 91% to 92% and retained 99.7% to 99.8% of vital-sign labels. A terminology exercise began with 2,208 raw variants and mapped 99% of them. These measurements support the usefulness of the underlying preprocessing utilities, but they are not an agent accuracy score on real patients.
For reproducibility, the team replayed 2,176 executions from 128 pipelines and obtained byte-identical outputs when dependencies were pinned. A change in the pandas software version altered results until the environment was fixed. That is a valuable warning for any research group: a fluent explanation from an agent does not guarantee that a pipeline will reproduce later. Software versions, parameters, inputs and intermediate outputs need to be recorded alongside the conversation.
The authors also show why preprocessing choices can affect a substantive conclusion. Different ways of consolidating a cohort changed the estimated 90-day emergency-department revisit rate by as much as 3.2 percentage points. The benchmark does not say which estimate is clinically correct. It demonstrates that apparently technical choices can move an outcome enough to matter, so a domain expert must approve definitions before an agent automates them.[1][2]
Prompt injection and privacy need a cautious reading
THReD included eight tasks with malicious instructions planted in data or context. Under the authors' pre-specified resistance criterion, the agent resisted seven of eight. The paper also reports that it did not propose the particular harmful action requested by the planted text. With only eight tasks, neither statement establishes robust security. A single missed injection can be consequential in a system that can read files, alter data or export results.
The synthetic design reduces privacy risk and makes exact scoring possible. It also leaves open how the agent would behave with messy, rare or contradictory patterns in live clinical records. In the real-data tool-validation experiments, records remained within the local environment and were not sent to the language model. This distinction should remain visible: the paper validates a benchmark and a tool-assisted workflow, not a permission model for sending identifiable health information to a commercial model.
Any operational pilot would need least-privilege tool access, approved data boundaries, logging, independent output checks and a clear human owner for cohort definitions. It should also test attacks that are tailored to the local tools rather than relying on eight benchmark prompts. Privacy review, data-use agreements and security assessment remain prerequisites.[1][2]
Evidence limits, funding and what would change the assessment
The paper is an unreviewed preprint submitted on 2 November 2026. One model configuration was evaluated, and the authors designed the prompts, tasks, gold answers and checks. The simulated researcher followed a fixed interaction pattern. The synthetic tables cannot reproduce every ambiguity in real clinical data, and the real-data experiments sampled 5,000 records per cohort for some analyses. Multiple subgroup comparisons were not adjusted, while the terminology reference relied on one expert plus model-assisted annotation.
The authors report funding from the Robert Wood Johnson Foundation Health Data for Action programme, grant 78961, and disclose no conflicts of interest. They also disclose using an AI writing assistant. The persistent Zenodo archive helps others inspect or reproduce the benchmark, but open materials do not substitute for independent replication or prospective evaluation.
The assessment would strengthen if independent hospitals ran the benchmark on additional models and data systems, if experts blinded to agent condition judged scientific validity, and if prospective studies measured time saved, correction burden, security events and downstream research errors. For now, THReD supports a bounded conclusion: a documented tool environment let one agent complete many synthetic preprocessing tasks, while missing context and open-ended goals exposed clear weaknesses. That is useful current research, not proof of autonomous clinical-data handling and not a Breaking development.[1][2]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring ai agents: compare the finding and its evidence limits.
Can an AI agent safely summarise years of myeloma care?
A retrospective study found that a skill-guided agent agreed with expert answers on 80.1% of 469 questions from 100 patient records. It helped most on very long records, but most system errors were judged clinically significant and no live-care benefit was tested.
Read the next reportExplore the full source and evidence record
What this means for people
- Clinical researchers could automate repeatable preparation steps, but domain experts still need to approve cohort and outcome definitions that can change results.
- Patients' records were not sent to the evaluated model; any real deployment would require separate privacy, security and data-governance approval.
- Research teams need pinned environments and audit trails because software-version and preprocessing choices changed reproducibility and reported outcomes.
Global context
Clinical data structures, coding systems and privacy law vary sharply across institutions and countries. A workflow calibrated to one US health-information exchange cannot be assumed to transfer to another hospital, national health system or language without local dictionaries, governance and validation.
What the evidence does not yet show
- The paper is a preprint and has not been peer reviewed.
- One agent configuration was tested on synthetic tables designed by the study team.
- The authors created the prompts, gold answers and automated checks, which can favour the benchmark's own assumptions.
- Real clinical records were used for local tool validation, not given to the agent; no clinical outcomes were tested.
- Eight injection tasks are too few to establish security, and one did not meet the pre-specified resistance criterion.
What to watch next
- Independent replication across hospitals, data models and language-model providers.
- Prospective measurement of researcher time, correction burden and downstream analytical errors.
- Broader security testing with local tools, permissions and adversarial inputs.
- Evidence that experts agree with agent-produced cohort definitions and transformations.
- Peer review and updates to the public benchmark archive.
Living evidence record
Impact record IAI-0R2CGJZ
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
2 direct sources across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Agent-assisted clinical data preprocessing
Topics: Clinical data · AI agents · Health research · Reproducibility · Prompt injection
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.