Seventy-four medical GenAI trial reports left core reproducibility details missing
An audit covering 73 randomized trials and 24,150 participants found that no applicable report fully described how poor-quality or missing input data were handled.
The short version
- What happened
- The review screened 16,485 records and included 74 reports from 73 randomized trials involving 24,150 participants; the median trial size was 100.
- Who it affects
- A patient or clinician cannot judge a trial's relevance if the report does not specify which model, inputs and human checks produced the result. Reproducible reporting is a prerequisite for safe comparison and regulation.
- What remains unproven
- 74 reports from 73 RCTs
Keep in mind
74 reports from 73 RCTs
24,150 participants
Reporting audit—not effectiveness review
Author-developed signaling questions
English-language peer-reviewed trials
Mostly text-based systems
No risk-of-bias assessment
Source analysis · 1 linked source. Primary analysed source date: 28 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
Trial reports often omitted what would let others reproduce the AI
Medical generative-AI trials frequently failed to state the model version, input rules, deployment setup or error behaviour needed to understand and reproduce an intervention, according to a peer-reviewed Journal of Medical Internet Research review published on 28 October. Among 74 reports, 17 fully reported the algorithm version, 21 described integration into the trial setting, 25 described human–AI interaction and 27 reported performance errors.
The largest gap concerned the data entering the system. Only three of 73 reports fully stated input-data eligibility and nine of 74 described how inputs were acquired or selected. None of 72 applicable reports fully explained how poor-quality or missing input data were handled. For a clinician or patient, these omissions matter because the same named model can behave differently when prompts, interfaces, access modes or incoming records change.[1]
The audit covered 73 randomized trials
Reviewers searched PubMed, CENTRAL and Embase for peer-reviewed English-language randomized controlled trials published from 2022 to 3 April 2026, then used citation searching. From 16,485 retrieved records, they included 74 reports representing 73 trials and 24,150 participants. The median trial enrolled 100 people, with an interquartile range of 50 to 247. Large language models accounted for 73 reports, and GPT-family systems for 45.
Two reviewers independently screened, extracted and rated each report against CONSORT-AI items, with disagreements resolved by a third. Agreement was substantial. The team also checked protocols and registrations for information missing from papers. This was a reporting-adherence study: it deliberately did not assess risk of bias or pool clinical effectiveness, so its findings cannot tell readers whether GenAI improved health, decisions or workflow.[1]
The distinction that matters
Four reproducibility details were often missing
The audit counted reports that fully satisfied each CONSORT-AI item.
Model version
17 of 74 reports
23.0% fully reported
Human–AI interaction
25 of 74 reports
33.8% fully reported
Poor or missing inputs
0 of 72 applicable reports
No full reporting
Existing guidance may need a GenAI-specific extension
Only nine of 74 reports explicitly cited CONSORT-AI. Citation accompanied better reporting for some interaction and data-sharing items, but major technical gaps persisted. The authors argue that conventional checklist language does not fully capture system prompts, inference settings, commercial access modes and changing third-party models. Their evidence supports a formal consensus process for a GenAI-specific extension; it does not itself create a new reporting standard.
The signaling questions used to score compliance were developed by the review team and have not been independently validated. The English-only, peer-reviewed RCT filter may also miss other randomized evaluations, and most included interventions were text based. The work was funded by the Republic of Korea's Ministry of Food and Drug Safety, with no funder role reported and no declared conflicts. Progress would mean validated criteria and trial reports that publish exact intervention specifications, inputs, failures and human roles.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring patient safety: compare the finding and its evidence limits.
Can chatbots match urologists on cancer decisions?
In 100 synthetic cancer cases, three leading chatbots were less often fully successful than two early-career urologists and made safety-critical errors far more often. Repeated answers also changed substantially.
Read the next reportExplore the full source and evidence record
What this means for people
- A patient or clinician cannot judge a trial's relevance if the report does not specify which model, inputs and human checks produced the result. Reproducible reporting is a prerequisite for safe comparison and regulation.
Global context
The trials were international, while the review team was based in the Republic of Korea. Commercial models and deployment conditions change across countries, making exact version, access and input reporting essential for transfer.
What the evidence does not yet show
- 74 reports from 73 RCTs
- 24,150 participants
- Reporting audit—not effectiveness review
- Author-developed signaling questions
- English-language peer-reviewed trials
- Mostly text-based systems
- No risk-of-bias assessment
What to watch next
- GenAI-specific CONSORT extension
- Validated reporting instrument
- Exact model and prompt disclosure
- Input-quality rules
- Human-oversight descriptions
- Multimodal trial reporting
Living evidence record
Impact record IAI-0V4S2NF
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
29 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Journal of Medical Internet Research published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 29 October 2026
Research question, topics and regions
CONSORT-AI reporting adherence in randomized trials of generative AI for medical use
Topics: Clinical trials · Generative AI · Reproducibility · Reporting standards · Patient safety
Regions: Republic of Korea · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.