Can AI explain bowel leakage safely? A 20-answer study found weak next-step guidance
Three clinicians rated answers from four chatbots to five commonly searched phrases. The median actionability score was 40%, and every model produced inaccurate or irrelevant statements.
The short version
- What happened
- Researchers tested five common US search phrases once in each of four free chatbots, producing 20 answers that three clinicians scored with DISCERN and PEMAT.
- Who it affects
- People who hesitate to discuss bowel leakage may turn to a chatbot first, but high reading levels and weak action steps can leave them unsure what to do or why professional assessment matters.
- What remains unproven
- Only five prompts and 20 one-shot answers
Keep in mind
Only five prompts and 20 one-shot answers
Short Google search phrases may not resemble conversational patient questions
Outputs were generated in February 2025 from model versions now replaced
Poor rater agreement for the understandability measure
No patient comprehension, behaviour, diagnosis or health outcome was measured
Source analysis · 1 linked source. Primary analysed source date: 26 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Clear-looking answers often lacked usable next steps
A peer-reviewed US study found that chatbot answers about fecal incontinence could look organised while still giving patients weak practical guidance. Researchers took five commonly searched phrases from Google Trends and entered each once into ChatGPT 4o mini, Perplexity sonar, Llama 3.2 and Gemini 2.0 flash on 24 February 2025, producing 20 answers.
Three pelvic-health clinicians rated the answers. Median quality was 44 out of 80 on DISCERN, while median PEMAT actionability was 40%. Readability ranged from a median US grade level of 9.4 for Perplexity to 13.0 for Llama—above the fifth- to sixth-grade target cited for consumer information. A colorectal surgeon separately recorded 25 inaccurate or irrelevant statements: three for Llama, four for Gemini, eight for ChatGPT and ten for Perplexity.[1]
A small baseline, not a ranking of today's chatbots
The apparent 77.8% understandability score needs care: agreement between raters was poor, so the authors said understandable content could not be concluded confidently. Model-level quality and readability differed overall, but the study had only five paired prompts; no pairwise comparison remained significant after correction. It cannot support a league table.
The queries were brief search terms rather than detailed patient questions, each answer was sampled once, and all four model versions are now superseded. No patients tested comprehension, followed advice or reported outcomes. For someone seeking help with an embarrassing condition, the useful message is practical: chatbot text may be a starting point for questions, not a substitute for clinical assessment or an authoritative action plan. Repeated testing of current models with patient-written prompts and measured comprehension would change the assessment.[1]
The distinction that matters
What the 20-answer evaluation found
Five short search phrases were entered once into four free model versions; clinicians scored the resulting text rather than testing patients.
Quality
Three raters used the 16-item DISCERN tool
Median 44/80
Actionability
Raters applied the PEMAT next-step criteria
Median 40%
Clinical review
A surgeon checked relevance and accuracy
25 inadequacies across 20 answers
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 44 min 28 sec · Background explainer
Is this the AI future we want? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Published 2026-07-01. Source checked 14 October 2026. Check captions on the publisher’s video page.
Check the healthcare evidencePhoto: Daniel Sone / National Cancer Institute / Unsplash · Licence
Continue the story
Read next
Continue exploring patient information: compare the finding and its evidence limits.
Simpler diabetes advice was easier to read—but no AI prompt preserved every strength
In 384 model answers, a sixth-grade prompt improved readability while doctor-style prompts produced more comprehensive responses. The study tested text, not whether patients understood it or made safer decisions.
Read the next reportExplore the full source and evidence record
What this means for people
- People who hesitate to discuss bowel leakage may turn to a chatbot first, but high reading levels and weak action steps can leave them unsure what to do or why professional assessment matters.
Global context
The prompts came from US Google Trends and the clinical raters worked in New York. Terminology, health literacy, access to care and model localization differ internationally, so the measured scores should not be assumed to transfer unchanged.
What the evidence does not yet show
- Only five prompts and 20 one-shot answers
- Short Google search phrases may not resemble conversational patient questions
- Outputs were generated in February 2025 from model versions now replaced
- Poor rater agreement for the understandability measure
- No patient comprehension, behaviour, diagnosis or health outcome was measured
What to watch next
- Repeated sampling from current models with version and date recorded
- Patient-authored prompts and observed comprehension across literacy levels
- Tests against current clinical guidance in multiple countries
- Whether citations, safety advice and concrete next steps improve without adding complexity
Living evidence record
Impact record IAI-0WHAGYX
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
26 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Techniques in Coloproctology published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 26 October 2026
Research question, topics and regions
The quality, understandability, actionability and readability of chatbot answers to common US searches about fecal incontinence
Topics: Patient information · Health literacy · Large language models · Pelvic health
Regions: North America · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.