Did expert review improve a pediatric AI assistant? Ratings rose, but 1 in 8 sessions still fell short
In 2,000 sampled consultations, paediatricians rated the optimized system clinically appropriate in 87.4% of sessions, up from 61.4%. The before-and-after study measured expert ratings, not patient outcomes, and the system's developer helped conduct the research.
The short version
- What happened
- The retrospective study covered 16,074 consultations about 8,266 children across 31 of China's 34 provincial-level divisions; 43.8% of sessions occurred outside routine clinic hours.
- Who it affects
- Families often used the assistant outside clinic hours, when escalation routes matter most.
- What remains unproven
- Retrospective before-and-after design without a randomized concurrent control.
Keep in mind
Retrospective before-and-after design without a randomized concurrent control.
One physician rater per sampled session and no reported patient outcomes.
Most users were in Chongqing, limiting national and international generalization.
The system's commercial co-developer participated in development and supplied four authors.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct answer: expert-guided changes improved ratings, not proven clinical outcomes
Paediatricians judged a publicly available Chinese health assistant more favourably after the developers revised its prompts, knowledge base, tools and validation rules. In a random sample of 1,000 consultations before that work, 614 sessions, or 61.4%, were rated clinically appropriate. In a separate 1,000-session sample after optimization, 874, or 87.4%, received that rating. The share rated good overall—meaning that the response fully addressed the caregiver's question with actionable information—rose from 65.2% to 74.3%. Those are substantial changes in expert assessment and a useful signal that continuing clinical review can expose correctable weaknesses in a deployed assistant.
They are not evidence that the system made children healthier or safer. The study was a retrospective before-and-after comparison, not a randomized trial with a concurrent unchanged assistant. Different users, questions or seasonal illnesses could have affected the two samples. The reviewers examined deidentified text rather than observing what caregivers did next. The researchers did not count correct diagnoses, treatment changes, emergency referrals, delayed care, adverse events or patient outcomes. The appropriate conclusion is therefore that an expert-guided development cycle improved paediatricians' ratings within this service, while clinically consequential benefit and harm remain unmeasured.[1]
What the real-world dataset contains—and who used it
The study analysed every consultation submitted to Xiaoyi between its launch on 7 April and 31 December 2025. The WeChat mini program recorded 7,243 caregiver accounts, 8,266 child profiles, 16,074 consultation sessions and 54,771 conversational turns. Children had a median age of 4.2 years; 4,705 of 8,266 profiles, or 56.9%, were male. Registrations came from 31 of China's 34 provincial-level divisions, although Chongqing accounted for 65.6% of the users. The broad geographic count should therefore not be mistaken for a nationally representative sample. Promotion through the hospital's WeChat account and posters also shaped who encountered the free service.
Use clustered around practical questions. Symptom consultation was the reason for 7,845 sessions, or 48.8%; report interpretation accounted for 2,531, or 15.7%; and triage guidance for 2,515, or 15.6%. Child health care, pulmonology and gastroenterology were the most frequent subspecialty labels. Among symptom consultations, fever and cough dominated. Importantly, 7,046 of the 16,074 sessions—43.8%—occurred outside the stated 08:00 to 17:30 clinic hours. That pattern supports the claim that families wanted information when routine services were less available, but it does not establish that the assistant substituted for a clinician or prevented an unnecessary visit.[1]
The distinction that matters
What improved—and what was not measured
The study compared expert ratings before and after a bundle of system changes; it did not follow clinical outcomes.
Before
1,000 sessions rated
61.4% clinically appropriate
Optimization
Prompts, knowledge, tools and rules changed
No single component isolated
After
A different 1,000 sessions rated
87.4% clinically appropriate
Unmeasured
No outcome linkage
Benefit and harm to children remain unknown
How 40 paediatricians produced the before-and-after comparison
A multidisciplinary panel of 40 paediatricians first agreed assessment criteria, then reviewed two independently drawn samples. One thousand sessions came from the pre-optimization period and 1,000 from the post-optimization period. Each session was assessed by one paediatrician for overall quality and clinical appropriateness. Reviewers also labelled six kinds of turn-level deficiency, including insufficient information gathering, inaccurate interpretation, poor clinical recommendations and communication problems. The researchers compared categorical session ratings with chi-square tests and analysed turn-level deficiency rates with generalized estimating equations to account for multiple turns within the same session.
The turn-level result moved in the same direction as the session ratings. Reviewers found a predefined deficiency in 790 of 4,073 turns before optimization, or 19.4%, and in 236 of 3,419 afterwards, or 6.9%. The reported odds ratio was 0.30, with a 95% confidence interval from 0.24 to 0.37 and P below .001. Suboptimal clinical recommendations remained the most frequent problem after the changes. The denominator is essential: the result describes reviewed conversational turns, not unique children with a clinical error, and several deficient turns could occur in one consultation without producing a documented injury.[1]
What changed inside the assistant
Xiaoyi combined the Qwen3-32B language model and Qwen3-VL-32B vision-language model with retrieval and other tools on local institutional infrastructure. It was not connected to the hospital's electronic health record. After the first evaluation, the team revised prompts, expanded the knowledge base and available tools, and added validation rules. That is a bundle of changes rather than a single isolated intervention. The study cannot say which component produced the improvement, whether a smaller rules-based change would have done the same, or how much maintenance will be needed as medical guidance and user behaviour change.
The research team also used language models to classify all sessions by subspecialty, reason for consultation and symptoms, with paediatrician validation. In random validation samples of 1,000 sessions, discordance was reported at roughly 3% to 4%, and symptom matching differed in 3.1%. That supports the labelling pipeline for broad description, while leaving room for misclassification in rare or ambiguous questions. The system's responses carried a disclaimer that they were health guidance rather than diagnosis or a substitute for professional consultation. A disclaimer is relevant context, but it does not itself prevent over-reliance when a worried caregiver receives confident, specific advice.[1]
Commercial involvement and study design narrow the inference
The Children's Hospital of Chongqing Medical University and Zuoyi Tech co-developed the assistant. Four study authors were affiliated with Zuoyi Tech: the conflicts statement identifies one as the company's chief executive and three as employees, and says the company group participated in technical development. Other authors declared no competing interests. The article lists hospital and public research affiliations and identifies authors who acquired funding, but the accessible funding text does not change the central disclosure: the product developer helped build the system and contributed authors to its evaluation. That does not invalidate the measurements, but it raises the value of independent replication.
Several methodological choices also favour caution. The pre- and post-optimization samples were different rather than matched, there was no unchanged control system running over the same dates, and each consultation had one physician rater rather than independent duplicate review. Reviewers knew which content they were assessing within the improvement programme, and the paper does not establish blinding to period. Users were self-selected, most came from one municipality, and no overseas registrations were recorded. The evidence is stronger than a vendor testimonial because the article is peer reviewed and reports denominators and criteria, but weaker than a prospective controlled clinical evaluation.[1]
What families and health services can use now—and what evidence is still needed
For families, the study shows why an answer from a child-health assistant should remain a starting point rather than a diagnosis. The system was most often asked about symptoms, reports and triage—the very situations in which age, examination findings and changing severity can alter advice. Caregivers should be given clear red flags, easy escalation to a clinician and an explanation of what information the system lacks. For health services, the practical lesson is that deployment must include regular sampling by the relevant specialists, predefined failure categories, versioned changes and a way to investigate any report of delayed or inappropriate care.
Confidence would increase with a prospective comparison conducted over the same period, independent double-rated samples, preregistered safety outcomes and linkage—under appropriate consent and privacy controls—to later service use and clinical records. Researchers should measure whether advice changes caregiver decisions, whether urgent cases are escalated promptly, whether unnecessary visits fall without missed illness, and whether performance is equitable across ages, regions and conditions. External teams should reproduce the findings on other models and languages. Until then, the study supports expert monitoring as an improvement process; it does not establish that a paediatric chatbot can safely replace access to a clinician.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
What are clinicians asking an AI evidence platform?
A JAMA study counted 271 million OpenEvidence queries from 952,008 US clinicians. Treatment and management dominated a classified sample—but the logs cannot show whether answers were correct, used in care or improved patient outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- Families often used the assistant outside clinic hours, when escalation routes matter most.
- A clinically appropriate rating does not show that a caregiver followed the advice or that the child benefited.
- Health services need specialist sampling and incident review after deployment, not only a launch-time benchmark.
Global context
Health systems are testing conversational tools to extend information outside normal hours, especially where paediatric access is uneven. This study adds unusually large real-world usage data, but it also shows the evidence gap between better expert-rated text and safer care. Regulators and hospitals will need outcome monitoring, escalation design and independent evaluation alongside response-quality audits.
What the evidence does not yet show
- Retrospective before-and-after design without a randomized concurrent control.
- One physician rater per sampled session and no reported patient outcomes.
- Most users were in Chongqing, limiting national and international generalization.
- The system's commercial co-developer participated in development and supplied four authors.
What to watch next
- Prospective outcome-linked evaluations.
- Independent duplicate physician ratings.
- Performance on urgent and rare paediatric conditions.
- Equity across regions, ages and language varieties.
Living evidence record
Impact record IAI-0HJ4X66
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Journal of Medical Internet Research published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
How caregivers used a publicly available paediatric health assistant in China and whether expert-guided changes improved paediatricians' ratings of its responses
Topics: Paediatrics · Clinical AI · Caregivers · Health information · Expert review · Patient safety
Regions: China · Asia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.