Can a rating scale show whether a health chatbot is good? CHARS tested experience—not clinical safety
A Nigeria-led study built a 25-item scale using a literature review, a 24-person Delphi panel and ratings from 979 adults across 36 free chatbots. Reliability was high, but accuracy, harm and patient outcomes were outside the test.
The short version
- What happened
- Researchers reduced 67 literature-derived items through three Delphi rounds with 24 panellists, then tested a 25-item scale with 979 convenience-sampled adults aged 20–35.
- Who it affects
- Patients and carers may benefit when services systematically capture whether a chatbot is understandable and usable, but a pleasant interaction must not substitute for accurate care.
- What remains unproven
- Convenience sample restricted to adults aged 20–35; 69.6% were women and the study does not establish representativeness of patients or the general public.
Keep in mind
Convenience sample restricted to adults aged 20–35; 69.6% were women and the study does not establish representativeness of patients or the general public.
Only active, free, English-language app-store chatbots were eligible; institutional, paid and non-English systems may behave differently.
Participants rated random subsets, each bot had more than 25 ratings, and the accessible record does not give a single uniform rating denominator for every chatbot.
Psychometric reliability and factor fit do not validate clinical accuracy, safety, privacy, fairness, therapeutic effectiveness or patient outcomes.
The dataset is available only on reasonable request, and independent external validation or predictive validity has not been reported.
Source analysis · 1 linked source. Primary analysed source date: 6 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct answer: CHARS can structure user ratings, but it cannot certify a safe health chatbot
A peer-reviewed study has produced a 25-item instrument for rating the perceived quality of health and social-care chatbots. The Chatbot Rating Scale, or CHARS, asks about design, interactive user experience and functionality. In the validation study, 979 adults used it to assess random subsets of 36 free English-language health and mental-health chatbots. Statistical tests found a stable three-factor structure and high internal consistency, suggesting that the questions tend to measure related aspects of the experience in this sample.
That is useful for comparing interfaces or organising user feedback. It is not the same as evaluating clinical accuracy, safety or effectiveness. A chatbot can be clear, personalised and easy to use while giving incorrect advice; a cautious tool can feel frustrating while avoiding harm. CHARS did not compare answers with clinical guidelines, measure privacy or security, test bias, follow patients or show that higher scores predict sustained use or better health. The paper itself calls for testing predictive validity and generalisability.[1]
The scale began with 3,678 records and 67 possible items
The team searched six databases for work published from 2001 through 2024 and also reviewed grey literature. Of 3,678 records identified, 32 publications remained after deduplication and title, abstract and full-text screening. The researchers extracted candidate evaluation concepts and created an initial pool of 67 items across design, personalisation, engagement, usability and functionality. Two reviewers screened records, with disagreements resolved by a third.
A modified Delphi panel then considered whether each item was relevant, clear and appropriate. The 24 panellists included clinicians, psychologists, information-technology specialists, user-experience designers, chatbot developers, researchers, students and end users. Three rounds took place over four months, with an 80% agreement threshold for retention and discussion of ambiguous items. The process left 25 questions in five conceptual dimensions before the large-sample statistical analysis regrouped them.[1]
The distinction that matters
What CHARS covers—and what it leaves outside
The validated factors describe perceived experience and function, not the full clinical-risk record.
Design
Presentation and interface
Included in CHARS
Experience
Usability, engagement and personalisation
Included in CHARS
Function
Whether expected features work
Included in CHARS
Clinical safety
Accuracy, harm and escalation
Requires separate testing
Outcomes
Health benefit and sustained use
Not established
Who rated the chatbots and what they were asked to do
The quantitative sample consisted of 979 people aged 20 to 35 recruited through professional networks, online communities and interest in digital health. It was a convenience sample rather than a probability sample of patients or the public. Women made up 69.6% of participants; almost half described themselves as beginner chatbot users, 38.0% as intermediate and 10.1% as advanced. Most accessed chatbots on smartphones. The group included health professionals, researchers, mental-health experts, students and other end users.
Researchers initially screened 314 apps and selected 36 active, free, English-language health or mental-health chatbots from major app stores. Ten were used for a pilot. In the main study, each participant rated a random subset rather than all 36. A standard prompt chain was used to make interactions more comparable, and each assigned chatbot was used for at least 10 to 15 minutes and at most 30 minutes. The paper says every chatbot received more than 25 independent ratings, but the headline record does not provide one uniform per-bot denominator.[1]
Five proposed dimensions became three statistical factors
Parallel analysis, exploratory factor analysis and confirmatory factor analysis supported three broad factors rather than the five categories produced by the Delphi process. Personalisation, engagement and usability clustered together as interactive user experience; design remained separate; and functionality formed the third factor. All 25 items were retained and redistributed. The exploratory solution accounted for 68% of total variance, and reported fit measures were within commonly used thresholds.
Internal-consistency coefficients were 0.88 for interactive user experience and 0.90 for both design and functionality. Standardised factor loadings were generally high, and the average-rater intraclass correlation was reported as 0.97 with a 95% confidence interval of 0.97 to 0.98. Measurement-invariance analysis suggested similar factor structure across men and women in this sample. These statistics show coherent ratings; they do not show that the underlying quality judgement is clinically complete.[1]
Reliability is not validity for a medical decision
A scale can be reliable when respondents answer its items consistently, yet omit the property that matters most. CHARS concentrates on observable interaction and perceived function. The study did not test whether a symptom recommendation matched a clinician's assessment, whether crisis advice escalated correctly, whether citations were genuine, whether a model exposed personal data or whether performance differed by language, disability, health literacy or clinical severity.
The chatbots also changed between use and publication, as consumer services do. A rating applies to the tested version, account state, region and prompt path, not permanently to a brand. Standardising prompts improves comparability but may miss how people actually ask fragmented, urgent or ambiguous questions. A 10-minute interaction may reveal basic usability and functionality without exposing long-conversation failures, memory behaviour, subscription barriers or deterioration after an update.[1]
How practitioners could use the scale without overclaiming
A health service or research team could use CHARS as one layer of a broader evaluation. User ratings can reveal unreadable layouts, confusing navigation, weak personalisation, poor conversational flow and functions that fail to complete. Repeating the measure with different user groups could identify where experiences diverge. Developers could track whether a redesign improves a specific dimension while keeping the underlying question set stable.
Clinical governance would still need separate tests: expert review against current evidence, predefined high-risk scenarios, red-team testing, privacy and security assessment, subgroup analysis, escalation and human-oversight checks, logging, incident response and post-deployment monitoring. Procurement should avoid turning a high CHARS score into a safety badge. The most honest label would be 'user-experience and functionality rating under specified conditions', accompanied by the version, date, participant group and clinical-safety evidence from other methods.[1]
What would change the assessment
Confidence would increase if independent teams reproduce the factor structure with older adults, adolescents, patients managing specific conditions, carers, people with disabilities and speakers of languages other than English. Validation should cover paid and institutionally deployed systems as well as free app-store chatbots. Researchers should test whether CHARS scores predict continued voluntary use, task completion or understandable information without simply rewarding agreeable wording.
The decisive next step is to connect experience with safety and outcomes without collapsing them into one number. A prospective study could pair CHARS with blinded clinical-accuracy scoring, privacy review, bias tests and patient-reported outcomes. It could then ask whether a chatbot that feels better to use is also safer and more helpful, or whether the dimensions sometimes conflict. Until that evidence exists, CHARS is a promising structured questionnaire—not a clinical certification.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring mental health: compare the finding and its evidence limits.
Can a maternal-health AI chatbot improve anxiety or depression? A 400-person trial found no 12-month benefit
A US randomized trial found no meaningful improvement in anxiety, depression, sleep, postpartum care or emergency visits after a year. The chatbot was used heavily, but it also relied on human safety follow-up and the trial was not designed to rule out small effects.
Read the next reportExplore the full source and evidence record
What this means for people
- Patients and carers may benefit when services systematically capture whether a chatbot is understandable and usable, but a pleasant interaction must not substitute for accurate care.
- Clinicians and procurement teams can use the scale to structure user-experience evidence only if clinical accuracy, privacy, bias, escalation and outcomes are assessed separately.
- People outside the 20–35 age range, non-English speakers and those with disabilities or low digital confidence were not adequately represented for broad deployment claims.
Global context
The research was led by investigators affiliated with organisations in Nigeria, South Africa and the United States and received no specific funding. It contributes a user-centred instrument from a field where evaluation frameworks are often designed in high-income settings. Yet the included apps were English-language and the participant age range was narrow. For global use, the wording and factor structure would need cultural and linguistic adaptation followed by fresh validation—not direct translation alone. Different health systems also set different boundaries for advice, professional oversight and data protection.
What the evidence does not yet show
- Convenience sample restricted to adults aged 20–35; 69.6% were women and the study does not establish representativeness of patients or the general public.
- Only active, free, English-language app-store chatbots were eligible; institutional, paid and non-English systems may behave differently.
- Participants rated random subsets, each bot had more than 25 ratings, and the accessible record does not give a single uniform rating denominator for every chatbot.
- Psychometric reliability and factor fit do not validate clinical accuracy, safety, privacy, fairness, therapeutic effectiveness or patient outcomes.
- The dataset is available only on reasonable request, and independent external validation or predictive validity has not been reported.
What to watch next
- Independent replication across ages, languages, disabilities and clinical conditions.
- Predictive-validity studies linking scores with sustained use and successful task completion.
- Joint evaluations that keep user experience, clinical accuracy, privacy and patient outcomes as separate reported dimensions.
- Versioned testing showing how scores change after chatbot model or interface updates.
- Evidence that procurement teams do not misinterpret the scale as a clinical-safety certificate.
Living evidence record
Impact record IAI-0Y4XWZD
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Scientific Reports published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
How a new 25-item rating instrument was developed and psychometrically tested for perceived quality across free English-language health and mental-health chatbots
Topics: Health chatbots · Mental health · User experience · Measurement · Clinical safety · Digital health
Regions: Nigeria · South Africa · United States · Africa · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.