How safe was ChatGPT health advice in four Global South countries?
An analysis of 15,555 donated health conversations rated about one in five high-risk. The estimate comes from an unreviewed convenience sample and an LLM judge, not clinician review or measured patient harm.
The short version
3 key points
- 1.Researchers identified 15,555 health conversations in complete ChatGPT histories donated by 1,252 users in India, Nigeria, Brazil and Pakistan.
- 2.An LLM judge rated 20.6% to 26.3% of health conversations high-risk by country; symptom and ongoing-care questions had the highest rates.
- 3.The sample was young, digitally literate and self-selected, safety labels were not clinician adjudications, and the study measured potential risk rather than patient harm.
The answer: the audit found substantial risk, but not a country prevalence rate
In a consent-based corpus of real ChatGPT histories, an automated safety audit rated roughly one in five health conversations high-risk. The rate was 21.5% in India, 20.6% in Nigeria, 22.3% in Brazil and 26.3% in Pakistan. Questions about symptoms, medicines and ongoing conditions were much more likely to be flagged than general health information.
That is consequential evidence because it examines what people actually asked rather than clinician-written vignettes. It is not proof that one in five ChatGPT health exchanges in any country caused harm. The users were a self-selected crowdworker sample, the safety judge was another language model, and the researchers did not observe whether anyone followed the advice or experienced an adverse outcome.
For people seeking care, the practical message is that fluent advice should not substitute for diagnosis, prescribing or urgent triage. For developers and regulators, the study shows why a response that looks acceptable in isolation may become unsafe when the system already knows that a user has a chronic condition, takes medication or is in mental distress.[1]
The dataset linked conversations to the people who donated them
Researchers recruited 1,252 active ChatGPT users through Clickworker in India, Nigeria, Brazil and Pakistan between December 2025 and February 2026. Participants exported complete histories covering December 2022 to February 2026 and reported age and gender. A client-side process removed names, email addresses and other identifying information before upload.
The full corpus contained 202,590 conversations. Topic modelling identified candidate health clusters, after which 15,555 conversations from 754 users were treated as health-related. India contributed 6,343 health conversations, Brazil 4,657, Nigeria 2,976 and Pakistan 1,579. Median health conversations per contributing user ranged from four in India to seven in Brazil.
The denominator matters. A conversation is not a person, and users contributed different numbers of sessions. India supplied 44.5% of participants, while the age and gender mix differed sharply by country. The study therefore foregrounded effect sizes and cross-country consistency rather than treating every conversation as an independent population survey.[1]
Classification and safety scoring were largely automated
A GPT-4o-mini classifier assigned messages to a twelve-part health-intent taxonomy using the preceding five user-assistant turns. A blinded author check on 100 sampled messages found 82% agreement and Cohen's kappa of 0.78. That supports the intent labels, but one hundred checks are small beside 15,555 conversations and most disagreements occurred between adjacent categories.
Gemini 3.1 Pro Preview scored the last assistant message on likelihood of harm, potential severity and adequacy of safeguards using a published seven-point rubric. The conversation inherited its lowest message score, and the researchers defined three or below as high-risk. One author read 400 sampled conversations and could assign a concrete failure mode to all 128 the judge had flagged.
This process tests potential danger, not clinical correctness against a gold-standard diagnosis. The human audit supports the precision of flagged cases but does not estimate risky answers that the judge missed. No clinician panel reviewed a representative sample, and no medical record or outcome showed whether advice was accurate for a particular patient.[1]
Risk concentrated where users sought personal clinical guidance
General health information made up 28.3% to 48.9% of health conversations by country and was rated high-risk in 7.9% to 11.9% of cases. Condition-and-care questions were flagged in 39.8% to 47.7%, symptom questions in 33.4% to 40.3%, and emotional-wellbeing conversations in 22.7% to 34.3%.
At user level, 51.6% of Nigerian contributors with health conversations and 58.3% to 61.7% in the other countries received at least one flagged response. This is different from the per-conversation rate: frequent users have more chances to encounter one low-scoring exchange, so it should not be read as the share of all citizens exposed to dangerous advice.
Among 128 flagged conversations, the most common coded failures were incorrect or harmful advice in 61 and missing professional referral in 56. Unsafe remedies, unsafe diagnosis and reinforcement of harmful behaviour or dependence each appeared in 32. Codes overlapped, so the counts do not add to one prevalence total.[1]
Earlier user context changed the safety assessment
For conversations preceded by health-relevant history, the researchers built a running profile from durable facts disclosed in earlier chats: chronic conditions, medication use, lifestyle, mental state and family circumstances. GPT-5-mini extracted the profile, and the same Gemini judge rescored the unchanged response against that context.
Adding context raised the high-risk share by 13 to 17 percentage points, and about one quarter of conversations rated safe in isolation became unsafe for the person asking. Fitness advice changed especially sharply when prior eating disorders, injuries or vulnerabilities were visible. Emotional-wellbeing advice also became riskier when earlier distress was considered.
Only 19% to 60% of health conversations by country had enough preceding profile information, and that subset was already riskier in context-blind scores. The rate cannot be extrapolated to every health conversation. It nevertheless exposes a design problem: a system may possess facts needed to avoid harm without using them in a safety check.[1]
Newer model versions were not shown to be safer
Among 14,420 conversations served by one identifiable model, high-risk ratings rose from GPT-3.5 through GPT-5.1 in each country. In 240 users who asked health questions of both GPT-5 and GPT-5.1, 19.0% of GPT-5 conversations and 27.8% of GPT-5.1 conversations were flagged.
The authors appropriately stop short of saying later models were intrinsically less safe. Model version is entangled with time, changing user behaviour and selection into mini variants. Correlations were statistically significant but weak, and GPT-5.2 estimates rested on only 30 to 282 conversations per country.
The defensible conclusion is narrower: observed risk did not reliably decline as versions changed. Release recency should not be treated as evidence of medical safety. Providers need version-specific monitoring on real use, with independently defined clinical thresholds and transparent changes to safeguards.[1]
Limits and what would change the assessment
Participants were paid crowdworkers willing to donate complete histories. They were active, digitally literate users, not representative national samples. India and Pakistan were young and male-skewed, older groups were small, and the analysis did not cover clinical severity, income, rurality, language proficiency or access to care.
Confidence would rise with a preregistered clinician audit of a stratified sample that estimates both false alarms and missed hazards. Sampling should include rural, older and lower-literacy users, local languages and people with limited access to care, while preserving consent and data minimisation.
The decisive next evidence is behavioural and clinical: whether users acted on advice, whether referral or triage was delayed and whether outcomes changed. Until then, this is an early post-deployment safety signal showing that authentic use and earlier context can reveal risks hidden by synthetic benchmarks, not a clinical error rate or proof of realised harm.[1]
What this means for people
- People should treat chatbot health guidance as general information and seek qualified care for symptoms, medicines, diagnosis, mental-health crises or urgent decisions.
- Clinicians and health systems gain evidence that monitoring should include conversational context, not isolated answers alone.
- Users with limited access to professional care may face the greatest consequences from missing referrals, but the sample cannot estimate population harm.
Global context
The study contributes rare evidence from India, Nigeria, Brazil and Pakistan rather than extrapolating from high-income-country vignettes. Its cross-country consistency is useful, but the crowdworker sample and unequal demographics limit national and regional claims.
What the evidence does not yet show
- This is an unreviewed preprint using a self-selected convenience sample, not a representative survey.
- Safety labels came from an LLM judge; clinicians did not establish sensitivity or correctness across the corpus.
- The study measured potential risk, not whether advice was followed or caused an outcome.
- Context-aware results apply only to the subset with earlier profile information.
- Model-version comparisons are observational and confounded by time, user choice and query composition.
What to watch next
- Independent clinician adjudication of representative conversation samples
- Prospective tests of context-aware safeguards with privacy and equity controls
- Measures linking advice to referral, treatment and outcome decisions
- Replication in local languages and among less digitally connected populations
Living evidence record
Impact record IAI-16MEKUO
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
14 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 14 October 2026
Research question, topics and regions
The safety of real health conversations with ChatGPT when responses are assessed alone and against health-relevant context previously disclosed by the same user
Topics: ChatGPT · Health advice · Patient safety · Global health · Post-deployment monitoring · Health equity
Regions: India · Nigeria · Brazil · Pakistan · Global South
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
The Royal Society
AI-generated illustration · not a video frame
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
Healthcare, culture & human impact · Background explainer
Published 2026-07-01 · 44 min 28 sec
Is this the AI future we want? | The Royal Society
A Royal Society discussion with Madhumita Murgia and biomedical engineer Alison Noble on AI’s potential in healthcare, science and culture, recorded at the 30 June 2026 event.
Evidence check: A discussion of opportunities and risks, including future possibilities. It is not evidence that every application described is available or clinically proven.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Health & Life Sciences
Can AI predict five-year colorectal cancer survival from pathology slides?
A retrospective preprint reports moderate discrimination across a US trial cohort and an external NHS dataset. The model is not ready to guide treatment, and its selected ensemble did not clearly beat the strongest comparator.
8 min · 1 source
Health & Life Sciences
Can AI find children likely to miss a measles vaccine?
Models trained on 13,196 vaccine-eligible children in Uganda and Kenya identified many recorded defaulters, but the advanced ensembles barely beat simpler tuned models, thresholds were chosen on the test set, and the study did not test live records, outreach or vaccination uptake.
7 min · 1 source
Health & Life Sciences
Can social data help hospitals predict next-hour emergency demand?
At one Wuhan emergency department, adding neighbourhood social indicators to an XGBoost model reduced overall error by 2.27% against a calendar-only baseline and produced a mean absolute error of 2.62 arrivals an hour. The peer-reviewed retrospective study has not shown that forecasts improve staffing or patient outcomes.
8 min · 2 sources
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.