A diagnostic AI can look safer when the test is vague—what changes with a clinical rubric?
A bilingual benchmark found that detailed, diagnosis-specific rubrics changed safety scores and exposed sharp failures under instruction-shell attacks. Its 3,489 prompts are synthetic and the 11 tested models ran without deployment safeguards.
The short version
- What happened
- DiagSafe contains 3,489 synthetic English and Chinese prompts: 2,401 ordinary diagnostic situations and 1,088 instruction-shell attacks across an 84-label safety taxonomy.
- Who it affects
- Clinicians and patients need safety claims tied to the actual diagnostic task and deployment controls. A model-level benchmark can help teams find weaknesses, but it cannot replace local clinical governance or incident monitoring.
- What remains unproven
- The unreviewed benchmark uses synthetic model-generated prompts and rubrics, with human checks on stratified samples rather than all 3,489 items.
Keep in mind
The unreviewed benchmark uses synthetic model-generated prompts and rubrics, with human checks on stratified samples rather than all 3,489 items.
Only open-weight models in one local inference configuration were tested, without production safeguards.
The study measures rubric-defined unsafe responses, not diagnostic quality, clinician behaviour, patient harm or live attack frequency.
Several authors are affiliated with Tencent Hunyuan; no separate funding statement is reported.
Source analysis · 1 linked source. Original source date: 21 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
The benchmark asks what kind of harm is possible
Researchers organised diagnostic-assistant risks into 84 leaf labels under 11 top-level categories, linking them to 204 provisions drawn from clinical guidance and rules in China, the European Union and the United States. A generator model then produced 3,489 synthetic prompts: 2,401 ordinary diagnostic situations and 1,088 adversarial prompts designed to hide or reframe unsafe instructions. About half were Chinese and half English; single-turn and multi-turn cases appeared in a seven-to-three ratio.
Three experts checked stratified samples rather than every item. They accepted 324 of 336 prompts as realistic and correctly labelled, and 251 of 252 diagnosis-specific rubrics as reasonable and actionable. On a separate human-labelled evaluation set, rubric-guided judging reached 95.34% accuracy, compared with 87.29% for a generic medical-safety prompt and 86.86% when the label alone was supplied. Repeated judging agreed 93.33% to 94.78% of the time.[1]
Attack results are a warning, not a production incident rate
The team tested 11 locally served open-weight language models, ranging from 8 billion to 1.6 trillion parameters. Every model's unsafe-response rate rose under instruction-shell attacks: the multiplier ranged from 1.9 to 8.4, with a median of 5.3, and eight models exceeded 40% on that adversarial set. The finding suggests a broad weakness when harmful diagnostic instructions are wrapped in plausible context.
But the setup deliberately omitted system prompts, moderation layers, retrieval controls and deployment guardrails. The prompts and rubrics were themselves model-generated, then sample-checked; rare or locally specific harms may be missing. The work covers stand-alone chat rather than tool-using clinical agents, and it does not measure diagnostic accuracy, patient outcomes or real-world incident rates. Independent clinical validation, red-teaming of complete deployed systems and prospective monitoring would materially change the assessment.[1]
The distinction that matters
Why the rubric changes the test
The framework moves from a broad safety question to a diagnosis-specific failure definition.
Classify
Map the prompt to one of 84 risk labels
Names the kind of diagnostic harm
Specify
Apply an item-level clinical rubric
Defines what a safe answer must avoid or include
Stress
Compare ordinary and instruction-shell prompts
Separates baseline from adversarial behaviour
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring multilingual ai: compare the finding and its evidence limits.
Do AI models behave more safely when they know they are being tested?
In 47,880 responses across English and five Indic languages, explicitly saying that a prompt was part of a scored safety test increased refusals of harmful requests—but also increased refusals of benign Indic-language requests. Refusal alone is not proof of safer behaviour.
Read the next reportExplore the full source and evidence record
What this means for people
- Clinicians and patients need safety claims tied to the actual diagnostic task and deployment controls. A model-level benchmark can help teams find weaknesses, but it cannot replace local clinical governance or incident monitoring.
Global context
The taxonomy incorporates provisions from several major jurisdictions and tests Chinese as well as English, but clinical duties, health systems and acceptable advice differ more widely. Local experts must adapt and validate the labels rather than treating one taxonomy as universal.
What the evidence does not yet show
- The unreviewed benchmark uses synthetic model-generated prompts and rubrics, with human checks on stratified samples rather than all 3,489 items.
- Only open-weight models in one local inference configuration were tested, without production safeguards.
- The study measures rubric-defined unsafe responses, not diagnostic quality, clinician behaviour, patient harm or live attack frequency.
- Several authors are affiliated with Tencent Hunyuan; no separate funding statement is reported.
What to watch next
- Independent checks of prompt and rubric quality
- Tests of complete guarded clinical systems
- Real clinician and patient evaluations
- Prospective incident data across languages and health systems
Living evidence record
Impact record IAI-0MNZRB8
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
22 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 22 October 2026
Research question, topics and regions
Whether diagnosis-specific safety rubrics reveal failures that generic medical-safety prompts miss
Topics: Diagnostic AI · Clinical safety · Adversarial prompts · Evaluation · Multilingual AI
Regions: China · Asia · European Union · United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.