Can LLM judges replace clinical experts? A German audit changed 57.2% of pointwise ratings
Six open-weight models evaluated four German clinical-text datasets under three judging formats. Against fixed expert references, automated substitution changed most pointwise categories, swapped 41.7% of pairwise winners and missed 10.2% of major or critical errors in patient letters.
The short version
- What happened
- The audit covered six judges, four prompt configurations and three paradigms using fixed expert reference data from four German hospital datasets.
- Who it affects
- Clinicians and patients can be affected indirectly when flawed automated evaluation selects a poorer documentation or communication system.
- What remains unproven
- All four datasets were German and came from one academic-hospital setting.
Keep in mind
All four datasets were German and came from one academic-hospital setting.
The audit tested six open-weight models, not every closed or future frontier model.
Protected clinical data cannot be released publicly, limiting full external reproduction.
Human reference panels were finite and disagreed on some items; consensus is not absolute truth.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 8 min The evidence, methods, implications and limitations
The direct finding: an automated judge could change which clinical AI system appears better
A peer-reviewed audit in npj Digital Medicine reports that none of six open-weight language models, under any of four prompting configurations, was a reliable substitute for expert clinicians across three common evaluation formats. In the paper's decision-impact analysis, replacing the human panel with a selected LLM judge shifted 57.2% of pointwise categories, changed the winner in 41.7% of paired comparisons and missed 10.2% of major or critical clinical errors in patient-letter segments. These are not small differences in a cosmetic score: they can change which generator a research team, benchmark or hospital governance group concludes is preferable.
The authors also found position bias—answers changed when the left and right candidates were reversed—and identity bias related to whether a text was human- or model-generated. Automated judges tended to overrate criteria described as succinct and accurate, inverting the expert panel's clinical priorities in the case study. The paper's central conclusion is appropriately narrow: for these German clinical-text evaluations, the tested LLM judges were not safe replacements for experts. It does not show that language models have no role in evaluation; it shows that their errors must be measured at the decision level before their scores are allowed to select a clinical system.[1]
Four datasets test three different kinds of judgement
The audit assembled four datasets from a large German academic hospital. The clinical-report set contained summaries of 35 radiology, 35 pathology and 35 surgery reports generated by Qwen3-235B-A22B-Instruct. Four clinicians rated radiology and pathology summaries and three rated surgery summaries against seven adapted PDSQI-9 quality criteria, producing 2,695 ratings. The nursing set contained 47 full-documentation summaries and 50 clinical-endpoint summaries, each scored by at least four raters on ten criteria, for 3,990 ratings. Those two datasets represent pointwise evaluation: one text receives an ordinal or binary score for each criterion.
The discharge-summary set tested pairwise evaluation. Four clinicians compared 35 human-written and AI-generated summary pairs on five criteria, yielding 700 ratings and allowing left, equal or right responses. The molecular-tumour-board patient-letter set tested error detection. It contained 710 segments generated by Llama-3.3-70B-Instruct; two oncologists independently identified clinical errors and graded their severity. A segment counted as erroneous if either expert found a major or critical error, reflecting the higher cost of a miss. Together these datasets cover 8,095 pointwise or pairwise expert ratings plus 710 safety-screening segments, but all remain retrospective German hospital material rather than a live deployment.[1]
The distinction that matters
Three judging formats—and the decision each can distort
The study tested different output formats because each automated failure changes a different downstream decision.
Pointwise
Score one text against a criterion
57.2% of categories changed in the case study
Pairwise
Choose between two candidate texts
41.7% of winners changed
Pass or fail
Detect major or critical clinical errors
10.2% of errors were missed
Bias check
Reverse order and compare identities
All judges showed position and identity bias
Governance
Compare the resulting decision with experts
Validate impact before substitution
Six judges and four prompts produced no universal configuration
The candidate judges were Qwen3-235B-A22B Instruct and Thinking variants, gpt-oss-120b, Qwen3-Next-80B-A3B-Instruct, Magistral-Small-2509 and medgemma-27b-it. They span three model families and several size tiers, including a medically pretrained model. The authors deliberately limited the audit to open-weight systems that could be hosted inside European hospitals without sending protected data to an external commercial service. That makes the result particularly relevant to on-premise procurement, while also limiting generalisation to closed frontier models that were not tested.
Each judge used four configurations: the original human rubric as a baseline, a rule-augmented prompt using the annotator codebook, a demonstration-based prompt with example decisions, and a multiagent majority vote across three models. Every automated judgment ran three times under recommended model settings, with constrained decoding on locally hosted A100 and H100 GPUs. The experiment consumed 61.1 GPU-hours. No prompt or ensemble worked reliably across all three paradigms. A technique that improved agreement for one criterion or dataset could fail on another, undermining the idea that a single prompt recipe can certify an automated clinical evaluator.[1]
Agreement averages can conceal the errors that matter
Many judge studies report a pooled agreement score. This paper instead examined absolute alignment, inter-rater reliability, distributional robustness and algorithmic robustness at item and criterion level. It used Gwet's coefficients, which are less distorted than Cohen's kappa when a label is rare, and stratified distribution comparisons by how strongly the experts agreed. Primary estimates received 95% confidence intervals from 1,000 item-level bootstrap resamples. For the pass-or-fail safety task, recall took priority because missing a major clinical error was treated as more consequential than raising a redundant flag.
That framework matters because a respectable average can hide systematic failure. If a judge rewards polished compression but undervalues omission of a clinically necessary detail, it may rank a concise summary above a safer one. If it prefers model-generated prose or the candidate shown first, a head-to-head benchmark can report the wrong winner. Repeated runs also reveal instability that a single deterministic-looking score hides. The decision-impact analysis translates those statistical properties into the question governance teams actually face: would replacing experts alter the category, the winning system or the number of dangerous errors detected? Here, the answer was often yes.[1]
The strongest caution is also a boundary on the claim
The study is unusually broad across judging paradigms, but it is not universal. All text was German, all clinical material came from one academic-hospital setting, and the generators were Qwen3 or Llama-3.3 models. The six judges were open-weight systems chosen partly for European deployment feasibility. A newer closed model, an English dataset or a narrowly defined criterion could perform differently. Some underlying annotations came from primary studies not yet fully published, although the rubrics are provided in the supplement. Protected clinical data prevent public release of the full datasets, which limits independent reproduction even though prompts, code and aggregate results can be inspected.
Experts are not an infallible gold standard either. The paper preserves their independent raw ratings and reports consensus rather than pretending disagreement does not exist, but panel size varies by dataset and clinical judgement remains contextual. The patient-letter reference uses the union of two oncologists' major or critical flags, intentionally favouring sensitivity. A different governance decision could choose another threshold. Those limitations do not erase the substitution failures; they define where the numbers apply. The actionable lesson is to validate an automated judge against qualified humans on the exact language, task, criterion and downstream decision before adopting it.[1]
What changes for AI evaluation—and what evidence would change this assessment
For clinical-AI teams, an LLM judge may still help triage large volumes, draft explanations or identify outputs for human review. It should not silently become the sole scorer that selects a model, approves a release or screens patient-facing text. Reports should disclose the judge model and version, prompt, order randomisation, repeated-run variance, expert calibration sample and the decision consequences of disagreement. Safety-critical criteria need human oversight and explicit recall targets. Procurement teams should ask whether a benchmark's ranking survives expert review, not merely whether an automated score is cheap and reproducible.
Confidence in substitution would rise if independent, multilingual, multi-centre studies showed stable criterion-level agreement, resistance to position and identity bias, high recall for serious errors and unchanged model-selection decisions across prospective datasets. It would fall further if deployed automated judges repeatedly reward style over clinical completeness or allow harmful errors to pass. The authors disclose outside consulting and institutional funding for two contributors but no competing interest for the remaining authors, and they report supervised use of generative AI in research and writing. This is consequential research, not Breaking: it audits an evaluation method rather than reporting an urgent public-health event.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
The same cancer case changed medical AI staging accuracy by over 50% when its format changed
Across 3,035 physician-validated perturbations, three medical language models gave markedly different clinical answers when equivalent cases were reformatted. The peer-reviewed study supports model-specific safety testing, not a claim about patient outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- Clinicians and patients can be affected indirectly when flawed automated evaluation selects a poorer documentation or communication system.
- Researchers may use model judges for triage, but safety-critical conclusions still require qualified human review.
- Hospital governance teams need language-, task- and criterion-specific validation rather than a generic benchmark score.
Global context
Automated model evaluation is spreading faster than expert review because it is cheap and scalable. In health care, that convenience can create a circular benchmark: one model generates text, another scores it, and a model ranking acquires authority without a clinician checking whether the preferred output is safer. This German audit provides a methods-backed warning that evaluation validity is local to the task and decision. The same principle applies beyond medicine wherever LLM judges influence procurement, publication or safety gates.
What the evidence does not yet show
- All four datasets were German and came from one academic-hospital setting.
- The audit tested six open-weight models, not every closed or future frontier model.
- Protected clinical data cannot be released publicly, limiting full external reproduction.
- Human reference panels were finite and disagreed on some items; consensus is not absolute truth.
What to watch next
- Independent multilingual replications.
- Audits of closed frontier judges on protected local infrastructure.
- Published decision-impact thresholds for clinical governance.
- Prospective evidence that human-plus-AI review improves speed without losing serious errors.
Living evidence record
Impact record IAI-108ONF8
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what npj Digital Medicine published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Whether open-weight language models can substitute for expert clinicians when scoring German clinical summaries, comparing discharge summaries and detecting errors in patient letters
Topics: LLM evaluation · Clinical AI · Model bias · Patient safety · German language · Open-weight models
Regions: Germany · Europe · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.