Can a medical AI give the same answer twice? 1.2 million generations separate consistency from accuracy
A peer-reviewed study repeated four language models 100 times across 608 diagnostic cases and five prompts. Its proposed scores tracked human judgements of meaning, but consistency was not the same as a correct diagnosis—and no patient outcome was tested.
The short version
- What happened
- Researchers ran ChatGPT-4, ChatGPT-4o-mini, LLaMA 3.2-1B and Qwen3.6-35B-A3B on 518 medical-exam questions and 90 rare-disease cases, using five prompts and 100 runs per combination: 1,216,000 generated responses, not 1,216,000 patients.
- Who it affects
- Clinicians need to know whether the same input can produce materially different advice, but repetition does not remove the need to verify clinical correctness.
- What remains unproven
- Peer-reviewed methods paper with an illustrative diagnostic-reasoning experiment, not a clinical trial or deployment study.
Keep in mind
Peer-reviewed methods paper with an illustrative diagnostic-reasoning experiment, not a clinical trial or deployment study.
Only 608 underlying cases, two datasets, four models, five prompts and selected generation settings were evaluated despite the large generation count.
The prior accuracy labels came from a single ChatGPT-4 generation, so the accuracy comparison was exploratory and restricted to one model–dataset combination.
Semantic similarity does not automatically identify clinically important differences, and internal metrics require token probabilities that many interfaces do not expose.
Embedding choice affected the scores; the strongest generation-setting sensitivity test used one model on the rare-disease dataset.
No clinician behaviour, workflow burden, diagnostic outcome, treatment, harm or patient benefit was measured.
Source analysis · 1 linked source. Primary analysed source date: 5 November 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The answer: repeating the test reveals a different property from getting one answer right
A peer-reviewed study in the Journal of the American Medical Informatics Association proposes four scores for a basic but often neglected question: if a language model is asked the same medical question again, or the prompt is changed, does it keep saying the same thing? The researchers distinguish repeatability—agreement when the model, prompt and generation settings stay the same—from reproducibility—agreement across planned changes such as different prompts. They also separate the meaning of the answer from the model's token-level certainty. That produces semantic and internal versions of both scores.
The framework matters because one correct-looking response cannot show whether a model will behave consistently when a clinician, researcher or patient tries again. Yet consistency is not a substitute for accuracy. In the paper's exploratory comparison, repeatability was generally not associated with a previously published single-run correct-or-incorrect label for ChatGPT-4 on the medical-exam cases. Reproducibility was associated with those labels, but only in that restricted analysis. A model can repeat an error with great confidence, and a variable model can occasionally be right. The study therefore supports measuring both dimensions; it does not validate any model for diagnosis.[1]
What was generated—and why the 1.2 million denominator needs care
The team evaluated four models: the commercial ChatGPT-4 and ChatGPT-4o-mini systems, plus the open LLaMA 3.2-1B and Qwen3.6-35B-A3B models. They used 518 free-response questions derived from US medical licensing examinations and 90 real-world rare-disease cases from the US National Institutes of Health Undiagnosed Diseases Network. Five chain-of-thought prompts asked for traditional, differential-diagnosis, intuitive, analytic or Bayesian reasoning. Each model received every case with every prompt 100 times. Multiplying 608 cases by five prompts, four models and 100 runs gives 1,216,000 generations.
That large number measures repeated model output, not clinical breadth. It is still only 608 underlying cases, two datasets and one family of diagnostic-reasoning tasks. The exam questions are standardised and reformulated from multiple choice; the rare-disease records are longer, incomplete and clinically heterogeneous. The primary runs used temperature 0.5 and top-k 30. Commercial models were accessed through Microsoft Azure OpenAI, while the open models ran on a secure Vanderbilt server. Rare-disease evaluations stayed inside institutionally approved secure environments. No new patient was diagnosed and no care plan changed because of the experiment.[1]
The distinction that matters
Four measurements, two different questions
The scores describe consistency. None independently establishes that a medical answer is correct or safe.
Same conditions
Semantic repeatability
Does the answer keep the same overall meaning?
Same conditions
Internal repeatability
Does token-level certainty remain stable?
Changed prompts
Semantic reproducibility
Does meaning persist across reasoning styles?
Changed prompts
Internal reproducibility
Does certainty persist across conditions?
How the four scores work in plain language
Semantic repeatability asks whether answers from unchanged runs occupy a similar meaning space. Semantic reproducibility asks the same question across the five prompt styles. The authors used a medical embedding model to represent meaning and then calculated agreement. Internal repeatability instead examines whether token-generation certainty remains stable across identical runs; internal reproducibility examines whether that certainty pattern remains stable when the prompt changes. The internal measures require token probabilities, which many closed interfaces do not expose. A high internal score can also accompany consistently low certainty, so it must be read alongside the mean certainty level.
The distinction prevents an attractive but misleading summary score. Two answers can use different wording while making the same diagnosis; semantic comparison should not mark every wording change as failure. Conversely, similar prose can conceal a clinically important difference. The authors explicitly acknowledge that their semantic and internal scores do not prioritise clinical consequence. An altered drug dose and a changed adjective should not carry equal weight in a safety review, even if an embedding treats their overall meanings as close. Clinical validation therefore has to add consequence-aware review, not merely adopt the proposed number.[1]
Human validation supports the metric—but within a bounded sample
To check whether the semantic scores matched human judgement, the researchers sampled 60 MedQA question–model–prompt units for repeatability and drew five response pairs from each, producing 300 pairs. For reproducibility they sampled 60 question–model units and compared all ten unique pairs across five prompts, producing 600 pairs. Two reviewers rated each pair on a five-point scale from fundamentally different meaning to the same overall meaning. A third reviewer assessed pairs where the first two differed by at least two points.
The semantic repeatability score correlated with the aggregated human ratings at Spearman 0.88, with a 95% bootstrap confidence interval from 0.79 to 0.93. Semantic reproducibility correlated at 0.66, interval 0.44 to 0.81; both p-values were below .001. Quadratic-weighted inter-reviewer agreement was 0.69 and 0.93 respectively. Those results support the intended interpretation of the metric for overall similarity in these sampled exam responses. They do not show that the reviewers judged diagnoses correct, that every medically important difference was captured, or that the correlations will transfer to discharge summaries, patient messages or live consultations.[1]
Model, prompt, dataset and generation settings all affected the pattern
The joint distribution of semantic and internal repeatability differed across prompts within every model–dataset combination. Bayesian prompting was among the more semantically repeatable strategies for both ChatGPT models, while ChatGPT-4o-mini tended to have lower internal repeatability under the differential-diagnosis prompt. On the exam questions, Qwen showed higher semantic and internal reproducibility than the other models. For the rare-disease cases, different models led on the semantic and internal dimensions. The plotted differences occupied the high end of the zero-to-one scale, and the authors caution that visually separated groups can represent numerically small differences.
Sensitivity tests reinforce the need to report conditions. Switching from the medical embedding model to a general embedding produced correlated but not identical semantic scores: Spearman 0.897 for repeatability and 0.680 for reproducibility. For Qwen on rare-disease cases, internal repeatability fell as temperature increased and rose with top-k within each temperature setting. Bayesian prompting remained among the more repeatable options, but these tests covered only one model and dataset for generation settings. A procurement or research comparison that omits model version, prompt, decoding settings, dataset and embedding choice cannot make its consistency result fully interpretable.[1]
What clinicians, researchers and patients can reasonably take from it
For a clinical team considering language-model assistance, the immediate lesson is procedural. Repeat the same representative cases, vary the prompts or users that will occur in practice, and inspect both answer meaning and consequence-relevant errors. Define in advance which differences require escalation. A stable wrong answer should fail an accuracy test; an accurate but unstable system may still create inconsistent work, uncertain review loads or unequal experiences. The published Python package can support research evaluation, but a local assurance case still needs task-specific references, human review and a plan for changes in provider models.
Patients should not read the study as evidence that a chatbot can diagnose a rare disease. The 90 UDN cases were used as evaluation inputs, not as a prospective clinical service, and sensitive data remain access-controlled. The paper reports no clinician workflow, waiting time, treatment, missed diagnosis, adverse event or benefit. Confidence would rise with preregistered prospective evaluation across more clinical tasks, pinned model versions, consequence-weighted review and observation of real users. Most important would be evidence that measuring and managing inconsistency improves decisions or patient outcomes without creating an unmanageable review burden.[1]
Funding, access and the evidence boundary
The work was supported by the US National Institutes of Health Common Fund, the National Human Genome Research Institute, the National Institute of Neurological Disorders and Stroke, the National Library of Medicine and the Potocsnak Center for Undiagnosed and Rare Disorders. The authors declared no conflicts of interest. MedQA cases are public; deidentified UDN phenotypic and genomic data are available through controlled dbGaP access rather than open release, reflecting their sensitivity. The framework's code is published as the llmReproScore package.
The strongest defensible conclusion is narrow: the study offers validated statistical tools for describing output consistency, and a large illustrative experiment shows that consistency depends on the model, prompt, dataset and generation conditions. It does not establish a universal cutoff, prove that one model is clinically superior or show that repeatability predicts safe care. A future assessment should change only when external teams reproduce the scores across tasks and link them to consequential differences, reviewer performance and patient outcomes. Until then, repeatability is one missing axis of evaluation—not a clinical safety certificate.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
The same cancer case changed medical AI staging accuracy by over 50% when its format changed
Across 3,035 physician-validated perturbations, three medical language models gave markedly different clinical answers when equivalent cases were reformatted. The peer-reviewed study supports model-specific safety testing, not a claim about patient outcomes.
Read the next reportExplore the full source and evidence record
What this means for people
- Clinicians need to know whether the same input can produce materially different advice, but repetition does not remove the need to verify clinical correctness.
- Patients may otherwise encounter inconsistent explanations or false reassurance; this experiment did not test live care or justify self-diagnosis.
- Researchers using model output as a study endpoint need to report repeated runs and changed conditions so model variability is not mistaken for a stable measurement.
Global context
The cases and institutions were US-based, while the four models originated from US and Chinese providers and open research communities. Health systems elsewhere have different languages, disease prevalence, clinical pathways and access to token probabilities. The transferable lesson is to test consistency locally under the actual prompts, users and settings—not to import the model rankings or a threshold from this experiment.
What the evidence does not yet show
- Peer-reviewed methods paper with an illustrative diagnostic-reasoning experiment, not a clinical trial or deployment study.
- Only 608 underlying cases, two datasets, four models, five prompts and selected generation settings were evaluated despite the large generation count.
- The prior accuracy labels came from a single ChatGPT-4 generation, so the accuracy comparison was exploratory and restricted to one model–dataset combination.
- Semantic similarity does not automatically identify clinically important differences, and internal metrics require token probabilities that many interfaces do not expose.
- Embedding choice affected the scores; the strongest generation-setting sensitivity test used one model on the rare-disease dataset.
- No clinician behaviour, workflow burden, diagnostic outcome, treatment, harm or patient benefit was measured.
What to watch next
- Independent replication across patient messages, documentation, triage and multilingual clinical tasks.
- Consequence-weighted metrics that distinguish stylistic variation from a clinically dangerous change.
- Pinned-version prospective tests with real clinicians and measured review burden.
- Evidence that managing inconsistency improves decisions or patient outcomes.
- Transparent provider access to the probabilities and version details required for reproducibility.
Living evidence record
Impact record IAI-0JZZX5Q
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Journal of the American Medical Informatics Association published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
How to measure whether language-model answers remain semantically and internally consistent across repeated runs and changed prompts
Topics: Clinical AI · Diagnostic reasoning · Repeatability · Reproducibility · Model evaluation · Patient safety
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.