Why did a thyroid AI benchmark look almost perfect?
A peer-reviewed audit found that the benchmark's documentation misdescribed its three classes and placed 89 diagnosed hyperthyroid records inside the majority class. A 92.6% accuracy baseline could simply predict that majority class.
The short version
- What happened
- The researcher matched 3,597 benchmark records to diagnosis codes in the same UCI archive with no discordance, finding three hypothyroid-spectrum strata rather than the documented hyperfunction class.
- Who it affects
- Clinicians and patients should not read a near-perfect benchmark score as evidence that a model is ready to diagnose thyroid disease; the study did not test prospective care, external hospitals or patient outcomes.
- What remains unproven
- This is a controlled audit of one public benchmark, not a validation study in a clinic or a representative sample of current patients.
The label audit changed what the benchmark was measuring
The ann-thyroid dataset is widely used to compare machine-learning classifiers, often with near-perfect-looking scores. Osman Eray at Akdeniz University audited the public benchmark before asking which algorithm won. The UCI files do not name their three classes, while the documentation describes a hyperfunction class. Matching 3,597 records to diagnosis codes distributed in the same archive produced one-to-one agreement and instead identified overt hypothyroid, subclinical hypothyroid and euthyroid groups. The audit also located 89 diagnosed hyperthyroid records inside the majority class.[1]
Who it affects
Clinicians and patients should not read a near-perfect benchmark score as evidence that a model is ready to diagnose thyroid disease; the study did not test prospective care, external hospitals or patient outcomes.[1]
Keep in mind
This is a controlled audit of one public benchmark, not a validation study in a clinic or a representative sample of current patients.
The diagnosis-code match clarifies the archive's class structure, but it does not establish that the historic records or hormone cutoffs meet today's clinical standards.
The journal is sharing an accepted peer-reviewed version before final copy-editing; the article may receive editorial changes when the Version of Record replaces it.
The study has one author, reports no external funding and declares no competing interests; independent replication would still strengthen the conclusion.
Source analysis · 1 linked source. Original source date: 20 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The label audit changed what the benchmark was measuring
The ann-thyroid dataset is widely used to compare machine-learning classifiers, often with near-perfect-looking scores. Osman Eray at Akdeniz University audited the public benchmark before asking which algorithm won. The UCI files do not name their three classes, while the documentation describes a hyperfunction class. Matching 3,597 records to diagnosis codes distributed in the same archive produced one-to-one agreement and instead identified overt hypothyroid, subclinical hypothyroid and euthyroid groups. The audit also located 89 diagnosed hyperthyroid records inside the majority class.
That imbalance makes overall accuracy a poor guide. Always predicting the majority class already scores 92.6% accuracy, while its macro-F1 is just 0.320 because the rarer classes are largely missed. The finding does not show that thyroid diagnosis is easy or that the labels are clinically definitive. It shows that a benchmark can reward a plausible-looking score even when its class meaning and minority performance have not been checked.[1]
Algorithmic refinements added less than the reporting suggested
The peer-reviewed study tested ten classifiers across ten stratified splits in a leakage-free pipeline. A factorial ablation separated feature selection from class balancing; nine selection methods and seven balancing strategies were compared with statistical tests and equivalence margins set in advance. Class balancing raised macro-specificity from 0.905 to 0.987, but feature-selection methods occupied a ROC-AUC range of only 0.0018 and balancing strategies a macro-F1 range of 0.0098. The detected orderings were reproducible, yet every pairwise difference remained inside the prespecified equivalence margin.
Metric choice also changed the story. In five of ten classifiers, the pipeline significantly reduced precision-recall area for minority classes while ROC-AUC moved by about 0.002. A transparent rule using two cutoffs on thyroid-stimulating hormone and total thyroxine reached 0.885 macro-F1 in 0.2 seconds, comparable with tuned logistic regression. For researchers and procurement teams, the practical lesson is to audit labels, report per-class metrics and compare against simple clinical rules before crediting a more elaborate model.[1]
The distinction that matters
How a high score can survive a weak benchmark
Three checks separate genuine model value from performance inherited from the data.
Labels
Match records to diagnosis codes
The documented class meaning changed
Baseline
Always predict the 92.6% majority
Accuracy looked strong while macro-F1 fell to 0.320
Comparator
Fit two hormone cutoffs
A simple rule rivalled tuned logistic regression
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Evidence check
This is a controlled audit of one public benchmark, not a validation study in a clinic or a representative sample of current patients.
The diagnosis-code match clarifies the archive's class structure, but it does not establish that the historic records or hormone cutoffs meet today's clinical standards.
The journal is sharing an accepted peer-reviewed version before final copy-editing; the article may receive editorial changes when the Version of Record replaces it.
The study has one author, reports no external funding and declares no competing interests; independent replication would still strengthen the conclusion.
Reporting basis: Source analysis. Original source date: 20 October 2026.
Continue the story
Read next
Continue exploring clinical ai: compare the finding and its evidence limits.
An Alzheimer’s MRI model scored 94.5%. On new data, it fell to 61.6%
For a patient, the important test is whether a model works beyond familiar images. This classifier’s accuracy dropped sharply on an external dataset, and it remains a research system rather than a clinical diagnostic tool.
Read the next reportExplore the full source and evidence record
What this means for people
- Clinicians and patients should not read a near-perfect benchmark score as evidence that a model is ready to diagnose thyroid disease; the study did not test prospective care, external hospitals or patient outcomes.
- Researchers and buyers can reduce false confidence by checking class definitions, minority-class precision and recall, and a transparent clinical baseline before comparing complex models.
Global context
Public tabular benchmarks are reused far beyond their original clinical setting because they are easy to download and compare. When documentation, labels and class balance go unexamined, leaderboard gains can travel into papers and product claims without demonstrating safer care in any country.
What the evidence does not yet show
- This is a controlled audit of one public benchmark, not a validation study in a clinic or a representative sample of current patients.
- The diagnosis-code match clarifies the archive's class structure, but it does not establish that the historic records or hormone cutoffs meet today's clinical standards.
- The journal is sharing an accepted peer-reviewed version before final copy-editing; the article may receive editorial changes when the Version of Record replaces it.
- The study has one author, reports no external funding and declares no competing interests; independent replication would still strengthen the conclusion.
What to watch next
- Independent reproductions of the label mapping and the leakage-free comparison
- Whether future papers retire misleading class names and publish macro-F1, precision-recall and per-class results
- Prospective, multisite thyroid studies comparing models with transparent clinical rules and clinician judgement
Living evidence record
Impact record IAI-17KOTKJ
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
20 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Scientific Reports published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 20 October 2026
Research question, topics and regions
Clinical machine-learning benchmark validity
Topics: Benchmark integrity · Clinical AI · Data labels · Class imbalance · Model evaluation
Regions: Türkiye · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.