Why did ‘uncertain’ AI choices fail to beat random sampling?
A peer-reviewed materials study found that several uncertainty-driven active-learning strategies often matched or trailed random selection on deliberately unfamiliar chemistry. Calibration changed confidence estimates, but usually did not reveal the information the models needed.
The short version
- What happened
- The experiments used 1,128 measured-solubility compounds and JARVIS-22 materials data, withholding fluorine- or iron-containing chemistries to create deliberately unfamiliar tests.
- Who it affects
- Materials scientists and drug-discovery teams use active learning to decide which costly experiment comes next. A poorly chosen uncertainty signal can spend that budget without buying the expected improvement.
- What remains unproven
- The paper evaluates three regression tasks and constructed distribution shifts; other scientific domains and acquisition objectives may behave differently.
Keep in mind
The paper evaluates three regression tasks and constructed distribution shifts; other scientific domains and acquisition objectives may behave differently.
Several comparisons have overlapping error bars, so the evidence supports ‘often no better’ rather than a universal ranking of methods.
The study is computational and retrospective. It does not measure time, cost or discovery success in a live laboratory.
The authors report Schmidt AI in Science and Canada First Research Excellence Fund support and declare no competing interests; one author lists an affiliation with Artificial, Inc.
Multi-source analysis · 2 linked sources. Primary analysed source date: 24 October 2026. Earliest linked supporting source: 21 November 2025.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Calibration fixed the scale, not the search problem
Active learning is meant to save experiments: a model chooses the next sample expected to teach it most. In this peer-reviewed study, researchers compared uncertainty estimates from neural-network ensembles, loss-landscape sampling, random forests and XGBoost. They tested aqueous solubility on 1,128 ESOL compounds and two JARVIS-22 materials properties—bandgap and formation energy—while withholding fluorine- or iron-containing materials to mimic unfamiliar chemistry.
For the large materials partitions, more than 56,000 in-distribution samples trained the models. The out-of-distribution test sets ranged from 163 to 1,087 materials depending on task and split. Calibration could make confidence intervals look statistically better aligned, yet uncertainty-maximising acquisition often matched or trailed random sampling. In the fluorine formation-energy task, loss-landscape uncertainty did not beat random selection until roughly half of the held-out training pool had been added; error bars overlapped in several comparisons.[1][2]
A warning for expensive discovery loops
A laboratory could waste scarce synthesis or assay capacity if a model’s widest error bars do not point to the samples that close its real knowledge gaps. The paper’s practical recommendation is to study uncertainty in the feature-input space and consider tools such as counterfactual explanations when data are genuinely out of distribution.
The result is bounded: it uses three prediction tasks and synthetic hold-outs based on chemistry membership, not prospective autonomous laboratories. Confidence would rise with preregistered, repeated campaigns in which real experiments—not database lookups—test whether alternative acquisition rules reach a target performance with fewer samples.[1][2]
The distinction that matters
What the study actually compared
The same loop was tested with different signals for choosing the next unfamiliar sample.
Train
Fit models without fluorine- or iron-containing materials
Creates a deliberate distribution shift
Choose
Add samples by highest estimated uncertainty or at random
Tests whether confidence spread finds useful missing information
Measure
Track prediction error on held-out chemistry
Random selection often matched or beat uncertainty maximisation
Watch the background, then check the evidence
Illustrative photo · not a video frame
Google DeepMind · 1 min 52 sec · Background explainer
Protein folding explained
YouTube loads only when you choose; playback starts when you press play.
A short introduction to why a protein’s shape matters. It provides background for understanding what AI structure prediction contributes to biological research.
Evidence check: This is the developer’s explanation. Predicting a structure does not by itself establish a treatment’s safety or benefit for patients.
Published 2020-11-30. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current researchContinue the story
Read next
Continue exploring active learning: compare the finding and its evidence limits.
Can a general-purpose LLM search crystal compositions?
In a closed computational benchmark, GPT-5.4 recovered 95.65% of 3,740 low-energy Elpasolite targets within 5,000 proposals. Iterative evaluator feedback drove the result; no new material was synthesised, and the approximate energy labels are not proof of stability or usefulness.
Read the next reportExplore the full source and evidence record
What this means for people
- Materials scientists and drug-discovery teams use active learning to decide which costly experiment comes next. A poorly chosen uncertainty signal can spend that budget without buying the expected improvement.
Global context
The team spans the University of Toronto, KAUST and the US National Institute of Standards and Technology. The datasets are public, making replication possible, although the accepted article’s early online version may still receive copy-editing changes.
What the evidence does not yet show
- The paper evaluates three regression tasks and constructed distribution shifts; other scientific domains and acquisition objectives may behave differently.
- Several comparisons have overlapping error bars, so the evidence supports ‘often no better’ rather than a universal ranking of methods.
- The study is computational and retrospective. It does not measure time, cost or discovery success in a live laboratory.
- The authors report Schmidt AI in Science and Canada First Research Excellence Fund support and declare no competing interests; one author lists an affiliation with Artificial, Inc.
What to watch next
- Prospective laboratory comparisons against random and diversity-based sampling
- Replication on classification, multi-objective and safety-constrained discovery tasks
- Input-space uncertainty measures that identify which chemistry is actually missing
- Released code matching the accepted article and sensitivity to different hold-out definitions
Living evidence record
Impact record IAI-0UP4M1Q
Evidence stage
Studied
Confidence
Supported
Reporting basis
Multi-source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
24 October 2026
Source trail
2 direct sources across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Evidence trail
Sources used for this report
Links checked 24 October 2026
Research question, topics and regions
Whether calibrated model uncertainty helps active learning select out-of-distribution materials or molecules more efficiently than random sampling
Topics: Materials discovery · Active learning · Uncertainty calibration · Out-of-distribution data · Scientific machine learning
Regions: Canada · Saudi Arabia · United States · Global
This report is labelled multi-source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.