Did training AI for diversity produce better science ideas?
An 8B model scored well on an automated idea-portfolio metric and matched a frontier model in a four-expert pilot. That is not evidence that the ideas became successful research.
The short version
3 key points
- 1.The researchers trained an 8B model on 7,172 document-grounded instances and evaluated 1,740 contamination-controlled cases across 16 domains.
- 2.Its automated creative-utility score was 0.64 versus 0.38 for Claude Opus 4.8, but the metric depended on model judges and a known published idea as the quality reference.
- 3.Four experts judged 60 idea portfolios; the trained model averaged 2.17 promising ideas out of three versus 2.20 for Claude, a small pilot that did not track scientific impact.
The training target rewarded a portfolio, not one polished proposal
A Salesforce-led team trained an 8-billion-parameter model to produce three scientific ideas that were individually promising and deliberately different from one another. On the authors’ automated ‘creative utility’ measure, its strongest version scored 0.64, ahead of 0.38 for Claude Opus 4.8 and 0.28 for GPT-6 Astra. That comparison shows the model learned the study’s portfolio objective; it does not show that its ideas became successful science.
The system received summaries of related papers and wrote structured idea cards covering a gap, claim, method and inspirations. During reinforcement learning, an idea earned less credit when it resembled an earlier idea in the same set. The model was trained on 7,172 instances and tested on 1,740 contamination-controlled cases drawn from 2026 papers across 16 domains.[1]
Automated scores and expert judgement told a more cautious story
Creative utility combined an automated judgement of quality against a known published ‘gold’ idea with model-judged distance between proposals. That makes the evaluation reproducible at scale, but it also rewards resemblance to what research later proved publishable and depends on other models’ preferences. A separate IdeaSeed check found every model’s population of methods less diverse than human proposals. The strongest no-thinking version scored 0.65 in normalised method entropy, against 0.87 for people.
The human pilot included four NLP experts, each with a PhD and at least four top-tier papers. They blind-rated 60 portfolios of three ideas across the systems. The trained model averaged 2.17 promising ideas and 2.83 unique ideas per portfolio; Claude averaged 2.20 and 2.72. That is encouraging, but four experts in one field cannot establish broad scientific value. The same team also handed 132 ideas to one coding agent for 90-minute, US$30 proof-of-concept runs. At least one idea received a self-reported supporting result in 33% of three-idea sets for the trained model versus 27% for Claude; most sets therefore produced no such support.[1]
The distinction that matters
What the diversity reward changed
Each new idea received credit for quality and for how far it moved from ideas already in the set.
Ground
Read summaries of related papers
Generate an idea tied to named sources
Compare
Judge quality against a published reference idea
Estimate whether the direction looks worthwhile
Diversify
Discount ideas similar to earlier ones
Reward a broader three-idea portfolio
Researchers still have to choose what deserves time
For research teams, generating a deliberately varied shortlist could reduce the tendency to repeat one fashionable direction. It could also create more low-cost proposals to inspect, shifting effort toward screening assumptions, novelty and feasibility. None of the tests measured whether an idea survived peer review, reproduced, helped another scientist or produced long-term impact.
Commercial interest is also material: four of six authors are affiliated with Salesforce AI Research and another completed the work there. The preprint is under review and reports no separate funding or competing-interests statement. A convincing next step would preregister idea selection, recruit independent experts across fields, fund comparable human and model proposals, and track blinded review, replication and research value over time. Until then, the study supports a narrower conclusion: set-level training can make model-generated idea portfolios look more varied without an obvious loss on small expert and coding-agent checks, not that it can automate scientific judgement.[1]
Evidence check
The study is an unreviewed, commercially affiliated preprint. Its primary metric relies on model judges, the expert pilot had four NLP researchers, and proof-of-concept outcomes came from one coding agent’s self-reports.
Reporting basis: Source analysis. Original source date: 15 October 2026.
Explore the full source and evidence record
What this means for people
- Scientists could receive a broader shortlist of possible directions, but may spend more time filtering plausible-looking ideas and checking source grounding.
- Research managers should not treat an automated creativity score as evidence that a proposal deserves funding or will survive peer review.
Global context
The training and evaluation corpus covered 16 research domains, but the human study was limited to NLP and ML expertise and the development team was predominantly affiliated with a US technology company.
What the evidence does not yet show
- The study is an unreviewed, commercially affiliated preprint. Its primary metric relies on model judges, the expert pilot had four NLP researchers, and proof-of-concept outcomes came from one coding agent’s self-reports.
What to watch next
- Independent, preregistered comparisons that track selected ideas through blinded review, funded execution, replication and measurable scientific contribution.
Living evidence record
Impact record IAI-11MQ8YD
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
16 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Wadhwa et al. (arXiv) published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 16 October 2026
Research question, topics and regions
Set-level reinforcement learning for diverse document-grounded scientific idea generation
Topics: Scientific research · Idea generation · Reinforcement learning · Research evaluation · Commercial interests
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Watch the background, then check the evidence
Google DeepMind
YouTube loads only when you choose. Press play in the player to start. If playback is unavailable, use “Watch on YouTube” below.
AI & scientific discovery · Background explainer
Published 2020-11-30 · 1 min 52 sec
Protein folding explained
A short introduction to why a protein’s shape matters. It provides background for understanding what AI structure prediction contributes to biological research.
Evidence check: This is the developer’s explanation. Predicting a structure does not by itself establish a treatment’s safety or benefit for patients.
Source checked 14 October 2026. Check available captions on the publisher’s video page.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Continue the story
There’s more to this story
Science & Research
Did the arthritis AI travel between cohorts?
A peer-reviewed analysis of gut-microbiome data from 2,238 people reached a mean internal ROC-AUC of 0.834, but its genus-level model fell to 0.439 in 39 independently processed Shanghai samples. The study is a warning about cross-cohort transfer, not evidence for a rheumatoid-arthritis diagnostic test.
8 min · 1 source
Science & Research
Can a failing AI policy reveal what a tumour model is missing?
A peer-reviewed simulation study used the different ways ten reinforcement-learning policies failed when moved into a richer tumour model to identify missing spatial dynamics. It is a method for interrogating models—not clinical evidence for an AI-designed cancer treatment.
8 min · 1 source
Science & Research
Who is responsible when AI agents do the research?
Human authors remain responsible, a new peer-reviewed review argues. AI research agents should be treated as powerful tools—not authors or scientists—and their use should trigger explicit oversight, disclosure and post-publication accountability.
8 min · 1 source
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.