Biology agents could look up protein facts—but geometry exposed a hidden weakness
A 131-task preprint found that three frontier agents were much stronger at metadata lookup than coordinate calculations. Overall scores concealed the split, and rankings changed with evaluation choices.
The short version
3 key points
- What changed
- ProteinGeoBench contains 131 expert-written tasks: 36 metadata, 51 sequence, 28 numeric and 16 concept-reasoning questions; the four-model comparison used 4,249 rollouts.
- Why it matters
- Researchers should verify coordinate calculations and inspect execution traces even when an agent gives fluent, correct-looking protein metadata.
- What to watch
- Independent reruns on the released CC BY 4.0 task suite
One score hid two very different abilities
An unreviewed benchmark study found that frontier agents could retrieve protein-structure metadata more reliably than they could calculate distances or angles from coordinates. ProteinGeoBench supplies structure files, a natural-language question and an executable verifier for 131 tasks: 36 metadata lookups, 51 sequence checks, 28 numeric calculations and 16 concept-reasoning tasks. The main comparison ran 4,249 rollouts across Claude Opus 5, GPT-5.6-sol, Grok 4.5 and Gemini 3.1 Pro Preview.[1]
Who it affects
Researchers should verify coordinate calculations and inspect execution traces even when an agent gives fluent, correct-looking protein metadata.[1]
Keep in mind
The preprint is company-affiliated and has not been peer reviewed.
The 131 tasks lean towards biophysics, and different experiments use different, sometimes purposively selected task subsets.
Thirty unpublished-structure tasks derive from two substrates, so they do not represent confidential or novel structures generally.
Source analysis · 1 linked source. Original source date: 19 October 2026.
Read the full report · 3 min The evidence, methods, implications and limitations
One score hid two very different abilities
An unreviewed benchmark study found that frontier agents could retrieve protein-structure metadata more reliably than they could calculate distances or angles from coordinates. ProteinGeoBench supplies structure files, a natural-language question and an executable verifier for 131 tasks: 36 metadata lookups, 51 sequence checks, 28 numeric calculations and 16 concept-reasoning tasks. The main comparison ran 4,249 rollouts across Claude Opus 5, GPT-5.6-sol, Grok 4.5 and Gemini 3.1 Pro Preview.
Overall means ranged from 0.707 to 0.816. The leading three were within 0.031, and bootstrap intervals did not resolve their order. Yet the family split was material: for example, Claude Opus 5 averaged 0.907 on lookup and 0.716 on numeric tasks. Twelve tasks scored below 0.5 for every model; numeric questions were 28 of 131 tasks but half of those shared failures.[1]
Evaluation settings changed the apparent result
With only one run per task, 23.9% of bootstrap replicates put a different model first. On a selected 15-task slice, changing the agent harness moved scores from -0.071 to +0.086, while removing network access hurt more than blocking eight structural-database domains. Those access contrasts were graded in different passes, so the residual cannot be cleanly attributed. Across a wider 26-model sweep, numeric and non-numeric performance rose together rather than revealing a fixed geometry ceiling.
For structural biologists, the useful lesson is operational: demand per-family results, repeat counts, tool permissions and network conditions before trusting an aggregate benchmark. Thirty tasks used structures absent from public databases during evaluation, but they came from only two substrates. The suite leans towards biophysics and several comparisons use small, purposively chosen subsets. The released tasks and verifiers make replication possible; independent labs should add broader unpublished structures and prospective research workflows before these agents are trusted with consequential analysis.[1]
The distinction that matters
Why the aggregate score can mislead
The benchmark combines tasks that depend on different abilities and resources.
Lookup
Retrieve recorded database metadata
High scores for three agents
Geometry
Calculate from atomic coordinates
Lower and shared failures
Evaluation setup
Change runs, tools or network access
Shift scores and rankings
Watch the background, then check the evidence
Illustrative photo · not a video frame
Google DeepMind · 1 min 52 sec · Background explainer
Protein folding explained
YouTube loads only when you choose; playback starts when you press play.
A short introduction to why a protein’s shape matters. It provides background for understanding what AI structure prediction contributes to biological research.
Evidence check: This is the developer’s explanation. Predicting a structure does not by itself establish a treatment’s safety or benefit for patients.
Published 2020-11-30. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current researchEvidence check
The preprint is company-affiliated and has not been peer reviewed.
The 131 tasks lean towards biophysics, and different experiments use different, sometimes purposively selected task subsets.
Thirty unpublished-structure tasks derive from two substrates, so they do not represent confidential or novel structures generally.
Reporting basis: Source analysis. Original source date: 19 October 2026.
Continue the story
Read next
Also about benchmarks
Can a geospatial AI agent reliably answer planet-scale questions?
A Google preprint reports large benchmark gains from specialist geospatial tools and, on open-ended tasks, planning with one Gemini model. The evidence is vendor-produced, unreviewed and does not establish reliable field decisions.
Read the 8 min reportAlso about benchmarks
DeepSeek’s open AI update: what its agent benchmarks show
DeepSeek reports stronger agent and coding benchmarks. Its own model card supplies the results; independent checks and real-work performance remain open questions.
Read the 5 min reportExplore the full source and evidence record
What this means for people
- Researchers should verify coordinate calculations and inspect execution traces even when an agent gives fluent, correct-looking protein metadata.
- Labs comparing products need matched tools, access and repeated runs; a leaderboard position alone is not evidence of reliable scientific work.
Global context
Scientific-agent benchmarks increasingly mix model capability with databases, code tools and orchestration. ProteinGeoBench shows why laboratories need configuration-level reporting and domain-specific error analysis before treating aggregate scores as evidence of research reliability.
What the evidence does not yet show
- The preprint is company-affiliated and has not been peer reviewed.
- The 131 tasks lean towards biophysics, and different experiments use different, sometimes purposively selected task subsets.
- Thirty unpublished-structure tasks derive from two substrates, so they do not represent confidential or novel structures generally.
What to watch next
- Independent reruns on the released CC BY 4.0 task suite
- Broader unpublished structures and prospective laboratory workflows
- Matched experiments that vary one harness or access factor at a time
Living evidence record
Impact record IAI-01RVJ2Z
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
20 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 20 October 2026
Research question, topics and regions
How model, task family, network access and agent harness affect structural-biology benchmark results
Topics: Structural biology · Scientific agents · Benchmarks · Protein research
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
The Impact Brief · Free
Keep following the stories that matter to you.
Get our weekday briefing on what changed and why it matters. Choose your topics below.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.