Why did some AI agents lose accuracy after receiving the same tool result?
In a 158-question geochemistry benchmark, changing long simulator reports from inline text to retrievable files reduced accuracy for two tested GPT versions. The preprint traces much of the loss to agents choosing the wrong evidence, not failing to compute the answer.
The short version
- What happened
- Researchers tested six models on 158 execution-verified multiple-choice questions using the deterministic PHREEQC geochemistry simulator and audited 5,773 complete trajectories plus 11,007 simulator calls.
- Who it affects
- Scientists and engineers using agents need logs that show which artefact supported the final answer, not only whether a tool ran successfully.
- What remains unproven
- This is a 19 October 2026 preprint and has not been peer reviewed.
The agents often computed the right value and still chose wrongly
A new preprint finds that an AI agent's surrounding software can change accuracy even when the scientific simulator produces the same result. Researchers gave agents access to PHREEQC, a deterministic US Geological Survey geochemistry program, and asked 158 four-option questions whose answers they had re-executed and verified. Long reports were either placed directly in the model's context, capped at 100,000 characters, or stored behind a section index that the agent could inspect with file-reading tools.[1]
Who it affects
Scientists and engineers using agents need logs that show which artefact supported the final answer, not only whether a tool ran successfully.[1]
Keep in mind
This is a 19 October 2026 preprint and has not been peer reviewed.
The 158 questions came from one geochemistry simulator benchmark; they do not represent all scientific or workplace tool use.
Only GPT-5.1 retained a statistically significant retrieval penalty after correction across headline model comparisons; several models showed no detectable net effect.
The accessible manuscript did not display a funding or conflict-of-interest statement.
Source analysis · 1 linked source. Original source date: 19 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The agents often computed the right value and still chose wrongly
A new preprint finds that an AI agent's surrounding software can change accuracy even when the scientific simulator produces the same result. Researchers gave agents access to PHREEQC, a deterministic US Geological Survey geochemistry program, and asked 158 four-option questions whose answers they had re-executed and verified. Long reports were either placed directly in the model's context, capped at 100,000 characters, or stored behind a section index that the agent could inspect with file-reading tools.
For GPT-5.1, indexed retrieval reduced pooled accuracy by 11.3 percentage points compared with inline reports; for GPT-5.2 the reduction was 7.9 points. Claude Opus 4.6, Claude Sonnet 4.6 and GPT-5.4 showed no detectable net change. After correcting tests across the five headline models, the GPT-5.1 difference remained statistically significant, while GPT-5.2 did not. Gemini 3 Flash Preview was described separately because many runs failed to submit an answer within the step budget.[1]
An audit located the failure in evidence use
The team inspected 5,773 complete agent trajectories and re-ran all 11,007 logged simulator calls. The GPT agents generated the keyed value about as often under both delivery methods. Under retrieval, however, they frequently skipped the authoritative report and read a shorter file they had created themselves. That file often contained extra decimal places, and the agents sometimes truncated rather than rounded to the precision used by the answer choices—a pattern the authors call evidence substitution.
A stronger paired test froze GPT-5.2's simulator output at 131 checkpoints and replayed each item three times under both delivery conditions. Indexed retrieval scored 61.8% across 393 continuations, versus 71.2% inline: a 9.4-point difference with a 95% bootstrap interval from 4.6 to 14.5 points lower. Because the computation was held fixed, this experiment more directly ties the gap to how the same evidence reached the agent.[1]
The distinction that matters
Where the answer can fail
The audit separates producing, seeing and using the correct value.
Build
Simulator produces the keyed value
Computation can be correct
Exposure
Harness decides what the agent reads
Authoritative report may be skipped
Closure
Agent selects and formats the answer
Derivative output can displace the source
The practical fix was specific—and the boundary is narrow
Adding an instruction to answer from the official simulator report raised accuracy by 6.3 to 7.9 points for the three GPT versions tested. Merely reminding agents to round correctly did not help. That distinction matters for laboratories and engineering teams: a traceable tool result is not enough if the agent can silently prefer a derivative file, summary or calculation that looks easier to use.
The result does not show that retrieval is generally harmful. It covers one simulator, one multiple-choice benchmark, fixed prompts and a limited set of model versions. Several models had no detectable net penalty, and GPT-5.4's better computations offset an estimated evidence-use loss. The questions also reward exact selection among four options, so they may magnify precision and rounding errors differently from an open scientific report. The manuscript is a preprint, not peer reviewed; it states no visible funding or conflict disclosure. Replication on open-ended scientific tasks, other tools and newer models would determine whether authoritative-source instructions are a robust safeguard or a benchmark-specific patch.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Evidence check
This is a 19 October 2026 preprint and has not been peer reviewed.
The 158 questions came from one geochemistry simulator benchmark; they do not represent all scientific or workplace tool use.
Only GPT-5.1 retained a statistically significant retrieval penalty after correction across headline model comparisons; several models showed no detectable net effect.
The accessible manuscript did not display a funding or conflict-of-interest statement.
Reporting basis: Source analysis. Original source date: 19 October 2026.
Continue the story
Read next
Continue exploring ai agents: compare the finding and its evidence limits.
Can an AI agent audit cancer studies? A 20-paper test found useful triage—and many false alarms
An unreviewed benchmark reports that an agent reproduced 77.4% of 2,190 claims from US cancer-registry papers. But expert review rejected nine of ten sampled robustness warnings as consequential changes.
Read the next reportExplore the full source and evidence record
What this means for people
- Scientists and engineers using agents need logs that show which artefact supported the final answer, not only whether a tool ran successfully.
- People affected by automated technical decisions may face errors that look like model reasoning failures but originate in file selection, precision handling or interface design.
Global context
Agent systems increasingly store long tool outputs outside the immediate model context. That can save tokens and support larger workflows, but it also turns evidence selection into part of the system's safety case. Audits therefore need to test the model, tools and harness together rather than treating the model alone as the product.
What the evidence does not yet show
- This is a 19 October 2026 preprint and has not been peer reviewed.
- The 158 questions came from one geochemistry simulator benchmark; they do not represent all scientific or workplace tool use.
- Only GPT-5.1 retained a statistically significant retrieval penalty after correction across headline model comparisons; several models showed no detectable net effect.
- The accessible manuscript did not display a funding or conflict-of-interest statement.
What to watch next
- Peer review and independent replication with other scientific, legal and operational tools
- Whether explicit source hierarchy improves open-ended work without hiding contradictory evidence
- Harness evaluations that report build, evidence exposure and final-answer errors separately
Living evidence record
Impact record IAI-1IB5W5R
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
20 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 20 October 2026
Research question, topics and regions
How tool-output delivery changes the accuracy of scientific LLM agents
Topics: AI agents · Tool use · Context management · Scientific computing · Preprint
Regions: United States · Global research
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.