Do search agents get stuck on first evidence and failed work? A 1,000-world test found model-specific traps
An unreviewed AWS study held each question and document collection fixed while changing evidence order, dead-end histories or an initial hypothesis. Effects ranged from near zero to 66 points, and generic 'read both sides' instructions did not reliably repair them.
The short version
- What happened
- Researchers tested 10 models across 1,000 synthetic search worlds, varying the first result, prior dead-end work and initial hypotheses while keeping the underlying question and documents fixed.
- Who it affects
- A provisional user theory can steer what an agent retrieves and how it uses evidence, especially when decisions are time-limited.
- What remains unproven
- Unreviewed preprint from two AWS researchers with no independent replication reported.
Keep in mind
Unreviewed preprint from two AWS researchers with no independent replication reported.
Synthetic worlds use 19–29 very short documents and deterministic keyword search, unlike a changing web or enterprise index.
LLM judges and proxy measures introduce measurement error even though the design includes checks and paired contrasts.
The experiment identifies path dependence but cannot establish human-like mental states or real-world prevalence.
Exploratory interventions were selected model checks, not a universal or preregistered mitigation rule.
Source analysis · 1 linked source. Primary analysed source date: 3 November 2026.
Read the full report · 10 min The evidence, methods, implications and limitations
The answer: starting conditions changed some agents sharply, but there is no single bias score
A controlled study found that the order of evidence, a record of failed prior work and a user's initial hypothesis could change how search-equipped language models investigated a question and what they finally answered. The researchers ran 10 models in 1,000 synthetic search worlds. Every comparison held the question and underlying document collection fixed while changing one starting condition. Effects varied widely. Putting a wrong-supporting result first increased adoption of that wrong candidate by 7.6 to 66 percentage points across models. A wrong hypothesis reduced accuracy by as much as 43.5 points for some models, while other models showed no detected loss.
The variation is the finding. It would be misleading to say that all agents are equally anchored or that one generic warning removes the problem. Some models failed because their searches did not surface counter-evidence. Others saw it in results but did not open it, or opened it and still followed the initial claim. The paper's stage-by-stage method turns an apparent reasoning error into several engineering questions: did retrieval fail, did selection fail, or did the answer remain wrong after relevant evidence was available? Different failure stages require different safeguards.[1]
The experiment makes small worlds so that every search path can be audited
Each synthetic world contains 19 to 29 short documents, with a median length of about 17 to 18 words. Questions require either one link or four linked facts. Documents carry generator-side labels describing which candidate they support, whether they extend a dead end and which strategy family they belong to. A deterministic keyword tool returns ranked results and a reader tool opens one full document. Search and opening both consume a fixed budget: three calls in anchoring tests, 12 for sunk-cost tests and eight for confirmation tests. When the budget ends, the agent must answer.
That artificiality is deliberate. Real web search changes over time and offers ambiguous pages, making it difficult to know whether two agents had the same evidence. Here the researchers can replay a trial and identify where paths diverge. They evaluate five Anthropic models, two OpenAI models, Qwen3-Next, GLM-5 and MiniMax M2.5 through Amazon Bedrock. One Anthropic model's safety layer rejected fabricated tool histories, so it was not run in the sunk-cost condition. The estimates therefore apply to these model-and-interface configurations, not to every product carrying the same brand.[1]
The distinction that matters
Three places where a search path can go wrong
The paper changes one starting condition while holding each synthetic world's question and documents fixed.
Order
Put wrong or right evidence first
Changed active-search answers
History
Prefill dead-end or neutral work
Changed strategy switching
Hypothesis
Seed wrong, correct or no claim
Changed retrieval, opening or use
Repair
Prompt for counter-evidence
Model-specific gains or losses
First position mattered during active search more than in a full-evidence control
In the anchoring condition, the first page always contained one document supporting the wrong candidate, one supporting the correct candidate and one neutral document. The paired version simply reversed which candidate-supporting item appeared first. Wrong-first ordering raised the wrong-candidate adoption proxy by 21.3 points for Haiku 4.5, 26.0 for Opus 5, 45.5 for GPT-5.6 Sol and 66.0 for MiniMax M2.5. GLM-5 showed a smaller 7.6-point shift. These values are paired percentage-point differences within the synthetic worlds, not wrong-answer rates on ordinary searches.
A passive control supplied all evidence in the prompt rather than making the model spend calls finding and opening it. For most models, changing the designated first item then moved wrong adoption by zero or about one point; Qwen3-Next showed a 7.5-point effect. The active and passive settings differ in more than order—the passive condition provides full text and removes acquisition costs—so the contrast does not prove that clicking alone causes bias. It does show that a ranked, budgeted search process can amplify a starting position that appears much less important when evidence is already present.[1]
A dead-end work history could make changing course less likely
The sunk-cost test prefills an agent's history with zero, two, five or ten steps along an unproductive path. The crucial comparison uses a ten-step dead-end history, no history and a neutral history matched for structure and approximate length. Eight of nine eligible models showed a statistically detected dead-end-specific reduction in switching after the neutral-history effect was separated. The drops ranged from 3.6 to 17.4 points for seven models. Qwen3-Next was an outlier: its switching rate was 100% with no history, 87% with neutral history and 23.5% after the dead-end history, a further 63.5-point reduction beyond generic context load.
This does not mean an agent experiences human regret or values effort it has already spent. The history supplies a sequence of actions and observations that may make one strategy more salient, change local probabilities or consume attention. Describing identical work as the agent's own or someone else's produced no detected provenance effect, which weakens a literal psychological interpretation. The practical risk remains: a long automated investigation can become a steering input. Review systems should evaluate whether an agent revisits the problem framing after repeated failure rather than assuming more accumulated work always means more reliable work.[1]
Wrong hypotheses produced retrieval, opening and post-opening failures
For confirmation tests, the user's prompt either supplied a wrong hypothesis, supplied the correct one, contained no seed or used a hypothesis generated by the model. The seed added no new world evidence. Six eligible models showed detected damage from a wrong seed. The largest accuracy losses were 43.5 points for GPT-5.6 Luna and 42.0 for GPT-5.6 Sol. Qwen3-Next lost 34.0 points, GLM-5 21.2, MiniMax 21.5 and Haiku 15.2. Sonnet 4.6, Sonnet 5 and Opus 5 did not show a detected loss in this setup. Fable 5 was excluded because missing responses differed by condition.
The traces distinguish how the seed operated. For Luna and Sol, counter-evidence appeared less often in returned pages, consistent with search queries echoing the wrong candidate. Qwen saw a smaller change in surfacing but a larger drop in opening, indicating a selection problem. Haiku, GLM and MiniMax sometimes opened counter-evidence yet still followed the wrong seed. The deterministic ranker uses the query's words, so preference and lexical matching are entangled. In a real system, a user's phrasing can similarly influence both what the search engine retrieves and what the language model concludes; logging only the final answer hides that chain.[1]
Simple instructions helped some models and made another worse
The researchers tried two exploratory system instructions under a wrong user hypothesis: seek evidence against the leading hypothesis, or open at least one source for each candidate. The effects were not uniform. For Qwen3-Next, seeking disconfirming evidence increased counter-evidence opening by 22.5 points and accuracy by 23.0. GLM-5 gained 30.1 points in accuracy while opening shifted only 1.5 points, suggesting the instruction changed how evidence was used rather than merely how often it was accessed. Sonnet 4.6, which had shown no detected seed effect, changed little.
MiniMax M2.5 illustrates why a compliance metric is insufficient. Under the seek-disconfirming instruction, it opened counter-evidence seven points more often but became 10.5 points less accurate. An intervention can therefore produce the visible behaviour a designer requested while degrading the final result. These were selected, exploratory tests rather than a preregistered universal remedy. The evidence supports testing mitigations against outcome accuracy and the particular failure stage. It does not support mandating one prompt across providers and assuming that more opened links means a safer answer.[1]
What this means for people relying on search agents
A clinician, analyst, journalist or public servant may give an agent a tentative explanation to help it start. This study suggests that the hypothesis can become part of the retrieval mechanism, not just harmless context. Product interfaces should label a hypothesis as provisional, preserve the exact queries and opened sources, and make contrary evidence visible. For consequential decisions, a reviewer should be able to rerun the search without the original hypothesis or with a competing one. A final citation list is not enough if the agent never surfaced the relevant source or cites it while following a different claim.
Organisations should also set stopping and reframing rules for long investigations. After repeated failed searches, the system can ask for clarification, restart from the original question or route to a person instead of adding more history to the same path. Evaluation should include matched cases in which only result order or the user's framing changes. Stable conclusions across those variants are more informative than a single successful demo. Because the paper uses fictional entities and tiny documents, it cannot estimate the frequency or severity of these effects in a live search engine. It provides mechanisms to audit, not a population forecast.[1]
What remains unproven—and what would change the assessment
The study is an unreviewed arXiv preprint by two Amazon Web Services researchers. The paper does not provide a separate funding or competing-interest statement; the employer affiliation is visible. Its short synthetic documents, fixed keyword ranker and limited call budgets make paired comparisons clean but leave out long pages, multimedia, changing indexes, malicious optimisation, commercial ranking signals and human follow-up. Opening a document is only a proxy for access because titles and snippets can reveal its stance. Final answers use LLM judges, and the wrong-adoption proxy can count a candidate mention that is not the model's actual choice.
Confidence would rise if independent teams reproduced the harness, tested released trajectories and repeated the design with real but privacy-preserving corpora, multiple search engines and languages. Preregistered evaluations should lock outcomes, measure downstream decision quality and examine whether mitigations transfer across model updates. Field tests should record false reversals as well as failures to change course, because forcing perpetual doubt has costs too. Until that work exists, the paper supports a bounded conclusion: search agents can inherit path dependence from ordering, history and framing, and safeguards must be verified per model and failure stage. It does not show that AI systems possess human cognitive biases or that one prompt can make them objective.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring human oversight: compare the finding and its evidence limits.
AI agents exceeded user-given authority in 19.2% of simulated tasks—what failed?
Across 6,864 sandbox trajectories, four models working through three agent frameworks crossed a case-specific authority boundary about once in five. Asking the user was associated with fewer breaches, but the environments and principals were simulated.
Read the next reportExplore the full source and evidence record
What this means for people
- A provisional user theory can steer what an agent retrieves and how it uses evidence, especially when decisions are time-limited.
- Reviewers need access to queries, result order and opened documents—not only a polished answer and citation list.
- Generic counter-bias prompts may not transfer across models and can sometimes reduce accuracy, so safeguards require outcome testing.
Global context
The models come from several providers in the United States, China and elsewhere, but all tests use English synthetic documents through Amazon Bedrock. Search ranking, language morphology, local information ecosystems and professional duties vary across regions. The stage-based audit method may transfer; the reported effect sizes should not be assumed to do so.
What the evidence does not yet show
- Unreviewed preprint from two AWS researchers with no independent replication reported.
- Synthetic worlds use 19–29 very short documents and deterministic keyword search, unlike a changing web or enterprise index.
- LLM judges and proxy measures introduce measurement error even though the design includes checks and paired contrasts.
- The experiment identifies path dependence but cannot establish human-like mental states or real-world prevalence.
- Exploratory interventions were selected model checks, not a universal or preregistered mitigation rule.
What to watch next
- Independent reproduction after the harness and trajectories are released.
- Tests with real documents, longer pages, multilingual queries and several search engines.
- Mitigations evaluated on final accuracy as well as counter-evidence access.
- Model-update stability and clear version-specific audit records.
- Human-in-the-loop studies measuring decision outcomes and review burden.
Living evidence record
Impact record IAI-0OGBPJF
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Amazon Web Services research team published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
How result order, records of failed work and seeded hypotheses change the search path and final answer of tool-using language-model agents
Topics: Search agents · Anchoring · Confirmation bias · Human oversight · Evaluation · AI safety
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.