Should an AI agent turn one failure into a rule? A registered experiment found chance can stick
Across 90,084 synthetic answers, a lesson written after a random failure sometimes pushed a fresh model session towards the less reliable tool. A companion preprint found that rewritten instructions also need to match the model expected to use them. Neither study establishes production harm or safe autonomous self-improvement.
The short version
- What happened
- A registered synthetic study tested 256 worlds, 11 model endpoints and 90,084 answers; after a chance failure, an AI-written lesson increased selection of the worse tool from 0.4% to 11.6% in the main comparison.
- Who it affects
- Teams using agents that revise prompts, procedures or memories should not let one failed run silently become policy. For work affecting patients, learners, employees, public services, money or access, retain the event record, test the proposed change and keep approval with an accountable person.
- What remains unproven
- Both studies are preprints and not peer reviewed
Keep in mind
Both studies are preprints and not peer reviewed
Main risk study uses synthetic worlds and a first-tool choice without execution
One stored lesson was tested, not accumulated memory
No independent replication or external human review reported for the luck study
Skill study's main testbed deliberately selected 103 sensitive pairs from 607 tasks
Five-round skill experiments were run once per round
No production incident rate or regional outcome measured
Funding and conflicts not stated in the accessible manuscripts
Multi-source analysis · 3 linked sources. Primary analysed source date: 27 October 2026. Latest linked supporting source: 28 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 5 min The evidence, methods, implications and limitations
A random failure became advice for the next decision
A registered preprint has isolated a plausible failure mode in agents that write lessons from experience. The researchers created 256 synthetic worlds, each with two interchangeable tools whose reliability was known. A scripted earlier run used one tool and either succeeded or failed by chance. A fresh session of the same model then saw the record plus one stored memory item and chose which tool it would use first next time. Across 11 model endpoints, the study collected 90,084 answers.
When the better tool had failed by chance, a lesson written from that run made readers choose the worse tool in 11.6% of next decisions. The rate was 0.4% with irrelevant memory and 0.4% when the lesson followed success. The estimated increase against irrelevant memory was 11.2 percentage points, with a 95% interval from 9.7 to 12.6 points. That is evidence that a model-written summary can outweigh the underlying record in this artificial setting—not a measured failure rate for deployed agents.[1]
Guardrails helped, but the effect varied by model
The authors tried two remedies. Asking the lesson writer to focus on decision quality rather than the outcome reduced worse-tool choices to 7.5%; an outcome-neutral instruction reduced them to 3.1%. An explicit statement that each run was statistically independent also weakened the effect descriptively. None of these controls eliminated it across every endpoint, which matters because a single aggregate safeguard can conceal different model behaviour.
The experiment was deliberately narrow. The reader named a first tool but did not execute it; it received one lesson, not a growing memory; and the worlds were synthetic. It had an authoritative reliability record, while real systems may have stale or incomplete evidence. The paper was not independently replicated, reports no external human review and does not separate the influence of lesson writer from lesson reader. Funding and conflicts were not stated in the accessible manuscript.[1]
The distinction that matters
How a one-off outcome can become a standing instruction
The risk appears when an observation, an explanation and an operating rule are treated as if they carried the same evidence.
Observe
One run succeeds or fails, sometimes by chance
Retain the raw event and uncertainty
Write
The model turns that outcome into a lesson
Test whether the claimed cause is supported
Reuse
A fresh session reads the stored lesson
Approve scope, monitor results and preserve rollback
Better instructions are possible, but they are model-specific changes
A companion preprint tested a different side of the same operational problem: adapting written skills without changing model weights. Seven models faced three benchmarks containing 607 tasks. Researchers selected 103 task–instruction pairs where model capability appeared especially important. On that selected set, a fixed meta-skill plus a stronger rewriting model improved weaker models by 5.1 to 14.1 points; five rounds of self-evolution ended 2.0 to 5.1 points above a comparison loop without the extra guidance.
Those figures are not a universal gain. The subset was chosen for skill sensitivity, each round ran once, retrieval was out of scope and the authors say not to overinterpret small exact gaps. Together, the papers support a cautious workflow: keep the raw event separate from the proposed lesson, test old and new instructions on comparable cases, scope the change to a named model and task, and keep consequential changes reversible and reviewable. The assessment would strengthen with independent replication, real executed workflows, accumulated-memory tests, preregistered production trials and evidence that safeguards transfer across models.[2][3]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring ai agents: compare the finding and its evidence limits.
Can an AI agent reliably improve the software that controls it?
Across 27 autonomous harness lineages and 2,025 game evaluations, two rounds of self-revision produced only a small, uncertain average gain on held-out tasks. The unreviewed result separates impressive software construction from dependable self-improvement.
Read the next reportExplore the full source and evidence record
What this means for people
- Teams using agents that revise prompts, procedures or memories should not let one failed run silently become policy. For work affecting patients, learners, employees, public services, money or access, retain the event record, test the proposed change and keep approval with an accountable person.
Global context
The mechanism could matter wherever written experience is reused, but neither study compared countries, languages, sectors or regulatory settings. Connectivity, tool availability, language and local retention duties may change what counts as a temporary failure and whether an event may be stored at all.
What the evidence does not yet show
- Both studies are preprints and not peer reviewed
- Main risk study uses synthetic worlds and a first-tool choice without execution
- One stored lesson was tested, not accumulated memory
- No independent replication or external human review reported for the luck study
- Skill study's main testbed deliberately selected 103 sensitive pairs from 607 tasks
- Five-round skill experiments were run once per round
- No production incident rate or regional outcome measured
- Funding and conflicts not stated in the accessible manuscripts
What to watch next
- Independent replication across model families
- Executed tasks rather than stated tool choices
- Long-lived and accumulated memory
- Counterfactual tests separating bad luck from changed conditions
- Human approval and rollback in high-impact workflows
- Open code and repeated runs for skill adaptation
- Production trials with preregistered outcomes
Living evidence record
Impact record IAI-1F5B9PT
Evidence stage
Studied
Confidence
Corroborated
Reporting basis
Multi-source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
28 October 2026
Source trail
3 direct sources across 2 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Evidence trail
Sources used for this report
Links checked 28 October 2026
Research question, topics and regions
Topics: AI agents · Agent memory · Self-improvement · LLM reliability · Operational controls · Preprint research
Regions: International
This report is labelled multi-source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.