Coding agents finished the job—but did they preserve what they changed?
Six frontier models completed 86.1% of 205 executable tasks yet backed up before the benchmark's regret point only 15.2% of the time. A plug-in layer improved recoverability, but the benchmark and evaluator were built by the same team.
The short version
- What happened
- RegretBench turns 205 regret reports from GitHub, Stack Overflow and other public posts into executable multi-turn tasks spanning local and external state.
- Who it affects
- Developers and other workers can lose files, configuration or remote records when an agent overreaches. Default recovery points could reduce harm, provided people can control retention and sensitive data are protected.
- What remains unproven
- The unreviewed benchmark translates public regret reports into authored tasks and may miss agent-specific or domain-specific failures.
Keep in mind
The unreviewed benchmark translates public regret reports into authored tasks and may miss agent-specific or domain-specific failures.
The system's designers also built the benchmark and primary evaluator; independent reproduction is absent.
Character-level similarity can miss semantic or operational differences, while indiscriminate backup can create privacy and storage risks.
Samsung supported the work; the authors say the findings do not necessarily reflect the funder's views.
Source analysis · 1 linked source. Original source date: 20 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
Capability did not imply reversibility
The team collected 205 public moments where people said they regretted not preserving prior state, then converted each into an executable multi-turn task. Changes included files, databases, messages and remote-service state. Six frontier models ran in coding-agent harnesses. They completed 86.1% of tasks on average but made a usable backup before the annotated regret action in only 15.2%.
That gap matters because a high task-completion score can hide irreversible side effects. The authors' audit of five mainstream harnesses found workspace or native-edit protection in several products, but no general coverage of state changed through external tools. RegretBench therefore scores both whether a backup occurred before the critical action and whether it held enough information to recover the affected state.[1]
An external backup layer improved recovery at a cost
BackUpper intercepts state-changing tool calls. Simple operations use rules; ambiguous calls go to a small backup subagent, with prior decisions distilled into reusable rules. On the 205 tasks it covered every annotated regret action and restored state to 92.2% average character-level similarity, versus 14.3% without restoration and 33.8% for post-hoc reconstruction. Reported overhead averaged $0.014 per task.
Coverage is not the same as complete recovery: regenerated identifiers and metadata caused residual differences, and unnecessary backups can create privacy, storage and retention costs. Human annotators confirmed 94% of judge-positive backups, while personalised scope was tested on 60 tasks. The benchmark derives from human regret reports that may not capture agent-specific failures, and the designers evaluated their own system. Independent, adversarial and longitudinal tests would change the assessment.[1]
The distinction that matters
Reversibility before a tool call
The safeguard sits outside the main coding agent and acts before state changes.
Inspect
Check each proposed tool call
Identify affected state
Preserve
Use a rule or backup subagent
Record enough state for recovery
Restore
Replay the backup after regret
Measure how closely the original returns
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring software engineering: compare the finding and its evidence limits.
OpenAI’s legacy-code scores jumped sharply—does that make migration safe?
OpenAI reports large gains on a 20-language SQLite task and a hidden-test COBOL migration evaluation. The test suite is public, but the results remain company-run and do not establish that production modernisation is safe.
Read the next reportExplore the full source and evidence record
What this means for people
- Developers and other workers can lose files, configuration or remote records when an agent overreaches. Default recovery points could reduce harm, provided people can control retention and sensitive data are protected.
Global context
Backup, privacy and retention obligations differ by sector and jurisdiction. A reversible-agent design should minimise collected state, encrypt it, enforce expiry and respect user or organisational policy.
What the evidence does not yet show
- The unreviewed benchmark translates public regret reports into authored tasks and may miss agent-specific or domain-specific failures.
- The system's designers also built the benchmark and primary evaluator; independent reproduction is absent.
- Character-level similarity can miss semantic or operational differences, while indiscriminate backup can create privacy and storage risks.
- Samsung supported the work; the authors say the findings do not necessarily reflect the funder's views.
What to watch next
- Independent benchmark replication
- Tests on production tools and remote services
- Privacy-preserving backup and retention controls
- Incident evidence showing whether recovery points reduce real user loss
Living evidence record
Impact record IAI-0K41KMT
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
22 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 22 October 2026
Research question, topics and regions
Whether coding agents preserve state before changes users later regret and whether a tool-call backup layer improves recovery
Topics: Coding agents · Backups · Reversibility · Automation · Software engineering
Regions: United States · North America · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.