Can AI agents diagnose real distributed-system failures? A new benchmark finds an evidence gap
Across 120 historical failures, leading agents often named the right file while missing the precise code blocks that caused the incident. The benchmark is an unreviewed preprint and tests diagnosis, not safe autonomous repair.
The short version
- What happened
- DistDebug-Bench contains 120 real bugs from 11 distributed systems, five programming languages and ten failure patterns, with 218 expert-labelled root-cause code blocks.
- Who it affects
- On-call engineers may gain a faster shortlist, but still need to verify the exact causal path before acting.
- What remains unproven
- Unreviewed preprint based on historical open-source incidents rather than prospective live deployments.
Keep in mind
Unreviewed preprint based on historical open-source incidents rather than prospective live deployments.
Some hard subgroups are small, including only seven cross-component opaque incidents.
Reference fixes may not capture every plausible causal intervention point.
The benchmark tests diagnosis, not autonomous repair, recovery time, security or operational harm.
Source analysis · 2 linked sources. Primary analysed source date: 2 November 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The direct answer: agents can narrow the search, but exact diagnosis remains unreliable
A new benchmark suggests that coding agents can often point engineers towards the right part of a large distributed system, but still struggle to identify the precise code that caused a failure. DistDebug-Bench evaluates 120 historical bugs drawn from 11 open-source distributed systems written in five programming languages. The authors labelled 218 root-cause code blocks and asked agents to diagnose each incident from the report, available evidence and the pre-fix repository. That is closer to real incident work than benchmarks that hand a model a neat failing test and a small file.
The distinction between finding a file and explaining a cause is the main result. Leading configurations achieved high file coverage, yet block-level coverage and precision fell substantially. Mini-SWE with GPT-5.5, for example, produced block precision of 34.7% and named an average of 3.6 blocks per diagnosis. An engineer may therefore receive a plausible shortlist containing the real cause alongside several irrelevant locations. That can save search time, but it cannot be treated as a verified root-cause analysis without checking the execution path and incident evidence.[1]
Why these failures are harder than ordinary coding tasks
Distributed-system incidents rarely present one clean error at one obvious location. In this collection, 45% of bugs required changes or reasoning across multiple root-cause blocks, 23% spanned multiple files, 69% of reports did not name the gold method and 63% contained no usable internal stack trace. The benchmark covers ten failure patterns, including timing, concurrency, state-machine and cross-component behaviour. Its input resembles the incomplete evidence an on-call engineer might actually receive: an issue description, logs or traces when available, and a large repository containing many plausible paths.
The hardest cases were those without a useful stack trace or method name. On 42 opaque, no-stack incidents, the best reported block coverage was 65.0%. On seven cross-component opaque bugs it fell to 40.5%. By contrast, local or adjacent bugs whose report named the method reached 95.8%, and no-stack reports that still named the method reached 94.8%. The direction is credible—specific clues help—but the seven-case cross-component result is exploratory. It should not be converted into a general failure rate for every production system.[1]
The distinction that matters
Where diagnosis quality falls away
The benchmark separates broad localisation from the exact code evidence needed for a causal diagnosis.
Method named
Local or adjacent bug
95.8% best block coverage
No stack
Method still named
94.8% best block coverage
Opaque report
No usable stack or method
65.0% across 42 cases
Cross-component
Opaque report
40.5% across seven cases
How the benchmark was labelled and checked
The researchers traced the historical fix for each issue and labelled the source blocks whose behaviour had to change to resolve it. Three annotators independently reviewed a retained set of 51 annotation sheets. Agreement was high at broad levels: Krippendorff's alpha was 0.919 for files and 0.866 for functions. Block coverage agreement was 0.861 and block-level F1 was 0.848. Those figures make the reference labels more credible than a benchmark built solely from automated commit diffs, although they do not make causality perfectly objective.
The benchmark also tries to separate genuine repository reasoning from memorised issue knowledge. Future commits were removed from the repositories. A leakage screen using GPT-5.4 reduced flagged attempts from 102 of 360 to four of 360 after the authors' controls. In a 66-case closed-book probe, revealing the issue identifier did not improve function coverage: it was 17.3% with the identifier and 21.4% without it. Allowing repository access raised coverage to 45.5%. This supports the view that code inspection mattered, while not proving that pretraining exposure was absent.[1][2]
Stability is a practical safety problem
A diagnosis tool should return similar evidence when asked the same question twice. The leading solution was flaky on 17% of incidents and on 26% of opaque, no-stack reports. That instability matters because incident response happens under time pressure. If repeated runs identify different components, an engineer must spend additional time deciding which chain of reasoning to trust. A confident narrative is not a substitute for a reproducible link from observed symptoms to the relevant code and state transition.
For engineering teams, the responsible use is as a search assistant rather than an autonomous incident commander. An agent can collect candidate files, connect log messages to code and propose hypotheses. A human can then compare those hypotheses with telemetry, deployment history and system invariants. The benchmark does not test whether an agent can execute a rollback, edit production code, coordinate teams or recognise when evidence is too weak. It also does not measure the cost of a false diagnosis, which can be much higher than the cost of asking an engineer to inspect an extra file.[1]
What this means for developers and incident teams
The result supports a tiered workflow. Low-risk assistance can begin with repository navigation, retrieval of related incidents and a cited list of candidate blocks. Higher-risk actions should require explicit evidence: the triggering request, relevant logs, the code path, the state that violated an invariant and a test that reproduces the failure. Teams can record whether the agent's first hypothesis was correct, how often it omitted a necessary component and whether it delayed or accelerated recovery. Those measures are more useful than a generic claim that an agent 'solved' an incident.
Tool builders should expose uncertainty and provenance at block level. A diagnosis that says only 'the storage module is responsible' is hard to audit. A better system links each claim to code, trace evidence and alternative explanations. It should also preserve the original evidence so an engineer can reproduce the reasoning. The benchmark's large gap between file discovery and precise block identification shows why broad localisation scores can flatter a product. Buyers evaluating incident-response agents should ask for exact-location precision, repeatability and performance on reports without clean stack traces.[1]
What remains unproven—and what would change the assessment
DistDebug-Bench is an unreviewed arXiv manuscript using curated historical open-source incidents. Historical fixes offer a defensible reference, but a commit may contain several reasonable intervention points, and an alternative repair might prevent the same failure. Some difficult categories contain only a handful of cases. The evaluated systems and model versions will also change. The benchmark does not establish performance on proprietary code, newly emerging failures, security incidents, live operational constraints or repositories whose documentation and test coverage are poor.
Confidence would rise with independent replication, larger cross-component subsets and prospective trials in which teams record diagnoses before the real cause is known. Such trials should measure time to a verified cause, false leads, omitted components, recurrence after repair and harm from incorrect actions. Evaluation should include repeated runs and controlled access to logs, traces and deployment metadata. Until then, the evidence supports a limited claim: agents can help search a complex codebase, but exact, stable and operationally safe root-cause diagnosis remains an open problem.[1][2]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring ai agents: compare the finding and its evidence limits.
Can an LLM selector make coding agents more reliable? Gains came with up to 8.7× the tokens
On 89 terminal tasks, generating four or eight candidate actions at every turn and letting a stronger model choose improved reported accuracy by 5–10 percentage points. Executed trajectories shortened, but total token use rose sharply and the preprint has not been peer reviewed.
Read the next reportExplore the full source and evidence record
What this means for people
- On-call engineers may gain a faster shortlist, but still need to verify the exact causal path before acting.
- Teams deploying agentic incident tools need repeatability and evidence links, not only a plausible explanation.
- A wrong autonomous repair could extend an outage or hide the real fault, an outcome this benchmark does not test.
Global context
Modern services depend on distributed databases, queues and cloud components whose failures can cross organisational and geographic boundaries. The benchmark spans multiple languages and projects, but open-source repositories are not representative of every bank, hospital, government service or industrial control system.
What the evidence does not yet show
- Unreviewed preprint based on historical open-source incidents rather than prospective live deployments.
- Some hard subgroups are small, including only seven cross-component opaque incidents.
- Reference fixes may not capture every plausible causal intervention point.
- The benchmark tests diagnosis, not autonomous repair, recovery time, security or operational harm.
What to watch next
- Independent replication on unseen repositories and newer incidents.
- Prospective trials measuring time to verified root cause and false leads.
- Repeatability across identical runs and model updates.
- Evidence-linked diagnosis that separates candidate retrieval from authorised remediation.
Living evidence record
Impact record IAI-0FUHM7G
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
2 direct sources across 2 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Whether coding agents can identify the root-cause code blocks behind real historical distributed-system failures when incident reports are incomplete
Topics: Software engineering · AI agents · Distributed systems · Incident response · Benchmarks · Reliability
Regions: Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.