Which coding agent works best when a bug report unfolds over several turns?
In an unreviewed 72-task benchmark, the one-turn leader was not the eight-turn leader. The result suggests that coding-agent rankings depend on how long users can clarify a problem—but the users were simulated and the benchmark was built by Tencent researchers.
The short version
- What happened
- Tencent researchers evaluated 11 coding agents on 72 executable repository tasks: 22 two-issue tasks mined from open-source history and 50 tasks adapted from existing one-turn benchmarks.
- Who it affects
- Developers may experience one agent as decisive on a clear ticket and another as better after clarification. Procurement tests should therefore include the organisation's real conversational workflow, not only a one-shot leaderboard.
- What remains unproven
- This is an unreviewed preprint from Tencent Hunyuan researchers, not an independent comparison. No separate funding or competing-interest statement was reported.
Keep in mind
This is an unreviewed preprint from Tencent Hunyuan researchers, not an independent comparison. No separate funding or competing-interest statement was reported.
Only 72 tasks from 42 repositories were used. Fifty were adaptations of existing one-turn benchmarks, and the remaining 22 covered two linked issues rather than long projects.
Users were simulated from aggregate transition patterns in confidential commercial-assistant sessions. Real developers did not interact with the evaluated agents in this study.
Two model families helped judge and review results around executed tests; the adjudication protocol and equivalent-test rewrites can affect rankings.
Source analysis · 1 linked source. Primary analysed source date: 21 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 5 min The evidence, methods, implications and limitations
The answer changes with the interaction budget
A newly posted preprint finds that there is no single coding-agent winner once a bug report becomes a conversation. On 72 repository tasks, Claude Fable 5.1 resolved 31.9% within the first turn, the best opening score. By the eighth turn, GPT-6-Astra had resolved 81.9%, narrowly ahead of Claude Opus 5 and Grok-4.6 at 80.6%. Claude Opus 5 led the paper's length-discounted score because it tended to finish successful tasks sooner.
That distinction matters for teams buying or evaluating coding assistants. A one-shot benchmark rewards an agent that acts quickly on a complete issue description. Real developers often start with an incomplete symptom, answer follow-up questions and correct a mistaken theory. The study's practical finding is therefore about evaluation design: rankings can change when agents get several chances to ask, edit and test—not proof that any named system is best for production software.[1]
How the researchers tried to imitate a developer conversation
The Tencent Hunyuan team assembled 22 tasks from linked issue histories in open-source repositories and converted 50 tasks from existing single-turn benchmarks. Every task included a repository state, a maintainer's patch and tests. The agent received information gradually, while tests ran after each turn. A task counted as resolved only when its required tests passed without breaking previously passing tests; an additional mechanism could recognise a functionally equivalent fix through a rewritten test that still failed on the original code.
The simulated user's wording came from eight persona patterns, such as a clear symptom with no suggested cause or a vague symptom paired with a wrong cause. Transition probabilities were calibrated on 20,859 sessions from an internally deployed commercial assistant; 13,530 of those sessions changed code. The researchers froze each task's information pool and persona sequence so competing agents faced the same trajectory. In a paired 50-task check, three of four selected agents scored better with an unrealistically clear, correct user; GPT-6-Astra's length-discounted score rose from 0.580 to 0.656.[1]
The distinction that matters
Why one leaderboard position is not enough
The same benchmark produces different leaders depending on what the evaluator rewards.
One turn
Measure who fixes the issue immediately
Claude Fable 5.1: 31.9%
Eight turns
Allow repeated clarification, edits and tests
GPT-6-Astra: 81.9%
Fewer turns
Discount successful runs that take longer
Claude Opus 5: highest LDPR, 0.640
Useful signal, but not a field trial
The benchmark also relies partly on AI to judge AI. Executed tests provided the primary evidence, but a Judge model wrote each verdict and a Reviewer from another model family could rewrite it. Reviewers changed 40.9% of scored-turn decisions, which shows that a single model judge would have materially altered results. It also means the leaderboard depends on this particular two-model adjudication design and its rules for equivalent patches.
The 72 tasks cover one maintenance episode, not months of collaborative development, architecture work, security review or deployment. The simulated users reproduce aggregate conversational patterns rather than identifiable people, and the commercial session data cannot independently verify that the personas represent other organisations. The paper is unreviewed and written by the company whose Hunyuan model is among those tested. A public release, independent replication and trials with real developers on longer projects would make the ranking claims more useful.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring coding agents: compare the finding and its evidence limits.
Coding agents finished the job—but did they preserve what they changed?
Six frontier models completed 86.1% of 205 executable tasks yet backed up before the benchmark's regret point only 15.2% of the time. A plug-in layer improved recoverability, but the benchmark and evaluator were built by the same team.
Read the next reportExplore the full source and evidence record
What this means for people
- Developers may experience one agent as decisive on a clear ticket and another as better after clarification. Procurement tests should therefore include the organisation's real conversational workflow, not only a one-shot leaderboard.
Global context
The tested agents span several providers, but repository languages, organisational practices and access constraints vary. Results from this task set should not be transferred directly to a particular team, regulated codebase or security-critical environment.
What the evidence does not yet show
- This is an unreviewed preprint from Tencent Hunyuan researchers, not an independent comparison. No separate funding or competing-interest statement was reported.
- Only 72 tasks from 42 repositories were used. Fifty were adaptations of existing one-turn benchmarks, and the remaining 22 covered two linked issues rather than long projects.
- Users were simulated from aggregate transition patterns in confidential commercial-assistant sessions. Real developers did not interact with the evaluated agents in this study.
- Two model families helped judge and review results around executed tests; the adjudication protocol and equivalent-test rewrites can affect rankings.
What to watch next
- Release of the promised benchmark and evaluation framework
- Independent reruns with the same tasks and frozen persona trajectories
- Studies in which real developers interact with agents
- Longer projects measuring review effort, escaped defects and maintainability
- Sensitivity to different judge models and equivalent-patch rules
Living evidence record
Impact record IAI-0L65C4S
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
22 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 22 October 2026
Research question, topics and regions
Whether coding-agent rankings change when repository bugs are presented through multi-turn conversations with simulated users
Topics: Coding agents · Software engineering · Benchmarks · Human-computer interaction · Developer tools
Regions: China · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.