Can an LLM selector make coding agents more reliable? Gains came with up to 8.7× the tokens
On 89 terminal tasks, generating four or eight candidate actions at every turn and letting a stronger model choose improved reported accuracy by 5–10 percentage points. Executed trajectories shortened, but total token use rose sharply and the preprint has not been peer reviewed.
The short version
- What happened
- The unreviewed study used 89 Terminal-Bench v2.1 tasks, two generator models, two selectors, best-of-four and best-of-eight settings, and four trials per task.
- Who it affects
- Software teams may get more reliable benchmark completion by spending substantially more inference at each decision.
- What remains unproven
- Unreviewed preprint on one 89-task terminal benchmark.
Keep in mind
Unreviewed preprint on one 89-task terminal benchmark.
Selectors were more capable than generators, complicating attribution.
No wall-clock, financial, energy, security or human-quality outcomes.
Behaviour analysis used an LLM judge on a subset rather than independent human adjudication.
Source analysis · 1 linked source. Primary analysed source date: 4 November 2026.
Read the full report · 8 min The evidence, methods, implications and limitations
The finding: choosing among several next steps helped, but it was not free
An MBZUAI Labs preprint reports that a coding agent became more accurate and more consistent when it generated several possible actions at every turn and delegated the choice to a stronger language model. On Terminal-Bench v2.1, the paper says turn-level best-of-four and best-of-eight selection improved task accuracy by about five to ten percentage points over the single-sample baseline. Executed trajectories also used roughly 40% to 50% fewer turns. The result suggests that some agent failures arise from a poor local choice rather than an inability to propose a useful action at all.
The efficiency claim needs its denominator. The agent executed fewer shell or editing steps, but it generated four or eight candidate actions before each selected step and paid for an additional selector prompt. Across the reported configurations, total token use rose to approximately 3.6 to 8.7 times the baseline. The experiment therefore trades parallel inference for shorter visible trajectories. That may be worthwhile when a failed task is expensive or candidate generation can run in parallel, but it is not evidence of lower operating cost, faster completion or lower energy use. Those outcomes were not measured.[1]
How turn-level best-of-N changes an agent loop
A conventional agent observes the current terminal state, generates one action, executes it and then observes the result. In the proposed loop, the generator produces N candidate actions from the same state. A separate selector sees the candidates and chooses one before any of them is executed. Only the chosen action changes the environment. The paper tested N equal to four and eight, with N equal to one as the baseline and an exploratory N equal to two that performed roughly like baseline. Selection was listwise: the selector compared the complete candidate set rather than scoring each option independently.
This architecture resembles inference-time scaling in reasoning systems, but the location of the extra computation matters. It is spent repeatedly at every interactive turn rather than once on a final answer. That lets the selector reject an early command likely to send the trajectory down a dead end, yet it also multiplies generation throughout a long task. The authors used Qwen3.6-35B and K2-Horizon-375B as generators, with Kimi-K3 and DeepSeek-v4.1-Flash as selectors. The selector was deliberately more capable than the generator, so the paper does not show that any extra model can provide the same benefit.[1]
The distinction that matters
Fewer actions did not mean less computation
The selector shortened the executed path by evaluating many actions that were never executed.
Observe
Read the current terminal state
One shared context
Generate
Propose 4 or 8 next actions
Inference multiplies
Select
A stronger model chooses one
Only one action executes
Trade-off
Repeat every turn
5–10 point gains, 3.6–8.7× tokens
The benchmark used 89 terminal tasks and four trials per task
The evaluation used Terminal-Bench v2.1, a set of 89 tasks performed in terminal environments, with mini-SWE-agent as the agent scaffold. Tasks can require inspecting files, writing or repairing code, running commands and satisfying an automated verifier. Each model and sampling configuration was run four times per task. Repetition allows the authors to examine both average task accuracy and consistency across runs. It does not produce 356 independent task types: the underlying challenge set remains 89, and repeated trials may share the same benchmark artefacts and scoring assumptions.
Terminal-Bench is useful because actions change state and mistakes can compound, making it more representative of agent operation than a static multiple-choice test. It is still one benchmark centred on software and terminal work. The paper does not cover office documents, web research, customer service, financial transactions, medical workflows or robots. It also does not evaluate whether a human engineer would accept the resulting patch, how often a technically passing solution is maintainable, or whether agents introduce security weaknesses. General claims about autonomous work should therefore remain separate from the measured terminal-task result.[1]
Shorter executed trajectories concealed much larger inference budgets
For the K2-Horizon generator, the baseline averaged 69.5 executed turns and about 72,000 tokens. With Kimi-K3 selecting four candidates, executed turns fell to 41.0, while the reported generator and selector totals were about 196,000 and 67,000 tokens—roughly 3.6 times baseline in all. At best-of-eight, turns fell to 33.9, but generation and selection rose to about 334,000 and 69,000 tokens, or 5.6 times baseline. DeepSeek selection produced a similar trajectory reduction and total-token multipliers of about 3.9 and 6.3 for N equal to four and eight.
The smaller Qwen generator started at 64.9 turns and about 50,000 tokens. Kimi selection reduced executed turns to 40.5 at best-of-four and 34.0 at best-of-eight, while total tokens reached roughly 5.2 and 8.2 times baseline. With DeepSeek as selector, the corresponding totals were about 5.7 and 8.7 times baseline. These are benchmark-level averages reported by the authors, not a price quotation. Provider pricing, batching, caching, parallelism and hardware utilisation would change financial and latency outcomes. The study did not convert tokens into money, wall-clock time or energy.[1]
Behaviour analysis suggests fewer repeated mistakes, with limited evidence
To understand the improvement, the authors examined 36 hard tasks and 360 trajectories with a language-model judge. They report that selection reduced unproductive repetition and made agents more likely to recover from an earlier mistake or choose a promising diagnostic action. This is consistent with the mechanism: when several alternatives are visible, the selector can prefer an information-gathering command or avoid repeating a failed approach. Fewer executed turns can therefore reflect better local decisions rather than simply more concise text. The analysis is descriptive and based on a subset of the benchmark, not an independently blinded human error study.
The design also leaves an important comparison unresolved. Because the selector is stronger, some of the gain may come from giving a better model control over each decision rather than from best-of-N diversity itself. The paper does not report the stronger selector acting as the generator under the same full harness and budget, nor does it test a selector weaker than the generator. Best-of-two was approximately baseline, suggesting that candidate count alone is not a smooth guarantee of improvement. Model-specific candidate diversity, calibration and the selector's ability to recognise good actions may all determine whether the method helps.[1]
What engineering teams can test—and what would change the assessment
For teams operating agents, the study offers a concrete design option for high-value steps: generate alternatives before an irreversible action and use an independent chooser. It may be more sensible to apply that selectively around destructive commands, uncertain patches or repeated failures than to multiply every turn. Operators should measure end-to-end completion, wall-clock latency, total tokens, monetary cost, security defects and human review time together. A shorter trace is not automatically a cheaper or safer trace, and a benchmark pass is not equivalent to a production-quality change.
Confidence would rise with preregistered tests on additional interactive benchmarks, direct comparisons against spending the same compute on a larger single agent, human review of patch quality, and measurements of latency, energy and cost. Ablations should test weak and equal-capability selectors, pairwise versus listwise choice, adaptive N, and selective use only when uncertainty is high. Field studies should ask whether developers resolve real tickets faster without accepting more defects. The accessible preprint reports no funding, acknowledgement or competing-interest statement and has not been peer reviewed, so independent reproduction is especially important.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring coding agents: compare the finding and its evidence limits.
Which coding agent works best when a bug report unfolds over several turns?
In an unreviewed 72-task benchmark, the one-turn leader was not the eight-turn leader. The result suggests that coding-agent rankings depend on how long users can clarify a problem—but the users were simulated and the benchmark was built by Tencent researchers.
Read the next reportExplore the full source and evidence record
What this means for people
- Software teams may get more reliable benchmark completion by spending substantially more inference at each decision.
- Shorter visible trajectories can simplify review, but the discarded candidates still consume compute.
- High-stakes actions may justify selective candidate comparison more readily than routine low-value steps.
Global context
Agent developers are increasingly shifting compute from model training to inference-time search and verification. Turn-level selection applies that trend inside an interactive loop, where early mistakes compound. The paper shows a plausible reliability gain but also makes the resource trade-off unusually visible: executing fewer actions can still require far more computation.
What the evidence does not yet show
- Unreviewed preprint on one 89-task terminal benchmark.
- Selectors were more capable than generators, complicating attribution.
- No wall-clock, financial, energy, security or human-quality outcomes.
- Behaviour analysis used an LLM judge on a subset rather than independent human adjudication.
What to watch next
- Equal-budget comparisons with stronger single agents.
- Adaptive sampling only at uncertain turns.
- Wall-clock and energy measurements.
- Human review of patch quality and security.
Living evidence record
Impact record IAI-04S3L7G
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Whether generating several candidate actions at each agent turn and selecting among them with a stronger model improves terminal-task accuracy and consistency
Topics: AI agents · Coding agents · Inference scaling · Reliability · Compute costs · Benchmarks
Regions: United Arab Emirates · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.