Can real tool traces train better AI agents? ToolSpace raises benchmark scores, not proof of safe work
The unreviewed preprint turns observed tool-call dependencies into replayable training tasks. A 35-billion-parameter model rose from 23.0 to 41.0 across seven benchmarks, but the evidence remains synthetic and benchmark-bound.
The short version
- What happened
- ToolSpace derives data, state and retry links from adjacent calls in tool-use traces, clusters 2,042 canonical tools into 804 environments and keeps generated tasks only after replay checks.
- Who it affects
- Workers may encounter agents that complete longer software workflows, but the paper does not measure workload, job quality, supervision time or mistakes affecting customers.
- What remains unproven
- Unreviewed preprint and author-run evaluation with no independent replication reported.
Keep in mind
Unreviewed preprint and author-run evaluation with no independent replication reported.
Synthesized environments and reverse-generated tasks do not reproduce the full ambiguity, permissions or consequences of live organisations.
The seven-benchmark average combines different native metrics and is not a universal task-success rate.
The 55.3% figure counts synthesized files, not compute, time, money or energy.
No funding or competing-interest statement was located in the manuscript text reviewed for this report.
Source analysis · 2 linked sources. Primary analysed source date: 3 November 2026.
Read the full report · 9 min The evidence, methods, implications and limitations
The direct answer: the method improves benchmark agents, but the real-world claim stops there
A new preprint reports that agents trained on tasks reconstructed from observed software-tool traces perform substantially better on seven long-horizon benchmarks. The clearest comparison uses Qwen3.6-35B-A3B. Its average score across workflow automation, skill composition, professional tasks, banking, function calling, legal workflows and office questions was 23.0 before ToolSpace training. Supervised fine-tuning on ToolSpace data raised the average to 41.0, and a further reinforcement-learning stage raised it to 43.2. The authors also trained a much larger GLM-5.2-based model, whose average rose from 44.6 to 56.6.
Those are large benchmark gains, not evidence that the same agents can safely complete unattended work. Each benchmark has its own score, so the seven-task average combines different measures rather than one universal success rate. The environments are reconstructed or simulated, the tasks are reverse-generated from successful tool chains and the evaluations were run by the research team. The responsible conclusion is specific: training on execution-linked tool compositions appears promising for benchmarked orchestration. Whether it reduces errors, supervision time or harm in a bank, clinic, office or public service remains untested.[1][2]
What ToolSpace actually builds from tool-use traces
The pipeline starts from recorded sequences of tool calls rather than inventing a database and then designing tools around it. The source inventory includes Toucan, which the paper describes as 495 Model Context Protocol servers, more than 2,000 tools and about 1.5 million trajectories; ToolBench and ToolLLM, with 16,464 APIs and roughly 469,000 API calls; and execution traces from coding-agent systems such as SWE-agent and OpenHands. Lower-confidence synthetic trajectories are also present, filtered against real tool names and schemas. From this material the researchers identify 2,042 canonical tools and 77,760 recurring tool-call patterns.
Adjacent calls become weighted links of three kinds. Data links mean an earlier output supplies an input to the next call; these account for 69% of reported edges. State links mean one call changes an object that a later call reads or updates; they account for 25%. Retry links capture a failed call followed by a successful alternative and make up 6%. Clustering the graph produces 804 environments. Only the 181 environments containing a state-changing operation receive a synthesized database. Query tools can restore recorded snapshots, while stateless tools replay cached returns. That design reduces the amount of fabricated state, although it still does not recreate a full organisation or its messy operating conditions.[1][2]
The distinction that matters
Reported average across seven long-horizon benchmarks
The table separates backbone scores from ToolSpace training. Each benchmark uses its own native metric, so averages are comparative summaries rather than a universal task-success percentage.
Qwen3.6-35B-A3B
Backbone
23.0 average
ToolSpace-35B-A3B
Supervised fine-tuning
41.0 average
ToolSpace-35B-A3B + RL
Further reinforcement learning
43.2 average
GLM-5.2 / ToolSpace-744B
Backbone → supervised fine-tuning
44.6 → 56.6
How replayable chains become training tasks
ToolSpace samples routes through the graph using a state-aware random walk. A proposed next call is allowed only when its inputs can be filled from earlier outputs, the current state or the task. More frequently observed links receive more weight. Before each call the system checkpoints the environment. If a call fails or creates a state inconsistent with the expected relationship, the system rolls back and tries another branch. The paper's worked demonstration contains 28 calls across six environments and two rollbacks. Retained chains are replayed from their initial state so that the same tools remain executable in sequence.
A language model then writes a natural-language goal from the successful chain without naming the tools or prescribing their order. Verification checks final state, important intermediate states and required output fields. A sample survives only if its chain can run again from the original snapshot and pass those assertions. The resulting corpus contains 34,376 supervised fine-tuning samples and 653 reinforcement-learning samples. The authors trained the 8-billion- and 35-billion-parameter models for one epoch on 16 Nvidia GPUs, while the 744-billion-parameter model used 256 GPUs. The paper says local ToolSpace results were averaged over three independent runs, a useful check on run-to-run noise but not an independent laboratory replication.[1]
What the comparisons show—and what they do not
The gains are not confined to one task. The 35-billion-parameter ToolSpace model improved over its Qwen backbone on all seven reported benchmarks. On AutomationBench the score rose from 8.2 to 45.8; on JobBench, from 6.4 to 40.0; on the banking benchmark, from 9.3 to 34.0 before reinforcement learning; and on the Berkeley Function Calling Leaderboard V4, from 60.0 to 74.3. Results still varied sharply by domain. Harvey LAB, a legal-workflow benchmark, moved only from 0.8 to 1.7 for the supervised 35-billion model, showing that a higher average can conceal a near-zero score on a demanding use case.
The authors also created harder versions of AutomationBench and the banking benchmark. The paper reports 300 and 98 retained tasks respectively after replay, deduplication and review by three doctoral-level experts. At least two experts had to approve each candidate; reported agreement was 93.1%, with a Fleiss kappa of 0.82. Seven frontier models lost between 11.3 and 38.2 points on AutomationBench Pro and between 14.4 and 25.6 points on the banking Pro set. This establishes that the new tasks are harder for the tested models. It does not establish that they are more representative of actual work, or that a low score predicts operational harm.[1][2]
Ablations support the mechanism, with an important cost caveat
The ablation results make the proposed mechanism more credible. On five benchmarks, removing state edges cut the 35-billion model's scores from 45.8, 42.4, 74.3, 34.0 and 48.9 to 36.5, 38.0, 70.1, 21.5 and 44.0. Removing data edges produced larger losses on several tasks, including a fall from 74.3 to 62.4 on the function-calling benchmark. Disabling rollback or replacing the weighted walk with uniform sampling also reduced performance. These comparisons hold the broad training setup closer to constant than comparing ToolSpace with unrelated systems, although they still come from the authors' own implementation.
The paper and repository describe a 55.3% reduction in synthesized files over 500 sampled environments: 2,724 files for a data-centric alternative versus 1,218 for ToolSpace. That is explicitly a proxy for synthesis work, not a measured reduction in money, runtime, electricity, API calls or engineering labour. The 744-billion-parameter training run used 256 GPUs, but the paper does not provide a complete energy or cost accounting. Buyers and research teams should therefore resist converting the file-count comparison into a claim that ToolSpace is 55.3% cheaper to train or operate.[1][2]
What this means for workers, developers and organisations
For developers of workplace agents, the practical insight is that tool ordering and state transitions deserve their own training data. A model can know how to call a calendar, spreadsheet or customer-record function and still fail when one result must be transformed, checked and carried through many later steps. ToolSpace's improvement on cross-environment tasks suggests that training examples should preserve those dependencies and recovery paths. It also suggests a better evaluation question: not only whether an agent called the right tool, but whether it left the system in the correct state after a long sequence.
For workers and managers, the paper is not a reason to remove review from consequential workflows. The generated tasks begin from successful traces and use machine-written goals; real work includes unclear instructions, conflicting permissions, stale records, adversarial content and exceptions that are absent from the data. A benchmark can verify a simulated final state while missing whether the action was authorised, fair or understandable to the person affected. A sensible deployment would start with reversible, low-impact tasks, log each state change, require approval for external communications or financial actions, and measure correction effort as well as completion. The study does not provide those safeguards or outcome measures.[1][2]
What remains unproven—and what would change the assessment
ToolSpace is an unreviewed arXiv manuscript from researchers at Renmin University of China and Fudan University. No funding or competing-interest statement was located in the manuscript text reviewed for this report. The official repository makes code and benchmark packages available, but it is maintained by the same team and is not independent confirmation. Some training traces are synthetic even though their tool schemas are filtered against real interfaces. Reverse task generation uses Kimi K3, which can shape the language and difficulty of the resulting examples. Public benchmark scores may also have been produced under different model endpoints or evaluation settings.
Confidence would rise with independent reruns from the released code, a clear accounting of trace licences and potential training-test overlap, and prospective evaluations in real organisations. Those studies should compare ToolSpace-trained and baseline agents on the same previously unseen workflows, with identical permissions and human oversight. They should measure successful completion, incorrect state changes, data exposure, recovery after failure, supervisor time, compute cost and the consequences for people whose records are changed. Until such evidence exists, ToolSpace is a substantial training and benchmark result—not proof that long-horizon agents are ready to run consequential work without supervision.[1][2]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring ai agents: compare the finding and its evidence limits.
Can AI agents diagnose real distributed-system failures? A new benchmark finds an evidence gap
Across 120 historical failures, leading agents often named the right file while missing the precise code blocks that caused the incident. The benchmark is an unreviewed preprint and tests diagnosis, not safe autonomous repair.
Read the next reportExplore the full source and evidence record
What this means for people
- Workers may encounter agents that complete longer software workflows, but the paper does not measure workload, job quality, supervision time or mistakes affecting customers.
- Developers gain a reproducible way to test stateful tool orchestration, yet still need permission controls, audit logs and recovery mechanisms for production use.
- People whose records or services are affected need a human appeal and correction route; benchmark completion does not establish consent, fairness or accountability.
Global context
Tool-using agents are being designed for office, banking, software and public-service workflows around the world. The underlying traces span many software interfaces, but the authors are based at two Chinese universities and the evaluation is English-language and benchmark-driven. Results should not be assumed to transfer to every language, legal regime, accessibility need or low-resource organisation.
What the evidence does not yet show
- Unreviewed preprint and author-run evaluation with no independent replication reported.
- Synthesized environments and reverse-generated tasks do not reproduce the full ambiguity, permissions or consequences of live organisations.
- The seven-benchmark average combines different native metrics and is not a universal task-success rate.
- The 55.3% figure counts synthesized files, not compute, time, money or energy.
- No funding or competing-interest statement was located in the manuscript text reviewed for this report.
What to watch next
- Independent reruns using the public code and fixed model endpoints.
- Audits for trace provenance, licensing and training-test overlap.
- Prospective workplace trials measuring wrong state changes, supervisor effort and recovery, not only completion.
- Actual compute, energy and monetary costs across model sizes.
- Safety evaluations with permissions, adversarial inputs and irreversible actions.
Living evidence record
Impact record IAI-1D9MHRA
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
2 direct sources across 2 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Whether training data built from observed dependencies between real software tools can improve long-horizon agent performance across several simulated work benchmarks
Topics: AI agents · Tool use · Training data · Benchmarks · Workplace automation · Research methods
Regions: China · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.