Can workplace AI agents follow a project's real history? Most SlackBench systems scored below 66%
An unreviewed benchmark asks agents to reconcile conversations, code and changing decisions across 140 questions. The best tested system averaged 80.25%, while false premises, incomplete searches and interface design remained consequential failure points.
The short version
- What happened
- SlackBench contains 140 questions over 15 reconstructed academic projects, about 3,000 messages, 14 GitHub repositories and 9 Google Docs; 35.0% of messages were human-written and 35.2% substantially human-rewritten.
- Who it affects
- Workers could spend less time reconstructing decisions, but a wrong or untraceable answer can create rework and unfair judgments.
- What remains unproven
- Unreviewed preprint and author-run benchmark without independent replication reported.
Keep in mind
Unreviewed preprint and author-run benchmark without independent replication reported.
Fifteen reconstructed academic projects do not represent every organisation, language or consequential workflow.
The human comparison included only three researchers who could use AI tools.
LLM judging was validated on selected outputs but was not identical to independent human scoring of every run.
The paper evaluates information retrieval and synthesis, not autonomous workplace actions or long-term effects on workers.
Source analysis · 1 linked source. Primary analysed source date: 3 November 2026.
Read the full report · 10 min The evidence, methods, implications and limitations
The answer: agents could reconstruct some project history, but none made verification optional
A new benchmark found that frontier AI agents could often answer questions whose evidence was scattered across team messages, code and documents, but performance depended heavily on the model, agent harness and route into the workspace. SlackBench evaluated 33 combinations on 140 questions. The highest overall score was 80.25% for Antigravity with Gemini 4 Argon. Only five combinations reached 70%, and most were below 66%. Those results do not mean every unanswered question would become a workplace error: the score is an unweighted average across four task categories, and each question was run three times. They do show why a polished agent response should not be treated as a verified project record.
The benchmark targets a practical gap in ordinary assistants. A decision may begin in one channel, change after an experiment, and become real only in a repository commit. A teammate asking what was finally shipped needs the agent to find the right conversation, follow its chronology, inspect the corresponding code and disregard an abandoned plan. SlackBench turns that work into measurable questions about search, abstention, summarisation and inference. Its central contribution is not a new model leaderboard alone; it is a test of whether an agent can assemble an answer from heterogeneous evidence without flattening all mentions into one timeless document.[1]
The workspace is reconstructed from real research activity, not a released company archive
Researchers created 15 projects in a live Slack workspace and connected them to 14 GitHub repositories and 9 Google Docs. Humans led projects while coding agents acted as research assistants, reporting progress and changing code. The export contains about 3,000 messages. The paper says 35.0% were written directly by humans and another 35.2% were substantially rewritten by humans. Staff then removed artificial-sounding language, replaced credentials with fake values and checked claims against the repositories. This human-in-the-loop reconstruction preserves chronology and artifacts without publishing private workplace conversations.
That design is more grounded than a set of isolated fictional emails, but it is still a simulation. The projects are academic research exercises, not hiring, medical care, payroll, procurement or public administration. Some conversations were produced by agents under prompts that shaped how they communicated. The team also edited the history before release. A benchmark can therefore test complex navigation without capturing every ambiguity, power imbalance, privacy duty or consequence found in a real organisation. It measures whether agents recover the authors' curated reference answer, not whether workers would trust the process or whether an answer is legally safe to act on.[1]
The distinction that matters
What SlackBench asks an agent to reconstruct
Each task may require several records, and a high score still depends on the access method and grading rubric.
Search
Locate messages, files and commits
58 questions
Abstain
Reject unknown or false premises
48 questions
Summarise
Combine evidence toward a goal
14 questions
Infer
Read distributed behaviour patterns
20 questions
Four task groups expose different kinds of workplace failure
The 140 questions are divided into 58 agentic-search tasks, 48 hallucination or abstention tasks, 14 goal-directed summaries and 20 circumstance-inference tasks. Search questions may require following both a conversation timeline and a repository history. Abstention questions ask about an unknown result, embed a false premise or impose a time limit that excludes later evidence. A correct answer must refuse the unsupported request and identify why. Summaries require gathering contributions across projects. Circumstance tasks distribute behavioural clues through long conversations, such as identifying a person who contributed little rather than matching one keyword.
This separation matters because a system that retrieves facts well can still mishandle uncertainty. In the authors' analysis, GPT-based agents sometimes answered a different question before acknowledging that the user's premise was wrong. The reported pattern appeared on 25% of the hallucination tasks for GPT 6 Astra and 21% for GPT 6.1 Sol, compared with 8% for Opus 5.5 and 6% for Muse Wonder when used with Claude Code. Those percentages belong to this benchmark and these harnesses; they are not provider-wide hallucination rates. The operational lesson is narrower: correction at the end of an answer may be too late if the first sentence already authorises an action.[1]
Access design changed scores, cost and the path an agent took
The researchers compared access through their Slack-like MCP tools with direct command-line access to exported workspace files. Across the tested pairings, file access improved average pass rates by 7.85 points. Codex CLI with GPT-5.5 rose from 45.52% overall with the paper's MCP configuration to 59.90% with files; its agentic-search score rose from 55.17% to 72.42%. Claude Code with Opus 5.5 changed only slightly overall, from 72.65% to 73.68%, while the balance across task types moved. The authors later tested the official Slack MCP on a filtered subset and found it outperformed their mock implementation, so the comparison is not evidence that MCP itself is inherently inferior.
The trajectory counts explain part of the difference. File-based agents spent more actions reading full context, while MCP agents used more message searches and incurred metadata and tool-schema overhead. The authors report a 54% cost reduction for one Claude Code and Opus 5.5 comparison and a 21% average token reduction for the other tested agents. Cost for the full suite ranged from under $1 to above $500 depending on model and harness. These are experiment-specific API costs, not a general procurement estimate. They demonstrate that an organisation evaluating workplace AI must test the complete access architecture, permissions and prompts, not infer reliability or cost from the base model alone.[1]
The human comparison offers context, but its denominator is small
Three researchers who had not built SlackBench completed the same tasks and could use AI tools. Their average score was 84.5%: 91% on search, 90% on abstention, 82% on summaries and 75% on circumstance inference. The paper gives a 95% bootstrap interval of 77.0% to 92.0%. The best agent's 80.25% falls inside that wide interval, so the result should not be simplified into a definitive human-versus-machine ranking. With only three people, the comparison cannot represent differences in experience, accessibility, fatigue or knowledge of team conventions.
Answers were primarily scored by GPT-5.4 using reference answers and rubrics. The team manually checked all 264 judgments covering one agent's 140 questions and reported 95.8% agreement; three different judge models also agreed above 95% on a larger subset. That makes automated scoring more credible than an unchecked judge, but disagreement still exists and the validation is not independent of the benchmark authors. Some summary tasks receive partial credit through checklists. A workplace buyer would need its own critical-error categories, because missing one safety-related fact can matter more than losing several points on a general summary rubric.[1]
What this means for workers and teams adopting an agent
For workers, the study supports a design principle: an agent should show the evidence path behind its answer and make uncertainty visible. A useful project assistant would link the decisive message, document version and commit, distinguish a proposal from the final state and say when the available record cannot answer the question. Employers should avoid using circumstance-inference outputs—such as alleged free-riding—as automated performance judgments. In SlackBench those signals were deliberately inserted into a reconstructed history; in real life, missing messages can reflect private work, disability, time zones, caring responsibilities or conversations outside the indexed system.
For technical teams, evaluation should be scenario-specific. Before connecting an assistant to internal chat and repositories, create questions about decisions known to have changed, records with conflicting versions, requests with false premises and cases where abstention is the correct outcome. Score the provenance and the first actionable sentence, not only the final paragraph. Keep access read-only until the system can reliably recover the current state, and require confirmation before it sends messages or changes code. The paper evaluates information work, not autonomous execution, confidentiality controls or the consequences of a wrong workplace decision.[1]
What remains unproven—and what would change the assessment
SlackBench is an unreviewed preprint. Its authors are affiliated with UC Berkeley, the University of Chicago and Bespoke Labs; the project was partly supported by NSF IFML and Bespoke Labs, while Berkeley's Sky Computing Lab lists gifts from many technology and finance companies. The paper's explicit limitations are brief: it tests frontier-lab systems and does not compare prompting strategies. Additional limits follow from the design: English-language reconstructed academic projects, agent-assisted conversations, a 20-minute limit per question, medium reasoning effort for the main table and scoring against author-curated answers. One top result in this benchmark cannot establish reliable performance across employers or languages.
Confidence would rise if independent groups reproduced the runs from the released environment, preregistered their scoring and tested agents on separately built workspaces. Field studies should evaluate teams over time, with permission boundaries, inaccessible private channels, ambiguous ownership and genuine consequences. They should report critical-error rates, provenance quality, corrections, worker time, privacy incidents and whether affected people can challenge an inference. Until then, SlackBench is useful evidence that collaborative history remains difficult—and that the interface around a model can materially alter the result. It is not evidence that an agent is ready to become an unsupervised colleague.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring workplace agents: compare the finding and its evidence limits.
Workplace AI agents struggled when records conflicted—what should teams verify?
Across 1,613 questions over simulated projects, the strongest tested model reached 68% accuracy with 50 retrieved records and remained weak on cross-source and historical conflicts.
Read the next reportExplore the full source and evidence record
What this means for people
- Workers could spend less time reconstructing decisions, but a wrong or untraceable answer can create rework and unfair judgments.
- Teams need visible provenance, version awareness and an abstention path before an agent is trusted with project history.
- Behavioural inference from workplace messages should not become automated performance management without due process and context.
Global context
The tested model providers and research institutions span several countries, but the workspace is English-language and constructed in the United States around academic research. Employment law, workplace monitoring, data protection, accessibility and collaboration norms vary internationally. The transferable finding is that distributed project history challenges agents; the specific leaderboard is not a universal workplace ranking.
What the evidence does not yet show
- Unreviewed preprint and author-run benchmark without independent replication reported.
- Fifteen reconstructed academic projects do not represent every organisation, language or consequential workflow.
- The human comparison included only three researchers who could use AI tools.
- LLM judging was validated on selected outputs but was not identical to independent human scoring of every run.
- The paper evaluates information retrieval and synthesis, not autonomous workplace actions or long-term effects on workers.
What to watch next
- Independent reproduction with released environments and fixed model versions.
- Tests in other languages and non-technical occupations.
- Critical-error and provenance scoring rather than only average pass rates.
- Field studies measuring correction time, privacy incidents and worker appeals.
- Comparisons across access interfaces with equivalent permissions and context.
Living evidence record
Impact record IAI-0CKSFEZ
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
6 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what SlackBench research team published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 6 November 2026
Research question, topics and regions
Whether frontier AI agents can recover project state from long, multi-party communication, repositories and documents, and abstain when a workplace question cannot be answered
Topics: Workplace agents · Collaborative work · Enterprise search · Software engineering · Benchmarks · Human oversight
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.