DeepSeek’s open AI update: what its agent benchmarks show

It beats named rivals on several tests, not all of them

The most striking company-reported results are on agent and software tasks. DeepSeek lists 77.0 on DeepSWE, against 73.0 for GPT-5.6 Sol and 74.0 for Opus 5; 60.5 on Automation-Bench, against 45.8 and 50.3; and 72.1 on Apex Bench, against 54.3 and 65.8. The pattern is not a clean overall win. On Humanity's Last Exam without tools, DeepSeek reports 42.6, below 44.5 for GPT-5.6 Sol and 56.3 for Opus 5. It also trails Opus 5 on Terminal-Bench, ProgramBench, NL2Repo-Bench, Chartography and BabyVision. Each benchmark measures a different task with its own scoring rules, so the rows should not be averaged into a general intelligence claim.[1]

The missing denominator is repeated, independently controlled runs

The model card names the Claude Code harness for Terminal-Bench and Mini-SWE-V2 for DeepSWE, but it does not state how many repeated runs produced the displayed values, show uncertainty intervals, publish complete prompts and trajectories, or demonstrate identical inference budgets across every comparator. Benchmark scores can move with sampling settings, tool permissions, retry rules and harness versions. DeepSeek's September technical report describes the underlying 552-billion-parameter backbone, sparse mixture-of-experts architecture and training on a 45-trillion-token multimodal corpus, but it is a company preprint and predates the 1017 update. It cannot serve as independent validation of the new table.[1][3]

For developers, access is real but operation is not frictionless

The practical change is that organisations can inspect, modify and self-host the released weights under a permissive licence. That can widen experimentation and reduce dependence on a closed API. It does not make deployment free or simple: the repository lists a very large model, recommends a one-million-token context window and at least 256,000 output tokens for some uses, and currently shows no hosted inference provider. Teams still need specialist infrastructure, security controls and their own evaluation on representative tasks. Before procurement or workflow redesign, the useful next evidence would be independent reruns with fixed budgets, published traces, failure analysis and measurements of latency, energy, cost and human correction burden.[1]

What adopting the model would actually involve

Access to the weights is only one stage of making a working system. DeepSeek’s card supplies a Python reference implementation for prompt encoding, with tests for conversation history, tool calls, reasoning settings and images. It also links a maintained toolkit with Rust libraries and Python bindings. The card explicitly leaves model inference, tool execution and HTTP transport to the caller. An organisation therefore needs to assess the surrounding application as well as the model’s benchmark score.

Our interpretation is that a useful adoption test should separate those responsibilities. A model may propose an action while application code decides whether to execute it. Teams can compare representative tasks, the permissions granted, failed attempts and the human work needed to correct an answer. These are evaluation questions, rather than findings established by DeepSeek’s release. The published table provides a starting point for selecting tests; it does not show how a particular employer’s complete system will perform.[1]