Can coding agents build trustworthy charts? VisCodeBench finds data and design errors remain
Seven agent-and-model configurations were tested on 2,672 visualisation tasks. The strongest systems completed many requests but still lost accuracy on difficult charts, and the benchmark does not measure whether people make better decisions from the output.
The short version
- What happened
- VisCodeBench evaluates seven agent-and-model configurations on 2,672 tasks from GitHub coursework, scientific figures and Our World in Data charts.
- Who it affects
- Analysts can gain a fast chart draft, but remain responsible for validating the values, denominator and visual encoding.
- What remains unproven
- Static, single-turn benchmark rather than an interactive production workflow.
Keep in mind
Static, single-turn benchmark rather than an interactive production workflow.
Only seven system configurations were tested, and model versions will change.
The scientific subset had a 2.69% response rate and is self-selected.
The benchmark measures chart outputs, not reader comprehension, decisions or downstream harm.
Source analysis · 2 linked sources. Primary analysed source date: 2 November 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The direct answer: capable systems still need a data check and a design check
Coding agents can now turn a natural-language request and a dataset into many plausible charts, but a new benchmark shows that completion is not the same as correctness. VisCodeBench contains 2,672 text-to-visualisation tasks across three settings: 843 GitHub course-assignment notebooks, 805 scientific visualisations and 1,024 charts derived from Our World in Data. The researchers tested seven combinations built around Codex, Claude Code and LIDA with open and commercial language models, then scored the output on seven dimensions.
The strongest systems completed most requests and often selected an appropriate chart, yet none was uniformly reliable. The best displayed-data similarity was 0.81 and fell to 0.69 on the hardest Our World in Data subset. That metric checks whether plotted values resemble the intended data, not whether a human reader notices a misleading scale or draws the right conclusion. The practical answer is therefore conditional: an agent can produce a useful first draft, but someone still needs to verify the data transformation, encodings, labels, accessibility and fit to the reader's question.[1]
A benchmark with three different kinds of chart work
The three subsets test different demands. GitHub-T2V uses 843 course tasks where a notebook and assignment prompt establish the expected output. ArXiv-T2V uses 805 scientific figures contributed by researchers who supplied data, code and natural-language instructions. OWID-T2V uses 1,024 public-facing chart tasks derived from Our World in Data. Together they cover classroom exercises, specialist research figures and explanatory charts intended for a wider audience rather than one narrow plotting library or domain.
The scientific subset has an important denominator. About 30,000 corresponding authors were invited; 1,151 valid submissions arrived from 848 participants, a response rate of 2.69%, and filtering produced 805 benchmark tasks. That creates a rare set of researcher-linked requests, but also a self-selected sample. The people who responded may be more comfortable sharing code and data than the broader research population. The dataset should not be read as a representative survey of how all scientists describe visualisation needs.[1]
The distinction that matters
One model, two agent frameworks, different results
On the OWID subset, the surrounding workflow changed performance even when GPT-5 was the shared model.
Codex + GPT-5
Task completion
100%
LIDA + GPT-5
Task completion
79%
Codex + GPT-5
Chart correctness
73.67%
LIDA + GPT-5
Chart correctness
54.37%
What the seven measures reveal
The evaluation separates task completion and latency from the quality of the chart. It checks chart type, similarity of displayed data, chart correctness, readability and accessibility. This matters because a system can run without error while plotting the wrong column, aggregating at the wrong level or hiding information behind poor contrast and labels. Conversely, a visually plain chart can be correct and useful. A single success rate would collapse these different failure modes and make it difficult for an analyst to decide what needs checking.
Claude Code paired with Opus-4.7 was about five chart-correctness points ahead of Codex paired with GPT-5 and roughly 20 seconds faster in the authors' tests. Yet the framework itself also mattered. With GPT-5 as the shared model, Codex achieved 100% task completion on the OWID subset compared with 79% for LIDA, while chart correctness was 73.67% versus 54.37%. Those comparisons show that model branding alone does not predict the output: tool access, execution, iteration and error handling shape performance.[1]
How natural are the requests—and how independent is the evidence?
The benchmark is more realistic than synthetic prompts alone, but it is not a direct capture of every user's first words. For the arXiv and OWID subsets, some initial requests were generated with GPT-5 and then revised by native-English-speaking experts with more than four years of programming experience. The authors release the original researcher-written arXiv requests separately. Expert revision improves clarity and consistency, but it may remove ambiguity that real analysts and students would bring to an agent.
The manuscript says the VisCodeBench paper was accepted to COLM 2026. It received Australian Research Council DECRA and CSIRO support and an unrestricted gift from Google. The University of Sydney approved the human-participant work under reference 2024/HE000844, and participants gave informed consent. Those disclosures do not invalidate the study, but readers should know them. The repository is maintained by the same research team, so it supports reproducibility rather than independent confirmation.[1][2]
What analysts, teachers and researchers can do with the result
For an analyst, the safest workflow is to treat the generated chart as executable reasoning that must be reviewed. First confirm the requested denominator, filters and aggregation. Then compare plotted values with a small table or independent calculation. Check axes, units, ordering, missing values and uncertainty. Finally, test text alternatives, colour contrast and whether the chart works without colour alone. A passing notebook run only establishes that code executed; it does not establish that the visual claim is accurate or accessible.
Teachers can use generated visualisations to expose reasoning rather than to bypass it. Asking students to identify the data transformation, defend the chosen chart type and audit an agent-produced result may reveal more than marking a polished image. Researchers should retain code, environment information and source data so a figure can be reproduced. Public-service teams need an additional editorial review because a chart about health, employment or climate can influence decisions even when the underlying error is subtle. The benchmark supports these checks; it does not quantify their real-world effectiveness.[1]
What remains unproven—and what would change the assessment
VisCodeBench is a static, single-turn benchmark of seven system configurations. The full dataset is kept private to reduce training contamination, which protects future evaluation but limits immediate independent inspection of every item. The systems took roughly 39 to 78 seconds in the reported runs, and costs varied substantially, but performance and pricing will change with new models. Results do not establish reliability in a collaborative workflow where a user clarifies the question, revises a chart or catches an error.
Most importantly, the benchmark evaluates outputs rather than human outcomes. It does not test whether readers understand the chart, whether a decision improves, or whether an analyst notices a wrong value. Confidence would increase with blinded replication, multilingual and accessibility-focused tasks, adversarial datasets, and prospective studies in which people use the tools under realistic deadlines. Researchers should measure error detection, time saved, correction effort and the consequences of missed errors. For now, the evidence supports assisted chart production with verification—not unattended publication of visual claims.[1][2]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring benchmarks: compare the finding and its evidence limits.
Can real tool traces train better AI agents? ToolSpace raises benchmark scores, not proof of safe work
The unreviewed preprint turns observed tool-call dependencies into replayable training tasks. A 35-billion-parameter model rose from 23.0 to 41.0 across seven benchmarks, but the evidence remains synthetic and benchmark-bound.
Read the next reportExplore the full source and evidence record
What this means for people
- Analysts can gain a fast chart draft, but remain responsible for validating the values, denominator and visual encoding.
- Teachers can use agent output as material for critique rather than accepting a polished chart as evidence of understanding.
- Readers may act on a plausible but incorrect graphic, so accessibility and data checks must happen before publication.
Global context
Charts mediate research, journalism, classrooms and public policy across languages and domains. The benchmark spans educational, scientific and public-data tasks, but its expert-revised English prompts do not represent every user, language, disability or low-resource setting.
What the evidence does not yet show
- Static, single-turn benchmark rather than an interactive production workflow.
- Only seven system configurations were tested, and model versions will change.
- The scientific subset had a 2.69% response rate and is self-selected.
- The benchmark measures chart outputs, not reader comprehension, decisions or downstream harm.
What to watch next
- Independent evaluation on the withheld tasks and new contamination-resistant sets.
- Multilingual, screen-reader and colour-vision accessibility testing.
- Interactive studies that measure correction effort and error detection.
- Whether production tools expose transformations and provenance before a chart is published.
Living evidence record
Impact record IAI-12AAHYR
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
2 direct sources across 2 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
How reliably coding agents turn natural-language requests and datasets into correct, readable and accessible visualisations across scientific, educational and public-data tasks
Topics: Data visualisation · Coding agents · Research methods · Accessibility · Benchmarks · Charts
Regions: Australia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.