Can a dashboard make AI compliance evidence easier to read?
In a 10-person study inside BNP Paribas, participants answered more questions correctly with a responsible-AI dashboard than with raw files. Nine were AI researchers, the dashboard always came second and no release decision used the results.
The short version
- What happened
- Ten participants answered six questions from raw evaluation files and six matched questions through VERA; correct answers rose from 27/60 to 42/60.
- Who it affects
- Compliance and risk staff often receive dense logs or a summary slide from engineers. A traceable interface could help them inspect the evidence themselves, but this study does not yet show that the intended reviewers reach better or safer decisions.
- What remains unproven
- Only ten people completed both phases; nine were AI researchers from the institution and there was no positive evidence from the target compliance-and-risk population.
Keep in mind
Only ten people completed both phases; nine were AI researchers from the institution and there was no positive evidence from the target compliance-and-risk population.
The raw-file phase always came first, so condition and order were completely aligned; attrition also left the matched question sets badly imbalanced.
The baseline artifact was small—three files, 119 lines and 12 KB—and the task measured factual reading rather than release decisions, legal compliance or operational outcomes.
The paper is an accepted conference manuscript authored mainly by staff of the bank that built and deployed the tool; it reports no separate funding or competing-interest statement.
Source analysis · 1 linked source. Primary analysed source date: 22 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
A requirement view helped readers find more of the requested facts
BNP Paribas and Dalhousie University researchers built VERA, a self-hosted interface that organises public benchmark results around a technical interpretation of the EU AI Act. Ten people who completed both study phases answered six questions from three raw files and six matched questions in the dashboard. They produced 42 correct answers out of 60 with VERA, compared with 27 out of 60 from the files. The largest gain came when participants had to count verdict bands; a coverage-count question instead favoured the raw log.[1]
The faster result is suggestive, not settled
The median of each participant's median response time fell from 56 seconds per question to 30 seconds. Seven people were faster and three slower, but the exact paired test returned p=.064, so the researchers did not reject the no-difference hypothesis for time. The dashboard also opened directly on the relevant filtered view, while raw-file users had to locate it themselves. That navigation advantage is part of the product's purpose, but this design cannot separate it from the presentation effect.[1]
The distinction that matters
What the study did—and did not test
The paper separates finding facts in evaluation evidence from judging whether an AI system is safe or lawful.
Raw files
Read three tabs holding 119 lines
27 of 60 answers correct
VERA
Use requirement views linked to evidence
42 of 60 answers correct
Release decision
Judge compliance and deployment risk
Not tested
The study does not show that compliance staff make better decisions
Nine of ten completers were AI researchers at the institution, not the compliance and risk staff the tool is meant to support. Everyone used the raw files first, so learning could help the second phase. Attrition also left eight participants on the same question-set assignment, weakening the intended counterbalance. Only five of nine people launched an evaluation unaided on the first attempt. Most importantly, the four model runs reported in the paper did not feed a release decision, and the COMPL-AI scores are technical proxies rather than official legal conclusions.
For banks and other regulated organisations, the useful idea is traceability: a score can open to its tests and model outputs, while an exported record stays tied to the run that produced it. That may make review evidence easier to challenge and reproduce. It does not decide whether a system complies, whether its risks are acceptable or whether the underlying benchmarks cover the real deployment. A counterbalanced study with experienced risk and compliance reviewers performing genuine decision tasks would materially strengthen the claim.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
UNESCO · 11 min 27 sec · Background explainer
UNESCO Talks: Interview with 2024 Nobel laureate Professor Daron Acemoglu
YouTube loads only when you choose; playback starts when you press play.
Economist Daron Acemoglu discusses how choices about AI affect workers, inequality and society. Pair the interview with our comparison of task exposure, hiring and skills evidence.
Evidence check: An expert interview from 2024, offering a perspective on policy and work rather than a measured forecast of job losses.
Published 2024-10-14. Source checked 14 October 2026. Check captions on the publisher’s video page.
Read the AI and jobs guidePhoto: Dylan Gillis / Unsplash · Licence
Continue the story
Read next
Continue exploring ai governance: compare the finding and its evidence limits.
Does AI make finance greener? A 181-paper review finds the outcome evidence missing
Researchers found fast-growing use of machine learning and text analysis in sustainable finance, but technical performance rarely establishes a real environmental or social result.
Read the next reportExplore the full source and evidence record
What this means for people
- Compliance and risk staff often receive dense logs or a summary slide from engineers. A traceable interface could help them inspect the evidence themselves, but this study does not yet show that the intended reviewers reach better or safer decisions.
Global context
The dashboard uses COMPL-AI's public technical interpretation of the EU AI Act. That framework is not EU-funded, is not part of the Act's harmonised standards and is not official conformity-assessment software. Organisations in any jurisdiction would still need local legal review, governance and deployment-specific evidence.
What the evidence does not yet show
- Only ten people completed both phases; nine were AI researchers from the institution and there was no positive evidence from the target compliance-and-risk population.
- The raw-file phase always came first, so condition and order were completely aligned; attrition also left the matched question sets badly imbalanced.
- The baseline artifact was small—three files, 119 lines and 12 KB—and the task measured factual reading rather than release decisions, legal compliance or operational outcomes.
- The paper is an accepted conference manuscript authored mainly by staff of the bank that built and deployed the tool; it reports no separate funding or competing-interest statement.
What to watch next
- A counterbalanced replication with experienced compliance and risk reviewers
- Decision-quality measures rather than fact-retrieval questions
- Evidence that run-tied records change real approval or monitoring outcomes
- Whether the interface fixes its colour-only status encoding and difficult unaided launch path
Living evidence record
Impact record IAI-12ORP3L
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
23 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 23 October 2026
Research question, topics and regions
Whether a requirement-oriented dashboard helps staff read technical evidence from LLM evaluations
Topics: AI governance · Banking · EU AI Act · Compliance workflow · Responsible AI
Regions: France · Canada · European Union
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.