Tlaxcala’s assistant invented no fee in 40 audited AI answers—but half of clarifying users left
New analysis today of an unreviewed 27 October preprint: 380 production traces suggest deterministic routing can contain numeric errors, while slow generative replies and a 53% clarification drop-off expose the human cost of friction.
The short version
- What happened
- The Tlaxcala government assistant covered 1,143 procedures and 924 offices; its study analysed 380 organic production traces from 312 sessions over eight weeks.
- Who it affects
- A public assistant can avoid fabricating a fee yet still fail someone who waits half a minute or leaves an unclear menu. Completion, accessibility and recovery when the system is wrong matter as much as answer accuracy.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
One state portal
380 traces and 312 sessions
Only 40 generative production answers
Raw traces not public
Author designed and evaluated the platform
No service-completion outcome
Low traffic and CPU queueing
Limited fairness probe
Source analysis · 1 linked source. Primary analysed source date: 27 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The safest route usually avoided generation
A public-service assistant operating on the Tlaxcala state portal did not invent a fee, deadline or other number in 40 audited generative answers, according to an unreviewed Mexican preprint submitted on 27 October. The more important design choice was what happened before generation: 340 of 380 production traces, or 89.47%, were answered through deterministic retrieval. Only 40 reached the language-model synthesis layer.
The service covered 1,143 government procedures and 924 offices. Researchers examined 380 organic traces from 312 sessions between 15 February and 15 April 2026, alongside a separate 200-query stratified benchmark. The complete system reached 86.50% strict top-one accuracy on that benchmark—ten percentage points above a simpler binary gate—and averaged 86.41% across five folds, with a wide 7.50-point standard deviation.[1]
Zero observed inventions is not zero risk
Two analysts who were not involved in development audited the production answers blind. They found zero numeric inventions among the 40 generative cases, and the paper reports that all 40 synthetic attempts to inject an unsupported number were intercepted. Because the production denominator is small, however, zero out of 40 still corresponds to a reported 95% upper bound of 8.81%. The guardrail also checks number provenance, not whether the system retrieved the wrong neighbouring procedure, used a stale rule or swapped one concept for another.
The user journey was less reassuring. Thirty queries triggered a clarification menu and only 14 people replied, meaning 53.33% disappeared before the next turn. All 14 who continued converged on a procedure, but counting only completers would hide the attrition. Speed was similarly uneven: the global median was 0.665 seconds, yet the 95th percentile was 30.412 seconds because generative work queued on CPU infrastructure.[1]
The distinction that matters
Three routes, three evidence standards
The system tried to match the answer method to how much source evidence it had.
Exact record
Return deterministic source text
340 of 380 production traces
Ambiguous request
Ask the citizen to choose a facet
14 of 30 users replied
Supported synthesis
Generate with numeric verification
0 inventions observed in 40 audited answers
A promising operational study with an embedded evaluator
The author was both the platform's technical architect and the paper's investigator. The paper declares no commercial or financial conflict and describes the infrastructure as state-owned and open-source, while using independent analysts for the audits. Even so, raw citizen traces were not released, so outsiders cannot reproduce the production analysis or inspect who abandoned the service. A fairness probe also passed only six of ten pairs when socioeconomic traits were embedded naturally, and the assistant gave a falsely specific answer in four of 20 tests about unindexed state branches.
For public-service teams, the evidence supports constrained routing and human review—not a claim that sovereign infrastructure makes an assistant reliable. The next study should publish privacy-preserving trace samples, test accessibility and language variation with citizens, measure whether people complete the underlying service, and compare latency, error and abandonment against a non-AI search route.[1]
Continue the story
Read next
Continue exploring retrieval-augmented generation: compare the finding and its evidence limits.
When could an AI chatbot's news answer infringe a publisher's rights?
In the EU Court of Justice's first generative-AI case, the Advocate General says live retrieval may infringe when a chatbot either accesses a publication unlawfully or produces a recognisable substitute. The Opinion is influential legal advice, not a judgment.
Read the next reportExplore the full source and evidence record
What this means for people
- A public assistant can avoid fabricating a fee yet still fail someone who waits half a minute or leaves an unclear menu. Completion, accessibility and recovery when the system is wrong matter as much as answer accuracy.
Global context
The Tlaxcala deployment is unusual because it reports organic use from a Latin American public-service platform rather than an English-language laboratory benchmark. Its locally encoded procedures and rules are also precisely why the results should not be generalised to another administration without retesting.
What the evidence does not yet show
- Unreviewed preprint
- One state portal
- 380 traces and 312 sessions
- Only 40 generative production answers
- Raw traces not public
- Author designed and evaluated the platform
- No service-completion outcome
- Low traffic and CPU queueing
- Limited fairness probe
What to watch next
- Independent replication
- Privacy-preserving trace release
- Service-completion outcomes
- Accessibility and Spanish-language user testing
- Tail-latency reduction
- Fairness across natural language variants
Living evidence record
Impact record IAI-0Y1M3AN
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
29 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 29 October 2026
Research question, topics and regions
Governed retrieval systems for citizen services
Topics: Public services · Retrieval-augmented generation · Citizen access · Numeric verification · Sovereign AI
Regions: Mexico · Latin America
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.