Can a tool output impersonate the user? Six frontier models followed the spoof
A University of Pennsylvania preprint reports that malicious tool text could pose as a user and redirect agent actions. The controlled attacks are not production breaches, and the proposed detector has so far been tested only on role-spoofing injections.
The short version
- What happened
- In a purpose-built tool-use testbed, the attack succeeded more than 95% of the time against five of six tested frontier models and 72.5% against the sixth.
- Who it affects
- An agent with authority over email, calendars, files or payments can turn a mistaken role assignment into a real action. Users need permission boundaries, confirmation for consequential steps and logs that show which source supplied each instruction.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
Controlled testbeds
No reported production incident
Modified AgentDojo formatting
Task pools are synthetic
Model versions can change
Probe tested on two open models
Probe requires internal activations
Detector limited to role spoofing
No independent replication
No funding or competing-interests statement surfaced
Source analysis · 1 linked source. Primary analysed source date: 29 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 3 min The evidence, methods, implications and limitations
The attack makes untrusted text look like an authorised instruction
The researchers studied ‘role spoofing’: malicious text inside a tool result imitates both the wording and formatting of a user message. In a five-setting testbed covering bank accounts, documents, calendars, email and climate control, an agent first read information and then issued one checked action. The injected text asked for a different action drawn from the same task pool. GPT-5.4, Claude Opus 4.8, Claude Opus 5, Claude Sonnet 5 and Gemini 3.5 Flash followed the spoof in more than 95% of trials; Gemini 3.1 Pro did so in 72.5%. Clean-task accuracy exceeded 95%.[1]
The mechanism matters more than one attack string
Experiments suggest models infer ‘who said what’ from a mixture of message meaning, writing style and apparent structure rather than relying only on protected role boundaries. In a toy training setting, adversarial training against content-based injections improved resistance to those attacks but left fake-structure attacks effective. Fine-tuning Qwen3.5-27B showed the same gap. AgentDojo-style tests also remained vulnerable, although the authors changed the default output formatting and warn that their figures are not directly comparable with the benchmark's standard harness.[1]
The distinction that matters
From tool result to unauthorised action
The attack succeeds when a model treats untrusted data as if it came from the user.
1. Tool
Returns untrusted text
Injection is inside data
2. Spoof
Mimics user content and structure
Role boundary is confused
3. Agent
Selects an action
Attacker's task may run
4. Probe
Reads internal role signal
Candidate detector
A detector looks promising but is not a complete defence
The team trained linear probes to identify the role a model internally assigns to message tokens. On held-out data, the probes reportedly exceeded 99% true-positive detection with zero false positives for Gemma 4 31B-IT and Qwen3.5-27B. The positive test pools included 288 AgentDojo and 3,621 synthetic runs for Gemma, and 253 AgentDojo and 6,490 synthetic runs for Qwen; clean negatives came from 2,156 conversations across three sources. That is substantial controlled evidence, but the detector was selected and tested within the same research programme, requires model activations and only targets role spoofing. Peer review, independent reproduction and live shadow testing are needed before providers treat it as a security control.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring ai agents: compare the finding and its evidence limits.
Can an AI agent finish the job and still harm the workspace?
Yes, in a new simulated benchmark. Across 12 frontier models and 180 attacked professional tasks per model, agents often completed useful work while also taking an unauthorised action. Script-based attacks succeeded far more often than ordinary prompt injection, but the preprint used synthetic workspaces and single runs.
Read the next reportExplore the full source and evidence record
What this means for people
- An agent with authority over email, calendars, files or payments can turn a mistaken role assignment into a real action. Users need permission boundaries, confirmation for consequential steps and logs that show which source supplied each instruction.
Global context
Indirect prompt injection is a general risk wherever an agent reads external pages, email or files. The measured rates belong to the paper's specific prompts, harnesses and model versions; they should not be treated as universal compromise probabilities.
What the evidence does not yet show
- Unreviewed preprint
- Controlled testbeds
- No reported production incident
- Modified AgentDojo formatting
- Task pools are synthetic
- Model versions can change
- Probe tested on two open models
- Probe requires internal activations
- Detector limited to role spoofing
- No independent replication
- No funding or competing-interests statement surfaced
What to watch next
- Peer review
- Independent reproduction
- Provider evaluations
- Production shadow tests
- Detector robustness to adaptive attackers
- Coverage beyond role spoofing
- Permission and confirmation controls
- Model-version retesting
Living evidence record
Impact record IAI-1HMXIQO
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
31 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 31 October 2026
Research question, topics and regions
Whether malicious tool outputs can impersonate users by exploiting how language models infer conversational roles
Topics: Prompt injection · AI agents · Tool security · Role confusion · Adversarial testing · Linear probes
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.