Can an AI assistant reveal a private condition without naming it? A synthetic benchmark found indirect clues survived privacy prompts
Across five models and 950 synthetic persona-task instances, privacy-aware instructions sharply reduced explicit disclosure but left attributes inferable in 16.2% to 37.4% of top-three tests. The benchmark used simulated data and an LLM recipient, not real communications.
The short version
- What happened
- InferPriv combines 10 synthetic personas, 10 sensitive attributes and 10 routine tasks into 950 instances, with five sampled replies per protected persona-attribute-task cell.
- Who it affects
- People using assistants for work, travel or bookings may disclose sensitive facts through helpful details they did not choose to share, even when the assistant avoids naming the fact.
- What remains unproven
- Unreviewed preprint
Keep in mind
Unreviewed preprint
Synthetic personas, memories and tasks
One LLM acts as recipient and utility judge
Small English-language scenario set
Worst-case any-of-five and top-three metrics
No real users, recipients or measured harm
Source analysis · 1 linked source. Primary analysed source date: 25 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The benchmark looks for clues, not only secret words
An unreviewed US preprint tests a privacy failure that ordinary keyword checks can miss: an assistant may avoid saying that a user has diabetes yet request a refrigerator for insulin, allowing a hotel coordinator to infer the condition. InferPriv pairs 10 synthetic personas with 10 sensitive attributes—including health conditions, religion and financial distress—and 10 routine coordination tasks. The final set contains 950 instances; implausible persona-attribute combinations are excluded.
Each persona has 27 ordinary memory statements plus three records describing one sensitive attribute. The researchers identified 98 attribute-task pairs where the sensitive records were unnecessary to complete the request. Five frontier models then drafted five replies for each protected cell. A separate language model, shown only the outgoing message, made up to three guesses about the writer. A control run without the sensitive records was subtracted to reduce chance guessing and background confounding.[1]
Privacy instructions stopped names more readily than implications
With a privacy-aware prompt, explicit disclosure across five replies fell to 0% for GPT-6.1-Sol and Gemini-3.8-Flash and 0.9% for Sonnet-5.5. Yet the correct attribute still appeared among the simulated recipient’s top three guesses in 25.2%, 16.3% and 16.2% of adjusted protected cells respectively. DeepSeek-V4.1-Flash and Muse-Spark-1.3 retained higher explicit and inferential rates. Task-completeness scores stayed between 90.2% and 96.2% under the privacy prompt.
The metric is deliberately conservative from a privacy perspective: if any of five sampled replies leaks, the cell counts as leaking. Top-three guessing gives the recipient several chances, so the top-one results are also important. They ranged from 13.0% to 33.4% under the privacy prompt. Increasing reasoning effort for one model lowered explicit disclosure but did not remove indirect leakage, and removing the sensitive attribute’s name from memory records barely changed its privacy-aware result.[1]
The distinction that matters
Direct versus inferential disclosure
A reply can keep the attribute unnamed while still communicating enough clues for someone to infer it.
Direct
Reply names diabetes
Keyword disclosure is visible
Inferential
Reply requests insulin storage
Condition may be guessed
Safer path
Minimize context and ask before sending
Needs real-user validation
This is a synthetic stress test, not observed surveillance
No people sent these messages and no real recipient inferred a health condition, religion or financial status. Personas, memories and tasks were synthetic; the same model family acted as the curious recipient and the task-completeness judge. The test covers a small set of English-language coordination scenarios, and its lexical matching may miss paraphrases or count broad guesses differently from a human. The paper does not state funding or competing-interest disclosures.
Even with those limits, the design isolates a practical distinction for product teams: direct disclosure and inferential disclosure are different tests. Before an assistant sends a message to an employer, hotel or service provider, it may need to minimize which memories enter the drafting context, express requirements without revealing their cause and ask the user before transmitting a revealing accommodation. Real-user studies, human-recipient evaluations and repeated tests across languages and tasks would show whether this benchmark predicts actual privacy loss.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring privacy: compare the finding and its evidence limits.
Can AI agents share lessons without sharing private traces?
FedSkillGit replaced whole-document sharing with filtered rule-level patches and beat independent learning in nine benchmark combinations. Its privacy evidence measured an AI audit, not resistance to extraction attacks.
Read the next reportExplore the full source and evidence record
What this means for people
- People using assistants for work, travel or bookings may disclose sensitive facts through helpful details they did not choose to share, even when the assistant avoids naming the fact.
Global context
Privacy norms, protected attributes, language cues and legal duties differ across jurisdictions. The benchmark offers a technical stress test, not a universal standard for lawful or acceptable disclosure.
What the evidence does not yet show
- Unreviewed preprint
- Synthetic personas, memories and tasks
- One LLM acts as recipient and utility judge
- Small English-language scenario set
- Worst-case any-of-five and top-three metrics
- No real users, recipients or measured harm
What to watch next
- Human-recipient validation
- Real-user consent studies
- Cross-language replications
- Input memory filtering
- User approval before revealing accommodations
- Independent tests on deployed assistants
Living evidence record
Impact record IAI-1BAVBPH
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
27 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what arXiv published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 27 October 2026
Research question, topics and regions
Whether personalized AI assistants can communicate clues that let a recipient infer a sensitive attribute even when the assistant does not explicitly disclose it
Topics: Personal agents · Privacy · Persistent memory · Data minimization · Benchmarking
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.