Can synthetic companies test Copilot without exposing customer data? Microsoft built three—but published no task results
Microsoft says it now tests enterprise AI in simulated pharmaceutical, capital-markets and software organisations. The method could reduce privacy risk, but the datasets, scoring rubric and Copilot accuracy results remain internal.
The short version
- What happened
- Microsoft built three synthetic enterprise environments covering pharmaceuticals, capital markets and software, with hundreds of simulated users and time-evolving work records.
- Who it affects
- Workers may benefit if realistic pre-deployment tests expose errors before an assistant handles meetings, safety reviews or financial work. They also need evidence that the tests represent real ambiguity and that failures discovered internally lead to product changes, not just a higher synthetic-data quality score.
- What remains unproven
- Company-reported method
Keep in mind
Company-reported method
Three synthetic environments
Datasets not public
Scoring rubric not public
No validation denominator
No Copilot task results
No comparison with real enterprise use
Cost savings not independently audited
Source analysis · 1 linked source. Primary analysed source date: 30 October 2026.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
The method replaces customer records with simulated workplaces
Microsoft Research says it has created an internal platform for evaluating Copilot in synthetic enterprise environments rather than real customer workspaces. Three simulated organisations cover pharmaceutical regulatory and safety work, capital-markets deals and compliance, and software development. Each contains hundreds of fictional users, organisational structures, meetings, emails, documents and project histories that change over time. That matters because questions such as what a team decided last week require a coherent history, not isolated test prompts.
The company says job roles and activity patterns are grounded in the US Department of Labor’s O*NET database, while public job postings shape titles and reporting structures and SEC filings inform organisational scale and financial context. An automated six-stage pipeline selects workplace tasks, adapts them to an industry, generates questions from the synthetic records, rewrites them into more natural language and checks grounding and uniqueness. No real customer data is supposed to enter the environments.[1]
Privacy-friendly test data is not the same as proven realism
Synthetic environments can make repeated tests safer and easier to share, especially when real meetings and emails contain commercially sensitive or personal information. They can also create a known answer for questions about a simulated organisation. But usefulness depends on whether the invented communications, power structures, omissions and changes resemble the messy situations in which workers actually use an assistant. A clean synthetic world may fail to reproduce ambiguous requests, undocumented decisions or conflicts between official records and informal practice.
Microsoft says an internal composite quality score improved 39% from its initial generation approach and that some industry environments scored above 0.9 out of 1. It also estimates about $100,000 saved per evaluation dataset and $300,000 across the project. These are company measurements. The post does not define the composite formula, disclose a validation denominator or publish human-review agreement. It says the framework measures three dimensions, then lists structural fidelity, contextual relevance, behavioural realism and content safety—four categories.[1]
The distinction that matters
What the platform tests—and what remains hidden
The pipeline creates a controlled workplace simulation, but the public evidence stops before Copilot's outcomes.
Inputs
O*NET, job postings and SEC filings
Synthetic roles, structures and business context
Test world
Generate meetings, emails and time-aware tasks
Repeatable enterprise questions without customer records
Missing check
Publish task results and real-world comparison
Not provided, so predictive validity remains untested publicly
The missing result is how Copilot performed
The platform has entered an internal Microsoft catalogue as a reusable Copilot evaluation baseline, but the announcement reports no task-level accuracy, citation quality, privacy leakage, safety failures or comparisons with real enterprise use. It therefore supports a narrower conclusion: Microsoft has built evaluation infrastructure designed to avoid customer data. It does not show that Copilot is reliable in pharmaceuticals, finance or software work, or that synthetic testing predicts live performance.
For employers and workers, the practical question is whether this kind of test reveals failures before an assistant reaches a sensitive workflow. Evidence that would change the assessment includes public benchmark tasks and answer keys, an independently reviewed scoring rubric, comparisons between synthetic and consented real-world tests, and results split by industry and error severity. Until then, the approach is a promising internal method with privacy advantages—not external proof of enterprise readiness.[1]
Watch the background, then check the evidence
Illustrative photo · not a video frame
The Royal Society · 2 min 31 sec · Background explainer
What is artificial intelligence? | The Royal Society
YouTube loads only when you choose; playback starts when you press play.
A short introduction to artificial intelligence from the Royal Society. Start here for basic concepts, then use current reporting to explore what has changed.
Evidence check: Published in 2018, before today’s generative AI tools. Useful for basic concepts, rather than an overview of current capabilities.
Published 2018-11-28. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current AI technologyPhoto: ThisisEngineering / Unsplash · Licence
Continue the story
Read next
Continue exploring synthetic data: compare the finding and its evidence limits.
Synthetic rare-lesion images helped five novice endoscopists—what remains unproven
A peer-reviewed study found better image-level recognition after novices trained on generated examples, but the same fixed test was used before and after and no patient outcomes were measured.
Read the next reportExplore the full source and evidence record
What this means for people
- Workers may benefit if realistic pre-deployment tests expose errors before an assistant handles meetings, safety reviews or financial work. They also need evidence that the tests represent real ambiguity and that failures discovered internally lead to product changes, not just a higher synthetic-data quality score.
Global context
The public grounding sources are largely US-based and the three simulated sectors do not represent every workplace or jurisdiction. Job roles, privacy obligations, reporting lines and communication norms differ across countries, so a reusable internal baseline is not automatically a global validation set.
What the evidence does not yet show
- Company-reported method
- Three synthetic environments
- Datasets not public
- Scoring rubric not public
- No validation denominator
- No Copilot task results
- No comparison with real enterprise use
- Cost savings not independently audited
What to watch next
- Public benchmark release
- Human validation sample and agreement
- Synthetic-to-real comparison
- Copilot task accuracy and failure rates
- Privacy leakage testing
- Industry-specific results
- Independent replication
Living evidence record
Impact record IAI-1QWGZQU
Evidence stage
Announced
Confidence
Developing
Reporting basis
Source analysis
Independent or research support
Not yet
Record status
Monitoring
Last checked
30 October 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Microsoft Research published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 30 October 2026
Research question, topics and regions
Whether synthetic enterprise environments can support realistic, privacy-preserving evaluation of workplace AI
Topics: Microsoft Copilot · Enterprise AI · Synthetic data · Evaluation · Privacy · Workplace technology
Regions: United States · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.