Why do diffusion models stop copying and start inventing? A study finds a middle phase
Across four image datasets, researchers found recurring outputs that copied neither the training set nor ordinary one-off generations—but the controlled benchmark does not establish how often commercial systems reproduce creative work.
The short version
- What happened
- Researchers trained grids of diffusion models on MNIST, Fashion-MNIST, CIFAR-10 and LSUN-Church while increasing the training-set size, then classified outputs as memorised, intermediate or generalised.
- Who it affects
- Artists, publishers and model users need audits that distinguish direct training-data copies from repeatable model-made patterns; this study offers a testable framework, not an estimate of how often either occurs in deployed products.
- What remains unproven
- The dense experiments used small, controlled image benchmarks and relatively compact diffusion models, not present-day commercial systems.
Keep in mind
The dense experiments used small, controlled image benchmarks and relatively compact diffusion models, not present-day commercial systems.
Nearest-neighbour thresholds were selected partly through manual inspection and depend on the pixel or perceptual distance metric.
The Stable Diffusion 1.4 analysis was qualitative and used candidate prompts drawn from prior studies.
The paper does not measure copyright infringement, harmful content, user outcomes or the frequency of copying in a deployed service.
Source analysis · 2 linked sources. Primary analysed source date: 25 October 2026. Earliest linked supporting source: 27 May 2025.
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Read the full report · 4 min The evidence, methods, implications and limitations
Between a copy and a novel image, the model formed repeatable attractors
The peer-reviewed study treats a diffusion model as a form of associative memory and asks what changes as it sees more training examples. On four image datasets—MNIST, Fashion-MNIST, CIFAR-10 and LSUN-Church—the team trained separate models across roughly 38 increasing dataset sizes, from very small subsets towards totals of 50,000 to 126,227 images. Generated pools were usually four times the corresponding training set; a CIFAR-10 repeat used eight times as many generated images.
Outputs were labelled memorised when they closely matched a training image, generalised when they were unique in both the training and generated pools, and 'spurious' when they recurred in the generated pool but were absent from training. As dataset size grew, copying fell and generalisation rose. The recurring middle category peaked near that boundary, and moving the model's width shifted where the transition happened. In this paper, 'spurious' describes a stable generated pattern; it does not mean a factual error or unsafe output.[1][2]
The finding may sharpen audits, but it is not a product risk score
The researchers measured nearest-neighbour distances using pixel-space distance for the simpler datasets and a learned perceptual metric for CIFAR-10 and LSUN-Church. They also compared the curvature of the model's energy landscape, finding sharper basins for memorised outputs, intermediate curvature for the recurring middle states and flatter geometry for generalised outputs. Up to 512 examples per category were used in the reported curvature calculations. A qualitative Stable Diffusion 1.4 check was much smaller and based on candidate prompts from earlier work.
For model builders, the practical message is that 'not a training-set duplicate' does not necessarily mean a model has learned a broad, smooth distribution: it may have created another narrow pattern that repeats. Audits could therefore compare outputs with both training data and other generations. But the thresholds were manually tuned with histogram and image inspection, the benchmarks are small, and no creator, consumer or deployment outcome was measured. Larger blinded audits on current production models, with pre-registered thresholds and independent data, would show whether the mechanism predicts real copying, diversity or safety failures.[1][2]
The distinction that matters
Three output states in the study
The classification depends on comparisons with both the training set and a larger pool of generated images.
Memorised
Repeated among generations and close to a training image
Evidence of a copied training pattern
Intermediate
Repeated among generations but absent from training
A stable new attractor, termed spurious by the authors
Generalised
No close duplicate in either comparison pool
A unique sample under the study's distance threshold
Watch the background, then check the evidence
Illustrative photo · not a video frame
Google DeepMind · 1 min 52 sec · Background explainer
Protein folding explained
YouTube loads only when you choose; playback starts when you press play.
A short introduction to why a protein’s shape matters. It provides background for understanding what AI structure prediction contributes to biological research.
Evidence check: This is the developer’s explanation. Predicting a structure does not by itself establish a treatment’s safety or benefit for patients.
Published 2020-11-30. Source checked 14 October 2026. Check captions on the publisher’s video page.
Explore current researchContinue the story
Read next
Continue exploring diffusion models: compare the finding and its evidence limits.
Can an audit tell whether an image trained a diffusion model?
A new preprint roughly doubled strict membership detection on one CIFAR-10 test and approached perfect results on heavily fine-tuned image models. It needs internal denoising access, and a positive score remains statistical evidence—not proof of dataset use.
Read the next reportExplore the full source and evidence record
What this means for people
- Artists, publishers and model users need audits that distinguish direct training-data copies from repeatable model-made patterns; this study offers a testable framework, not an estimate of how often either occurs in deployed products.
Global context
The collaboration spans US, Dutch and French institutions and includes IBM-linked funding and employment history. The mechanism could inform model evaluation worldwide, but model architectures, training corpora and legal tests differ across services and jurisdictions.
What the evidence does not yet show
- The dense experiments used small, controlled image benchmarks and relatively compact diffusion models, not present-day commercial systems.
- Nearest-neighbour thresholds were selected partly through manual inspection and depend on the pixel or perceptual distance metric.
- The Stable Diffusion 1.4 analysis was qualitative and used candidate prompts drawn from prior studies.
- The paper does not measure copyright infringement, harmful content, user outcomes or the frequency of copying in a deployed service.
What to watch next
- Pre-registered replication on modern large diffusion models and independent datasets
- Audits that compare outputs with both training data and other generations
- Sensitivity to perceptual metrics, thresholds, prompts and model architectures
- Whether the detected states predict real diversity, privacy or copying failures
Living evidence record
Impact record IAI-0T2A5PX
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
25 October 2026
Source trail
2 direct sources across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 2 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 25 October 2026
Research question, topics and regions
How diffusion models move from reproducing training examples to generating novel outputs as training-set size and model capacity change
Topics: Diffusion models · Memorisation · Generative AI · Training data · Model evaluation
Regions: United States · Netherlands · France
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.