Can a one-bit change send AI training down another path? A 500,000-network study finds a reproducibility limit
Across approximately 500,000 training runs, University of Sydney researchers found signatures consistent with interwoven, fractal-like training destinations. In one representative model, the smallest machine-representable parameter change altered the parameter-space destination in 47.7% of tested pairs, while prediction disagreement plateaued near 2.9%. The study identifies a candidate mechanism, not proof that every deployed AI system is inherently unpredictable.
The short version
- What happened
- The authors trained approximately 500,000 networks across VGGNet, ResNet and BERT configurations, using MNIST, CIFAR-10 and IMDB, several optimisers, regularisers and activation functions.
- Who it affects
- Engineers retraining safety-critical models may need to compare several trained instances, because identical code and similar aggregate accuracy do not guarantee identical learned behaviour.
- What remains unproven
- The analytical proof applies to a minimal two-layer network; deeper networks show empirical signatures consistent with riddling, not a complete geometric proof.
Keep in mind
The analytical proof applies to a minimal two-layer network; deeper networks show empirical signatures consistent with riddling, not a complete geometric proof.
The representative VGG-12 analysis uses MNIST with 50% corrupted training labels and is not a deployed foundation-model evaluation.
The 47.7% value concerns parameter-space destinations after least-significant-bit perturbations; prediction churn in the functional test was about 2.9%.
Approximately 500,000 runs span many configurations, but not every architecture, dataset, optimiser or safety-critical use.
The work identifies a candidate mechanism and diagnostic; it does not demonstrate a mitigation that preserves performance in production.
Source analysis · 3 linked sources. Primary analysed source date: 5 November 2026. Earliest linked supporting source: 7 October 2025.
Read the full report · 11 min The evidence, methods, implications and limitations
The answer: tiny changes can alter the trained destination, but the result is not a universal law of AI
A peer-reviewed Nature Machine Intelligence study published on 5 November reports evidence that almost indistinguishable starting points can send a neural network towards different trained solutions even when obvious sources of randomness are controlled. The researchers, Andrew Ly and Pulin Gong at the University of Sydney, propose that the destinations are organised like interwoven fractal regions known as riddled basins. In that geometry, every starting point leading to one outcome sits arbitrarily close to starting points leading somewhere else. Greater precision in the initial parameters may therefore deliver only a small gain in predicting the final destination.
The paper's strongest numerical example is deliberately specific. In a representative VGG-12 image classifier, changing a randomly selected parameter by the least significant bit—the smallest change available at the machine precision used—produced a different parameter-space destination in 47.7% of the tested close pairs. That does not mean the model gave a different answer on 47.7% of images. Different parameter arrangements can implement similar functions. When the team compared predictions instead of internal destinations, disagreement between nearby CPU-trained models plateaued at about 2.9% of inputs in that experiment.
Those results matter because reproducibility underpins debugging, assurance and incident investigation. If two runs with nearly identical starting states can learn measurably different functions, a single successful training run is weak evidence that the next run—or an independently retrained version—will behave identically. Yet the study does not show that all systems are uncontrollable, that a one-bit change always matters, or that any named medical, financial or public-service model will fail. The authors call riddling a candidate organising principle supported by analytical and empirical evidence across tested configurations.[1][2]
What a riddled basin means in plain language
Imagine a map in which every point represents a possible set of initial neural-network parameters. Colour each point according to the solution reached after training. In a simple landscape, large neighbouring areas share one colour: knowing the starting point more precisely makes the destination easier to predict. An ordinary fractal boundary complicates the border, but most points still lie inside a region where sufficiently accurate measurement recovers the destination.
A riddled basin is more severe. Regions leading to different attractors are interspersed at arbitrarily fine scales, so no point in one basin has a neighbourhood entirely free of alternatives. The paper measures this using an uncertainty exponent. The researchers perturb pairs of starting points by a distance labelled epsilon, train both networks and ask how often they finish at different destinations. If that uncertain fraction falls rapidly as epsilon shrinks, precision restores predictability. An exponent close to zero means the fraction barely falls: increasingly precise initialisation buys little certainty.
The authors first built a minimal two-layer tanh network in four-dimensional parameter space. That small system allowed them to derive the relevant symmetries, invariant subspaces, chaotic attractor and competing destinations analytically. They calculated Lyapunov exponents and mapped basins of attraction, establishing the classical route to riddling in the tractable model. This analytical result explains how the mechanism can arise. The larger-network experiments then ask whether comparable signatures appear in more realistic training, where proving the complete geometry is computationally and mathematically harder.[1][2]
The distinction that matters
From a tiny parameter change to a measurable behaviour difference
The study separates analytical evidence in a minimal network from empirical signatures in deeper models, then distinguishes internal destinations from actual predictions.
Minimal model
Derive symmetries, attractors and a route to riddling
Analytical mechanism in a two-layer network
About 500,000 runs
Test architectures, datasets, optimisers and regularisers
Riddling-like signatures across tested configurations
Representative VGG-12
Perturb initial weights down to one machine bit
47.7% different parameter destinations in the least-bit test
Function comparison
Compare predictions from nearby CPU-trained pairs
About 2.9% churn; exponent 0.00 ± 0.01
How the researchers tested about 500,000 networks
The empirical programme covered approximately 500,000 trained networks. Configurations included VGGNet, ResNet and BERT architectures; MNIST and CIFAR-10 image classification; IMDB sentiment classification; stochastic gradient descent, full-batch gradient descent and AdamW; L1 and L2 regularisation, batch and layer normalisation, and dropout; and tanh, ReLU and GELU activations. The paper says the qualitative signatures persisted across the tested combinations, including examples without artificial label noise. Code is available through Code Ocean and Zenodo, and the referenced datasets are public.
For high-resolution analysis, the main deep-network example was a VGG-12 model with tanh activations trained on MNIST after corrupting 50% of training labels. The unusual noise level is important: it makes the benchmark less representative of a clean production task, but creates a computationally manageable setting with many possible solutions. The model used stochastic gradient descent with momentum and weight decay, and each VGG-12 run in the central maps trained for 30 epochs. Test accuracy stabilised at about that point; non-origin destinations averaged 97.6% accuracy with a reported spread of plus or minus 0.5 percentage points.
The destination map contained 1,772 identified parity-invariant subspaces, distinguished by which neurons in the last hidden layer effectively vanished. For each uncertainty value, the team evaluated 1,000 initialisations around a fixed reference point and used bootstrap pairing to estimate the fraction reaching different destinations. The headline parameter-space exponent was 0.000 plus or minus 0.002 on both GPU and deterministic CPU training. CPU repetition controlled for accumulated hardware and scheduling randomness, strengthening the case that the measured sensitivity was not only a GPU artefact.[1][2]
The functional test is more relevant than different internal weights
Parameter differences alone can exaggerate practical concern because neural networks have symmetries: two sets of weights may be arranged differently yet produce the same predictions. The researchers therefore introduced a functional uncertainty exponent. Their simplest measure was churn—the fraction of dataset inputs for which two models selected different classes. They trained 250 CPU-based close pairs at each perturbation scale and compared their outputs, avoiding GPU non-determinism in the representative test.
As the initialisation difference shrank, mean prediction churn did not converge towards zero. It plateaued at approximately 2.9%, and the fitted functional exponent was 0.00 plus or minus 0.01. The team repeated the comparison with 32 measures from the model-similarity literature, including Cohen's kappa, centred kernel alignment and representational similarity analysis, and reports consistent qualitative results. This makes the finding more consequential than a claim about weight permutations alone: at least in the tested regime, increasingly close starts still produced a small but persistent difference in learned behaviour.
The size and meaning of 2.9% depend on the task and input distribution. It is not an error rate, a safety-incident probability or a direct estimate for deployed large language models. In a high-volume or safety-critical workflow, even a small disagreement rate could justify repeated training, ensemble comparison and targeted testing around high-consequence cases. In a low-risk classifier, the same number may have little practical effect. The study supplies a diagnostic for comparing runs; it does not set a universal acceptable threshold.[1][2]
A possible trade-off: the most effective learning rates were often less reproducible
Across most tested configurations, riddling-like signatures appeared near learning rates that produced the strongest training performance. Lowering the learning rate made parameter and functional outcomes more reproducible, but often weakened learning. The authors connect this to existing work on training near the edge of stability, where larger update steps can improve optimisation while introducing complex or chaotic dynamics. Their contribution is to focus on the global destination and learned function, not only whether two trajectories separate locally.
That trade-off should not be read as a fixed law that accuracy must always be exchanged for reproducibility. The supplementary tests show variation across configurations, and the paper points to possible alternatives such as changing network symmetries rather than merely reducing the learning rate. Symmetry-breaking methods have previously reduced dependence on initialisation, but the resulting basin geometry has not yet been characterised here. No new architecture or training recipe was prospectively shown to preserve performance while eliminating the reported instability.
For practitioners, the immediate implication is procedural. A model-selection pipeline should record seeds, data order, software, precision and hardware, then compare multiple independent training runs at the function level. Averaging accuracy can conceal whether different runs fail on different people or examples. When retraining a clinical or public-service model, teams should test disagreement, calibration and subgroup performance against the approved version rather than assuming that matching code and aggregate accuracy reproduce the same behaviour.[1][2]
What remains unproven—and what would change the assessment
The broad claim remains a hypothesis about organisation across deep learning, not a proof for every architecture and task. The complete symmetry-induced invariant subspaces are unknown for many modern networks, and computing destinations at the required resolution is expensive. The central VGG-12 experiment uses MNIST and 50% label noise, a deliberately simplified setting rather than a contemporary foundation model or a deployed decision system. BERT and other configurations contribute qualitative supporting tests, but the paper does not report a field study of patient, worker or citizen outcomes.
Machine precision also places a practical floor under the mathematical argument: perturbations smaller than the numerical representation become identical. The 47.7% result concerns internal destination changes after least-significant-bit perturbations in one reference configuration, while the 2.9% result concerns prediction disagreement in a separate functional comparison. Combining those figures into a claim that one bit changes 47.7% of answers would be false. The authors disclose Australian Research Council support and no competing interests; code and generated primary data are openly linked.
Confidence would rise if independent teams reproduced near-zero functional exponents in larger foundation models, clean real-world datasets and repeated end-to-end deployments. The practical risk assessment would also change if teams showed that symmetry-breaking, ensembling or another intervention reduced churn without harming performance, calibration or subgroup fairness. For now, the defensible conclusion is that training reproducibility can have an intrinsic geometric component in tested networks, and that model assurance should compare behaviours across runs rather than treating deterministic code as sufficient proof of reproducibility.[1][2][3]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring deep learning: compare the finding and its evidence limits.
Can Tangermeme reveal what genomic AI has learned?
A peer-reviewed Nature Methods toolkit standardises prediction, perturbation, attribution and sequence-design operations around genomic deep-learning models. It can turn black-box outputs into testable hypotheses, but a model explanation is still not proof of a biological mechanism.
Read the next reportExplore the full source and evidence record
What this means for people
- Engineers retraining safety-critical models may need to compare several trained instances, because identical code and similar aggregate accuracy do not guarantee identical learned behaviour.
- Clinicians, workers and public-service professionals should not interpret the study as evidence that their current system is unreliable; it identifies a mechanism that assurance teams can test for.
- Procurement and audit teams can ask vendors for run-to-run disagreement, calibration and subgroup results rather than accepting a single benchmark score.
Global context
The work comes from an Australian university and uses globally available benchmark datasets and open code. Its mathematical mechanism is not geographically specific, but the human consequences depend on where and how models are trained, validated and deployed. Organisations with limited compute may find repeated training and fine-scale reproducibility tests costly, making transparent vendor evidence especially important.
What the evidence does not yet show
- The analytical proof applies to a minimal two-layer network; deeper networks show empirical signatures consistent with riddling, not a complete geometric proof.
- The representative VGG-12 analysis uses MNIST with 50% corrupted training labels and is not a deployed foundation-model evaluation.
- The 47.7% value concerns parameter-space destinations after least-significant-bit perturbations; prediction churn in the functional test was about 2.9%.
- Approximately 500,000 runs span many configurations, but not every architecture, dataset, optimiser or safety-critical use.
- The work identifies a candidate mechanism and diagnostic; it does not demonstrate a mitigation that preserves performance in production.
What to watch next
- Independent reproduction of functional uncertainty exponents in larger language, vision and multimodal models.
- Tests on clean operational datasets and frozen production pipelines rather than deliberately noisy benchmarks.
- Whether symmetry-breaking, ensembling or alternative optimisation reduces run-to-run churn without weakening accuracy or fairness.
- Reporting standards that include functional disagreement and subgroup instability across training runs.
- Prospective evidence linking run-to-run differences with real safety incidents or decision outcomes.
Living evidence record
Impact record IAI-1BNINDC
Evidence stage
Studied
Confidence
Corroborated
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
5 November 2026
Source trail
3 direct sources across 2 source types.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Related-source reporting disclosure
This record analyses 3 linked source records around the same underlying development. The extra records add method, date or context, but they do not by themselves constitute independent replication of every performance claim or predicted outcome.
Evidence trail
Sources used for this report
Links checked 5 November 2026
Research question, topics and regions
Fractal training basins and reproducibility in deep neural networks
Topics: Deep learning · Reproducibility · AI reliability · Training dynamics · Model evaluation
Regions: Australia · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.