Mitigating Model Collapse through Dreaming Learning
How to stop AI models from forgetting diversity when trained on their own outputs
When generative models like GPT train on synthetic data they've created themselves, they gradually lose diversity and performance in a process called model collapse. Researchers found that this collapse appears as a mathematical narrowing in the model's internal structure—subdominant eigenvalues approaching one—which they can see directly by analyzing the model's spectral properties. A simple fix called Dreaming Learning, which trains the model alongside its own high-temperature samples, keeps this mathematical collapse from happening by preventing the model's internal values from becoming vanishingly small.
Model collapse is a practical threat to systems that continuously improve themselves using their own generated data—a pattern increasingly common in AI deployment. This research identifies exactly where the problem occurs mathematically and offers a concrete solution that works across different model types, from small Markov systems to GPT-2. The fix is computationally simple enough to implement in production, making it possible to train models on synthetic data without accepting inevitable performance degradation.