Broken scale symmetries in undercomplete linear autoencoders
How neural networks accidentally develop lopsided weight patterns during training
When neural networks train on simple tasks, they break a fundamental symmetry by preferring large weights in some layers over others — a biased drift that shouldn't matter mathematically but consistently happens anyway. This directed pattern emerges from random noise in training, follows predictable laws, and eventually hits a hard boundary where the network can no longer drift further.
Understanding how and why neural networks develop asymmetric solutions helps explain why standard training produces sharp, efficient models that generalize well. Since this bias is driven by noise in the training process itself rather than by the data or loss function, it reveals a hidden mechanism that could explain patterns observed in real neural networks — and potentially be leveraged to improve how networks learn.