MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
A smarter way to train giant language models faster and more efficiently
Researchers discovered that a key bottleneck in how language models learn comes from how much their internal weights can shift during training. They developed MuonGS, a new optimizer that allows weights to grow gradually instead of staying fixed, which lets the model explore more effectively and learn 15% faster than existing methods while reaching the same quality.
Language model pretraining is expensive—running for weeks on powerful computers. A 15% speedup means real savings in computing costs and energy use. The insight about controlled weight growth could apply to training many other types of neural networks, making AI development faster and cheaper across the board.