Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion
Why AI models get better at creative writing when trained to the point of seeming overfit
When researchers push large language models to memorize small datasets almost perfectly, the models paradoxically generate more creative and varied text. The researchers show this isn't simply the model sharpening its predictions—temperature scaling controls can't replicate the effect—and discovered the mechanism lies in the final neural network layer, which undergoes a geometric expansion that rescues rare words from obscurity.
Fine-tuning is one of the fastest ways to adapt AI models to specific tasks, but practitioners have long assumed that pushing training loss too low causes the model to overfit and fail. This work shows that apparent overfitting can actually improve real-world output quality, challenging a core assumption in how models are trained and opening a path to better performance with minimal computational cost.