Superlinear Convergence of Cyclic Gradient Methods with Historical Ritz Step Sizes
How recycling old calculation directions speeds up optimization algorithms
Mathematicians proved that a specific technique for reusing previous calculation directions can make optimization algorithms converge extremely fast—doubling the number of correct digits roughly every few steps. The trick works by strategically reusing a small set of historical step sizes in cycles, creating an exponential speedup that was theoretically predicted but had never been rigorously demonstrated for practical algorithms that keep only limited memory.
Optimization algorithms solve real problems in machine learning, engineering, and finance—training neural networks, designing systems, and analyzing data. Faster convergence means reaching accurate answers with fewer computer calculations, saving time and energy. This work explains exactly when and why reusing old information actually accelerates these algorithms, helping practitioners know which techniques to trust and how to tune them for their specific problems.