Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
Training neural networks without calculating gradients at all
A simple random-search algorithm can train deep neural networks without relying on backpropagation or gradients — the standard method that can cause training to fail. The method works by randomly tweaking each parameter and keeping changes that reduce errors, and successfully trained networks with over 20 layers, wide networks with thousands of neurons, and even Transformer models on image and language tasks.
Backpropagation's gradient calculations become unreliable in very deep networks, limiting how large and capable these systems can be. This gradient-free alternative sidesteps that problem entirely and works without requiring architectural tricks like batch normalization that researchers currently use as workarounds. If practical, it could unlock simpler ways to train neural networks and offer new insights into how these systems learn.