Title: The multiple timescales of gradient descent on the edge of stability
Abstract:
Neural networks are trained by gradient descent: repeatedly take a step downhill on the loss landscape, with a step size fixed in advance. Classical theory says that if the step size is too large relative to the curvature of the loss, the iterates should oscillate with growing amplitude and diverge. Yet in practice, deep learning routinely operates at exactly this threshold—a regime now called the edge of stability—and training does not fail. The iterates oscillate along the sharpest direction of the loss, but a nonlinear feedback effect pushes the curvature back down just enough to keep things under control. The resulting trajectory is jagged and hard to model with the usual continuous-time tools.
In this talk I will propose mathematical setting in which this behavior can be analyzed. The key assumption is that the loss looks like a sharp, narrow valley with a gentle slope along its floor. Treating the ratio of the two curvatures as a small parameter turns gradient descent into a singularly perturbed dynamical system, and a classical tool from asymptotic analysis—the method of multiple scales—reveals three separate timescales: fast oscillations across the valley, an intermediate feedback loop that regulates the curvature, and a slow drift along the valley floor. In particular, I will show that gradient descent converges to the central flow of Cohen et al. (2025) as the ratio converges to 0.