8 Jun 2026
While many different model configurations can fit training data perfectly, they often differ significantly in how they perform on unseen data. Modern networks are even capable of completely memorizing the training data — so why do they still generalize?
While many different model configurations can fit training data perfectly, they often differ significantly in how they perform on unseen data. Modern networks are even capable of completely memorizing the training data — so why do they still generalize?
This remarkable generalization is typically explained by implicit biases — the hidden regularizations introduced by the training algorithm, such as favoring smaller parameters (low norm) or flatter regions of the loss landscape (low sharpness).
In our new paper, accepted at the International Conference on Machine Learning (ICML) 2026, we demonstrate that these implicit biases conflict with each other at the Edge of Stability (EoS) regime. We find that neither norm nor sharpness minimization alone is sufficient to explain generalization; instead, optimal performance requires balancing the trade-off between them.
We validate these dynamics through extensive empirical experiments and analyze a simplified theoretical model to shed light on the underlying mechanisms.
👉 Read the full paper
Authors: Maria Matveev, Vít Fojtík, Hung-Hsu Chou, Gitta Kutyniok, and Johannes Maly