Gradient descent, momentum and Adam on the same loss.
Step Gradient descent Momentum Adam
The same valley, three update rules. Plain gradient descent bounces across the steep direction and crawls along the shallow one. Momentum builds speed along the floor. Adam divides each step by a running estimate of the gradient’s size, so both directions move at a similar pace. The loss is a quadratic 20 times steeper across the valley than along it.