See where each activation's derivative lets gradients pass.
f(x) , f′(x)
On the way back, every gradient is multiplied by the derivative of the activation at that unit’s input. Drag along the curve or use the slider: the slope of the dashed tangent to f is the height of f′. The red bands mark |f′| < 0.05, where sigmoid and tanh saturate and ReLU passes nothing at all.