What is the difference between batch, stochastic, and mini-batch gradient descent?

Batch uses the whole dataset per step (accurate, slow). Stochastic uses one example (fast, noisy). Mini-batch uses a small batch (the default) balancing speed, stable gradients, and helpful noise.

What does momentum add to SGD?

A velocity term (an exponentially weighted sum of past gradients) that accelerates consistent directions and damps oscillation across narrow valleys.

What does the Adam optimizer do?

Adam tracks exponentially weighted first and second raw moments of gradients, then uses bias-corrected estimates to scale each parameter update. The second moment is the mean squared gradient, not centered variance.

Why does Adam need bias correction?

Moment estimates initialized at zero are biased toward zero early on. Dividing each by its 1 − βᵗ factor corrects this initialization effect; it does not simply make every update larger.

Gradient Descent (SGD, Momentum, Adam)

Follow updates across a loss surface, then connect learning rate, momentum, and Adam to the update equations.

8 min readReviewed September 2026By Oliver Perrin · Machine Learning Engineer

Learn with a guided session

Jump to a section

Why this matters

Gradient descent updates parameters in the direction opposite the gradient of a loss. The learning rate sets the step size. Full-batch updates use the whole dataset, stochastic updates use one example, and mini-batch updates use a subset.

Momentum carries information from earlier gradients into the next update. Adam also rescales updates using a running second moment. These methods share the same aim, but they do not take the same path across a loss surface.

Build the mental model

Imagine a ball rolling on the loss surface. Plain SGD scales the current gradient by a learning rate (the step length is not fixed): it zig-zags across narrow valleys and crawls across plateaus. Momentum gives the ball mass: it builds up velocity in consistent directions and cancels back-and-forth oscillation. Adam additionally gives each parameter its own step size, scaled down for parameters with large, noisy gradients and up for those with small, steady ones.

Explore the ideaRelated Studio experiment

Follow the negative gradient

The update moves against the slope of one exact quadratic.

Exact inputs and equations appear with the figure.

Preparing the scientific figure…

Drag to rotate

Exact updates on the positive quadratic shown. This is not a measured training run, and it does not represent a neural-network loss landscape. No stochastic minibatches are sampled.

Authored analytic example

Experiment with related Studio inputs

Replay the path at a small learning rate, then increase it toward 1. Rotate the surface to inspect the overshoot.

An exact quadratic example with one minimum. Momentum, Adam, and stochastic gradients are discussed below but are not simulated here.

Mini-batching lets hardware process examples in parallel while controlling memory use. Batch size affects gradient noise, throughput, and optimization behavior; the useful range depends on the model, data, and hardware.

Work through the math

SGD with learning rate $\eta$ , on a mini-batch gradient $g_t = \nabla_\theta L_t$ :

\theta_{t+1} = \theta_t - \eta\, g_t

SGD with momentum ( $\beta \approx 0.9$ ): maintain a velocity $v$ that is an exponentially weighted sum of past gradients:

v_{t} = \beta v_{t-1} + g_t, \qquad \theta_{t+1} = \theta_t - \eta\, v_t

Adam combines a first moment (mean, like momentum) and second raw moment (mean squared gradient, not centered variance) with bias correction:

m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2

\hat{m}_t = \frac{m_t}{1-\beta_1^{t}}, \qquad \hat{v}_t = \frac{v_t}{1-\beta_2^{t}}, \qquad \theta_{t+1} = \theta_t - \eta\,\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

The bias correction matters early in training, when $m_t, v_t$ are still biased toward their zero initialization. Typical defaults: $\beta_1=0.9$ , $\beta_2=0.999$ , $\epsilon=10^{-8}$ . AdamW decouples weight decay from the gradient step and is widely used for training Transformers.

Key takeaway: Every variant shares one skeleton: estimate a gradient on a batch, take a step. Momentum shapes the step with gradient history; Adam rescales it per parameter; bias correction compensates for moment estimates initialized at zero.

Optimizer	Update uses	Per-parameter adaptivity	Where it shines
Batch GD	Full dataset gradient	No	Problems where full-gradient steps are affordable
Mini-batch SGD	Small-batch gradient	No	Scalable baseline; requires tuning
SGD + momentum	Exponentially weighted sum of past gradients	No	Large-scale vision (tuned)
Adam / AdamW	1st + 2nd moments, bias-corrected	Yes	Transformers, sparse gradients, fast convergence

Read the implementation

python

1import torch
2from torch import Tensor
3
4
5@torch.no_grad()
6def sgd_momentum_step(p: Tensor, grad: Tensor, v: Tensor, lr=0.01, beta=0.9) -> Tensor:
7    v.mul_(beta).add_(grad)            # v = beta*v + grad
8    p.add_(v, alpha=-lr)              # p = p - lr*v
9    return v
10
11
12class Adam:
13    """Dense-gradient example; all parameters are expected to update each step."""
14    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8):
15        self.params = list(params)
16        self.lr, (self.b1, self.b2), self.eps = lr, betas, eps
17        self.m = [torch.zeros_like(p) for p in self.params]
18        self.v = [torch.zeros_like(p) for p in self.params]
19        self.t = 0
20
21    @torch.no_grad()
22    def step(self):
23        self.t += 1
24        for i, p in enumerate(self.params):
25            g = p.grad
26            if g is None:
27                continue
28            self.m[i].mul_(self.b1).add_(g, alpha=1 - self.b1)
29            self.v[i].mul_(self.b2).addcmul_(g, g, value=1 - self.b2)
30            m_hat = self.m[i] / (1 - self.b1**self.t)      # bias correction
31            v_hat = self.v[i] / (1 - self.b2**self.t)
32            p.addcdiv_(m_hat, v_hat.sqrt().add_(self.eps), value=-self.lr)
33
34
35w = torch.randn(4, requires_grad=True)
36opt = Adam([w])
37(w.pow(2).sum()).backward()           # toy loss = ||w||²
38opt.step()

Questions and trade-offs

Conceptual: Why use mini-batch SGD instead of full-batch gradient descent? (Far cheaper per step, uses hardware efficiently, and gradient noise changes optimization and can help generalization, but does not guarantee it.)
Implementation: What does momentum add to plain SGD? (A velocity term (an exponentially weighted sum of past gradients) that accelerates consistent directions and damps oscillation in narrow valleys.)
Applied: What two statistics does Adam track, and what does each buy you? (First moment = momentum; second moment = per-parameter adaptive scaling of the step.)
Systems-level: Why is bias correction needed in Adam? (m and v start at zero, so early estimates are biased low; dividing each moment by its 1−βᵗ factor corrects initialization bias; it does not simply enlarge every update.)
Failure modes: When might SGD-with-momentum generalize better than Adam? (Some vision benchmarks favor tuned SGD+momentum, but the result depends on the learning-rate schedule, regularization, data, and compute budget. Compare held-out performance rather than assuming an optimizer guarantees generalization.)

Check your understanding

Without looking: write the update rule for SGD, SGD+momentum, and Adam. State what β₁ and β₂ control in Adam and why bias correction exists. Then explain how you would compare SGD and Adam fairly. Compare against Stage 3.

Sources and further reading

Kingma and Ba, Adam: moment estimates, bias correction, and update rules.
Loshchilov and Hutter, Decoupled Weight Decay Regularization: AdamW and optimizer-dependent weight decay.

Oliver Perrin

Machine Learning Engineer · Founder, LiminalML

LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.

Keep exploring the idea.

Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.

Ask a follow-up See pricing

Related concepts

Deep Learning

Backpropagation

Trace local derivatives backward through a computation and connect the chain rule to a manual implementation.

Classical ML

Bias-Variance Tradeoff

Separate systematic error from sensitivity to the training sample. Read an error decomposition without assuming every model follows a U-shaped curve.

Deep Learning

Batch Normalization

Normalize activations across a batch, apply a learned scale and shift, and understand why training and evaluation behave differently.

Gradient Descent (SGD, Momentum, Adam)

Follow updates across a loss surface, then connect learning rate, momentum, and Adam to the update equations.

8 min readReviewed September 2026By Oliver Perrin · Machine Learning Engineer

Learn with a guided session

Jump to a section

Why this matters

Build the mental model

Explore the ideaRelated Studio experiment

Follow the negative gradient

The update moves against the slope of one exact quadratic.

Exact inputs and equations appear with the figure.

Preparing the scientific figure…

Drag to rotate

Exact updates on the positive quadratic shown. This is not a measured training run, and it does not represent a neural-network loss landscape. No stochastic minibatches are sampled.

Authored analytic example

Experiment with related Studio inputs

Replay the path at a small learning rate, then increase it toward 1. Rotate the surface to inspect the overshoot.

An exact quadratic example with one minimum. Momentum, Adam, and stochastic gradients are discussed below but are not simulated here.

Work through the math

SGD with learning rate $\eta$ , on a mini-batch gradient $g_t = \nabla_\theta L_t$ :

\theta_{t+1} = \theta_t - \eta\, g_t

SGD with momentum ( $\beta \approx 0.9$ ): maintain a velocity $v$ that is an exponentially weighted sum of past gradients:

v_{t} = \beta v_{t-1} + g_t, \qquad \theta_{t+1} = \theta_t - \eta\, v_t

Adam combines a first moment (mean, like momentum) and second raw moment (mean squared gradient, not centered variance) with bias correction:

m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2

\hat{m}_t = \frac{m_t}{1-\beta_1^{t}}, \qquad \hat{v}_t = \frac{v_t}{1-\beta_2^{t}}, \qquad \theta_{t+1} = \theta_t - \eta\,\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

Key takeaway: Every variant shares one skeleton: estimate a gradient on a batch, take a step. Momentum shapes the step with gradient history; Adam rescales it per parameter; bias correction compensates for moment estimates initialized at zero.

Optimizer	Update uses	Per-parameter adaptivity	Where it shines
Batch GD	Full dataset gradient	No	Problems where full-gradient steps are affordable
Mini-batch SGD	Small-batch gradient	No	Scalable baseline; requires tuning
SGD + momentum	Exponentially weighted sum of past gradients	No	Large-scale vision (tuned)
Adam / AdamW	1st + 2nd moments, bias-corrected	Yes	Transformers, sparse gradients, fast convergence

Read the implementation

python

1import torch
2from torch import Tensor
3
4
5@torch.no_grad()
6def sgd_momentum_step(p: Tensor, grad: Tensor, v: Tensor, lr=0.01, beta=0.9) -> Tensor:
7    v.mul_(beta).add_(grad)            # v = beta*v + grad
8    p.add_(v, alpha=-lr)              # p = p - lr*v
9    return v
10
11
12class Adam:
13    """Dense-gradient example; all parameters are expected to update each step."""
14    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8):
15        self.params = list(params)
16        self.lr, (self.b1, self.b2), self.eps = lr, betas, eps
17        self.m = [torch.zeros_like(p) for p in self.params]
18        self.v = [torch.zeros_like(p) for p in self.params]
19        self.t = 0
20
21    @torch.no_grad()
22    def step(self):
23        self.t += 1
24        for i, p in enumerate(self.params):
25            g = p.grad
26            if g is None:
27                continue
28            self.m[i].mul_(self.b1).add_(g, alpha=1 - self.b1)
29            self.v[i].mul_(self.b2).addcmul_(g, g, value=1 - self.b2)
30            m_hat = self.m[i] / (1 - self.b1**self.t)      # bias correction
31            v_hat = self.v[i] / (1 - self.b2**self.t)
32            p.addcdiv_(m_hat, v_hat.sqrt().add_(self.eps), value=-self.lr)
33
34
35w = torch.randn(4, requires_grad=True)
36opt = Adam([w])
37(w.pow(2).sum()).backward()           # toy loss = ||w||²
38opt.step()

Questions and trade-offs

Conceptual: Why use mini-batch SGD instead of full-batch gradient descent? (Far cheaper per step, uses hardware efficiently, and gradient noise changes optimization and can help generalization, but does not guarantee it.)
Implementation: What does momentum add to plain SGD? (A velocity term (an exponentially weighted sum of past gradients) that accelerates consistent directions and damps oscillation in narrow valleys.)
Applied: What two statistics does Adam track, and what does each buy you? (First moment = momentum; second moment = per-parameter adaptive scaling of the step.)
Systems-level: Why is bias correction needed in Adam? (m and v start at zero, so early estimates are biased low; dividing each moment by its 1−βᵗ factor corrects initialization bias; it does not simply enlarge every update.)
Failure modes: When might SGD-with-momentum generalize better than Adam? (Some vision benchmarks favor tuned SGD+momentum, but the result depends on the learning-rate schedule, regularization, data, and compute budget. Compare held-out performance rather than assuming an optimizer guarantees generalization.)

Check your understanding

Sources and further reading

Kingma and Ba, Adam: moment estimates, bias correction, and update rules.
Loshchilov and Hutter, Decoupled Weight Decay Regularization: AdamW and optimizer-dependent weight decay.

Oliver Perrin

Machine Learning Engineer · Founder, LiminalML

LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.

Keep exploring the idea.

Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.

Ask a follow-up See pricing

Related concepts

Deep Learning

Backpropagation

Trace local derivatives backward through a computation and connect the chain rule to a manual implementation.

Classical ML

Bias-Variance Tradeoff

Separate systematic error from sensitivity to the training sample. Read an error decomposition without assuming every model follows a U-shaped curve.

Deep Learning

Batch Normalization

Normalize activations across a batch, apply a learned scale and shift, and understand why training and evaluation behave differently.