Batch Normalization

Batch normalization rescales activations using statistics computed across a mini-batch, then applies a learned scale and shift. For an input shaped (N, C), it treats each feature column separately.

Its most important distinction is training versus inference. Training uses batch statistics; evaluation normally uses stored running estimates. Confusing those modes can change predictions even when the model weights stay fixed.

For a batch of activations shaped (N, C) (N examples, C features) BN computes the mean and variance down each column (across the N examples) and standardizes that feature. Every feature ends up centered and scaled, regardless of what the previous layer's weights are doing. Then two learnable parameters, $\gamma$ and $\beta$, rescale and re-shift, so if the network is better off not normalizing a feature, it can learn $\gamma = \sqrt{\text{Var}}$, $\beta = \text{mean}$ and recover the original.

Contrast with Layer Norm, which normalizes across the features of a single example (along C, per row): independent of batch size, which is why Transformers use LN, not BN.

For a mini-batch $\mathcal{B} = \{x_1, \dots, x_N\}$ (per feature), BN computes:

$\epsilon$ is a small constant for numerical stability. $\gamma$ and $\beta$ are learned per feature.

Train vs. eval: the critical distinction. At training time, $\mu_\mathcal{B}$ and $\sigma_\mathcal{B}^2$ are computed from the current batch. The layer also maintains running estimates via an exponential moving average:

At inference, BN uses these fixed running statistics, not the batch, so a single example (or any batch size) produces deterministic outputs. Forgetting to switch to eval mode (model.eval()) is a classic bug: predictions then depend on whatever else is in the batch.

Key takeaway: Batch Norm is two different behaviors wearing one layer: batch statistics while training, frozen running statistics at inference. The model.eval() switch is not optional; it changes what the layer computes.

The normalization-family landscape, and when each applies:

1. Conceptual: What does Batch Norm actually normalize, and along which axis? (Each feature, across the examples in the batch: per-column for (N, C) input.)
2. Implementation: What changes between training and inference, and why? (Train uses batch stats + updates running averages; eval uses frozen running stats so output is independent of batch composition.)
3. Applied: Why does BN struggle with very small batch sizes, and what would you use instead? (Batch stats become noisy/unreliable; use Group Norm or Layer Norm, which don't depend on batch size.)
4. Systems-level: Why do Transformers use Layer Norm rather than Batch Norm? (LN normalizes per-example across features: independent of batch size and sequence length, and well-behaved for variable-length sequences and small/streaming batches.)
5. Failure modes: Your model trains well but predictions are unstable in production. What's a likely BN-related cause? (Forgot model.eval(), so BN uses batch stats at inference: outputs depend on what else is batched together.)

Without looking: write the four BN equations (mean, variance, normalize, scale-shift), name what $\gamma$ and $\beta$ are for, and explain in one sentence exactly what differs at inference time. Then compare against Stage 3.