Bias-Variance Tradeoff

A model can fail because its assumptions are too restrictive or because its fit changes too much with the training sample. Bias describes systematic error; variance describes sensitivity to the sampled data.

For squared error, the expected prediction error separates into bias squared, variance, and irreducible noise under the usual assumptions. The familiar U-shaped curve is a useful illustration, not a universal rule for every model.

The dartboard analogy: bias is how far your cluster of shots sits from the bullseye; variance is how spread out the shots are. Four regimes:

• Low bias, low variance: tight cluster on target (the goal).
• High bias, low variance: tight cluster, wrong spot (underfit).
• Low bias, high variance: centered on average but scattered (overfit).
• High bias, high variance: scattered and off-target (worst case).

In the classical illustration, total error traces a U-shape as complexity rises: shrinking bias initially helps, then growing variance dominates. This is a possible pattern, not a law of model complexity.

At a fixed test input $x$, let $y = f(x) + \epsilon$, where $f(x)=\mathbb{E}[y\mid x]$, $\mathbb{E}[\epsilon\mid x]=0$, and $\text{Var}(\epsilon\mid x)=\sigma^2(x)$. For a model $\hat{f}(x)$ fitted to a random training set independent of the test noise, expected squared error decomposes as:

Reading each term:

• Bias: how far the average prediction (over many training sets) is from $f(x)$. A linear model fitted to a nonlinear relationship can have substantial bias.
• Variance: how much the prediction changes with the training set. Highly flexible fits on small datasets can have substantial variance.
• Irreducible: $\sigma^2(x)$, the conditional noise variance given these input features. It is a lower bound on expected squared prediction error in this setting, not on every realized test error.

The left-hand expectation includes training-set randomness and independent test noise; the bias and variance terms average over fitted models. Neither finite data nor greater complexity requires a strict exchange between bias and variance. More representative data can reduce variance without increasing bias.

Key takeaway: Training and validation errors are diagnostic clues, not measurements of the decomposition. Noise, optimization failure, distribution shift, and leakage can also affect them. Check those possibilities before deciding a model simply needs more or less capacity.

Diagnosis to prescription:

A single-split illustration: fit polynomials of increasing degree and compare train and test error. Estimating bias and variance separately requires repeating training on independent datasets and comparing their predictions with the known signal.

1import numpy as np
2from numpy.polynomial import polynomial as P
3
4rng = np.random.default_rng(0)
5
6def true_f(x):
7    return np.sin(2 * np.pi * x)
8
9def make_data(n):
10    x = rng.uniform(0, 1, n)
11    y = true_f(x) + rng.normal(0, 0.2, n)   # signal + irreducible noise
12    return x, y
13
14x_train, y_train = make_data(20)
15x_test, y_test = make_data(500)
16
17for degree in (1, 3, 9, 15):
18    coeffs = P.polyfit(x_train, y_train, degree)
19    train_mse = np.mean((P.polyval(x_train, coeffs) - y_train) ** 2)
20    test_mse = np.mean((P.polyval(x_test, coeffs) - y_test) ** 2)
21    print(f"degree {degree:>2}: train={train_mse:.3f}  test={test_mse:.3f}")
22
23# degree  1: high train + high test  -> underfit (high bias)
24# degree  3: low train  + low test   -> good fit (degree 9 tests about the same here)
25# degree 15: ~0 train   + high test   -> overfit (high variance)

1. Conceptual: Define bias and variance in one sentence each, in terms of training-set randomness. (Bias: error of the average prediction vs. truth. Variance: how much the prediction changes across different training sets.)
2. Implementation: What do training and validation errors tell you? (High training error suggests checking fit, optimization, and noise. A large gap suggests checking overfitting and distribution mismatch. Neither directly measures bias or variance.)
3. Applied: You're overfitting. Name three levers and their possible effects. (More representative data can reduce variance without increasing bias. Stronger regularization or a simpler model can reduce variance while increasing bias; validate the effect.)
4. Systems-level: How does cross-validation relate to this decomposition? (It estimates generalization performance under appropriate sampling assumptions. Fold scores are correlated, and their spread is not the prediction-variance term in the decomposition.)
5. Failure modes: Does the classic U-curve always hold? (Not always: deep, overparameterized nets can show "double descent," where test error falls again past the interpolation threshold. Know the exception exists.)

From memory: write the squared-error decomposition and its assumptions, define each term, and explain why training/validation errors do not directly measure bias and variance. Then name one exception to the U-curve. Check against Stages 3 and 5.

Sources and further reading

• scikit-learn, Bias-variance decomposition example: repeated-dataset estimates of bias, variance, and noise.
• Belkin et al., Reconciling modern machine learning practice and the bias-variance trade-off: double descent and the limits of the classical U-curve.
• scikit-learn, Cross-validation: split assumptions and evaluating generalization.