How does boosting differ from bagging or random forests?

Boosting adds sequential corrections to an ensemble; bagging averages separately trained models. Reducing bias and reducing variance are common motivations, not exclusive guarantees.

What does each new tree fit in gradient boosting?

The negative gradient of the loss at the current predictions, which, for squared-error loss, equals the residuals.

What does the learning rate control in gradient boosting?

It sets each tree's contribution. Smaller values usually require more trees and can improve generalization, but that is not guaranteed; tune shrinkage and tree count together using validation.

Why is XGBoost so effective?

It uses second-order (gradient + Hessian) optimization, regularization on leaf weights and tree complexity, and engineering like histogram-based splits and parallelized tree construction.

Gradient Boosting & XGBoost

See one residual-fitting update, then connect the additive model to gradient-based boosting for other losses.

8 min readReviewed September 2026By Oliver Perrin · Machine Learning Engineer

Learn with a guided session

Jump to a section

Why this matters

Gradient boosting builds an additive predictor one weak learner at a time. Each new learner is fitted to a signal derived from the current model's loss, and a learning rate controls how much of that correction is added.

For squared error, that signal is the residual. For other losses, it is the negative gradient with respect to the current predictions. This is why residual fitting is a useful starting example but not a complete description of every boosting method.

Build the mental model

Boosting is sequential error-correction. The first model makes a crude prediction; you compute its residuals (what it missed); a second small tree predicts those residuals; you add a fraction of it to the prediction; recompute residuals; repeat. Each step chips away at the remaining error. Contrast with bagging (random forests), which trains many trees independently in parallel and averages them to reduce variance: boosting builds dependent, corrective trees and can reduce bias. These are useful tendencies, not an exclusive division: both methods can affect bias and variance.

Explore the ideaRelated Studio experiment

Fit a residual correction

A single regression stump moves the initial mean toward two groups of targets.

Exact inputs and equations appear with the figure.

Preparing the scientific figure…

One squared-error stump on five observations; not the complete XGBoost algorithm.

Authored analytic example

Experiment with related Studio inputs

The initial mean is 2.2. A stump splits the first two observations from the last three; a learning rate of 0.5 moves each prediction halfway toward its group’s target.

One exact squared-error stump update on five synthetic observations. This does not represent every boosting loss or tree configuration.

Inspect the related Studio values

x,y,series
1,1,Target
1,2.2,Initial mean
1,1.6,After one stump
2,1,Target
2,2.2,Initial mean
2,1.6,After one stump
3,3,Target
3,2.2,Initial mean
3,2.6,After one stump
4,3,Target
4,2.2,Initial mean
4,2.6,After one stump
5,3,Target
5,2.2,Initial mean
5,2.6,After one stump

Work through the math

Build an additive model stage by stage. Starting from a constant $F_0$ , at step $m$ :

F_m(x) = F_{m-1}(x) + \eta\, h_m(x)

where $\eta$ is the learning rate (shrinkage) and $h_m$ is the new weak learner. Gradient boosting fits $h_m$ to the negative gradient of the loss $L$ evaluated at the current predictions: the "pseudo-residuals":

r_{im} = -\left[\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)}\right]_{F = F_{m-1}}

For squared-error loss $L = \tfrac{1}{2}(y - F)^2$ , this gradient is exactly $r_{im} = y_i - F_{m-1}(x_i)$ : the ordinary residuals, which is why "fit the next tree to the residuals" is the intuition. Smaller $\eta$ means each tree contributes less, so you usually need more trees; whether this improves generalization must be checked on held-out data. XGBoost extends this with a second-order (Newton) approximation using gradients and Hessians, plus explicit $L_1/L_2$ regularization on leaf weights and tree complexity, which provide controls on model complexity without making it immune to overfitting. Regularize with learning rate, tree depth, subsampling of rows/columns, and the number of trees (early stopping).

Key takeaway: Boosting fits each new tree to the negative gradient of the loss (the residuals, for squared error) and adds a shrunken copy: sequential error-correction, while bagging averages separately trained models. Bias and variance effects depend on the data, learners, and regularization.

	Bagging / Random Forest	Boosting / XGBoost
Tree training	Independent, parallel	Sequential, dependent on prior trees
Each tree fits	Bootstrap sample of the data	Negative gradient (residuals)
Typical motivation	Reduce variance by averaging	Reduce error through successive corrections
Generalization controls	Tree size, feature sampling, ensemble size	Shrinkage, depth, subsampling, early stopping
Tabular-data role	Useful baseline to validate	Useful candidate to validate

Read the implementation

python

1import numpy as np
2from sklearn.tree import DecisionTreeRegressor
3
4rng = np.random.default_rng(0)
5X = rng.uniform(-3, 3, size=(300, 1))
6y = np.sin(X[:, 0]) + rng.normal(0, 0.1, 300)
7
8lr, n_trees = 0.1, 100
9pred = np.full(len(y), y.mean())            # F_0 = mean
10trees = []
11for _ in range(n_trees):
12    residuals = y - pred                     # = negative gradient for sq. loss
13    tree = DecisionTreeRegressor(max_depth=2).fit(X, residuals)  # weak learner
14    pred += lr * tree.predict(X)             # F_(m+1) = F_m + lr·h
15    trees.append(tree)
16
17mse = np.mean((y - pred) ** 2)
18print(f"train MSE after {n_trees} trees: {mse:.3f}")

Questions and trade-offs

Conceptual: How does boosting differ from bagging / random forests? (Boosting builds trees sequentially, each correcting the ensemble; bagging trains trees separately and averages them. Bias reduction and variance reduction are common motivations, not exclusive guarantees.)
Implementation: What does each new tree fit in gradient boosting? (The negative gradient of the loss at current predictions: the residuals, for squared-error loss.)
Applied: What does the learning rate control, and what's the tradeoff? (Shrinkage: how much each tree contributes. Smaller contributions usually need more trees; tune both together using validation rather than assuming smaller always generalizes better.)
Systems-level: Why is XGBoost both accurate and fast? (Second-order (gradient + Hessian) optimization, regularization on leaf weights, and engineering: histogram-based splits, sparsity awareness, and parallelized tree construction.)
Failure modes: Why can gradient boosting overfit, and how do you prevent it? (It keeps reducing training error tree by tree; control with learning rate, shallow trees, row/column subsampling, L1/L2 regularization, and early stopping on a validation set.)

Check your understanding

Without looking: write the additive update $F_m = F_{m-1} + \eta h_m$ , state what each tree fits (the negative gradient / residuals), and contrast boosting with bagging in one sentence. Check against Stage 3.

Sources and further reading

scikit-learn, Gradient boosting: loss gradients, shrinkage, and regularization.
Chen and Guestrin, XGBoost: second-order objectives and scalable tree construction.

Oliver Perrin

Machine Learning Engineer · Founder, LiminalML

LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.

Keep exploring the idea.

Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.

Ask a follow-up See pricing

Related concepts

Classical ML

Bias-Variance Tradeoff

Separate systematic error from sensitivity to the training sample. Read an error decomposition without assuming every model follows a U-shaped curve.

Classical ML

Logistic Regression

Connect a linear score to a sigmoid probability, a cross-entropy loss, and a decision threshold.

Classical ML

Principal Component Analysis (PCA)

Compare centered data with its projection onto a principal direction. Understand what variance retention does and does not preserve.

Gradient Boosting & XGBoost

See one residual-fitting update, then connect the additive model to gradient-based boosting for other losses.

8 min readReviewed September 2026By Oliver Perrin · Machine Learning Engineer

Learn with a guided session

Jump to a section

Why this matters

Build the mental model

Explore the ideaRelated Studio experiment

Fit a residual correction

A single regression stump moves the initial mean toward two groups of targets.

Exact inputs and equations appear with the figure.

Preparing the scientific figure…

One squared-error stump on five observations; not the complete XGBoost algorithm.

Authored analytic example

Experiment with related Studio inputs

The initial mean is 2.2. A stump splits the first two observations from the last three; a learning rate of 0.5 moves each prediction halfway toward its group’s target.

One exact squared-error stump update on five synthetic observations. This does not represent every boosting loss or tree configuration.

Inspect the related Studio values

x,y,series
1,1,Target
1,2.2,Initial mean
1,1.6,After one stump
2,1,Target
2,2.2,Initial mean
2,1.6,After one stump
3,3,Target
3,2.2,Initial mean
3,2.6,After one stump
4,3,Target
4,2.2,Initial mean
4,2.6,After one stump
5,3,Target
5,2.2,Initial mean
5,2.6,After one stump

Work through the math

Build an additive model stage by stage. Starting from a constant $F_0$ , at step $m$ :

F_m(x) = F_{m-1}(x) + \eta\, h_m(x)

r_{im} = -\left[\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)}\right]_{F = F_{m-1}}

Key takeaway: Boosting fits each new tree to the negative gradient of the loss (the residuals, for squared error) and adds a shrunken copy: sequential error-correction, while bagging averages separately trained models. Bias and variance effects depend on the data, learners, and regularization.

	Bagging / Random Forest	Boosting / XGBoost
Tree training	Independent, parallel	Sequential, dependent on prior trees
Each tree fits	Bootstrap sample of the data	Negative gradient (residuals)
Typical motivation	Reduce variance by averaging	Reduce error through successive corrections
Generalization controls	Tree size, feature sampling, ensemble size	Shrinkage, depth, subsampling, early stopping
Tabular-data role	Useful baseline to validate	Useful candidate to validate

Read the implementation

python

1import numpy as np
2from sklearn.tree import DecisionTreeRegressor
3
4rng = np.random.default_rng(0)
5X = rng.uniform(-3, 3, size=(300, 1))
6y = np.sin(X[:, 0]) + rng.normal(0, 0.1, 300)
7
8lr, n_trees = 0.1, 100
9pred = np.full(len(y), y.mean())            # F_0 = mean
10trees = []
11for _ in range(n_trees):
12    residuals = y - pred                     # = negative gradient for sq. loss
13    tree = DecisionTreeRegressor(max_depth=2).fit(X, residuals)  # weak learner
14    pred += lr * tree.predict(X)             # F_(m+1) = F_m + lr·h
15    trees.append(tree)
16
17mse = np.mean((y - pred) ** 2)
18print(f"train MSE after {n_trees} trees: {mse:.3f}")

Questions and trade-offs

Conceptual: How does boosting differ from bagging / random forests? (Boosting builds trees sequentially, each correcting the ensemble; bagging trains trees separately and averages them. Bias reduction and variance reduction are common motivations, not exclusive guarantees.)
Implementation: What does each new tree fit in gradient boosting? (The negative gradient of the loss at current predictions: the residuals, for squared-error loss.)
Applied: What does the learning rate control, and what's the tradeoff? (Shrinkage: how much each tree contributes. Smaller contributions usually need more trees; tune both together using validation rather than assuming smaller always generalizes better.)
Systems-level: Why is XGBoost both accurate and fast? (Second-order (gradient + Hessian) optimization, regularization on leaf weights, and engineering: histogram-based splits, sparsity awareness, and parallelized tree construction.)
Failure modes: Why can gradient boosting overfit, and how do you prevent it? (It keeps reducing training error tree by tree; control with learning rate, shallow trees, row/column subsampling, L1/L2 regularization, and early stopping on a validation set.)

Check your understanding

Sources and further reading

scikit-learn, Gradient boosting: loss gradients, shrinkage, and regularization.
Chen and Guestrin, XGBoost: second-order objectives and scalable tree construction.

Oliver Perrin

Machine Learning Engineer · Founder, LiminalML

LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.

Keep exploring the idea.

Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.

Ask a follow-up See pricing

Related concepts

Classical ML

Bias-Variance Tradeoff

Separate systematic error from sensitivity to the training sample. Read an error decomposition without assuming every model follows a U-shaped curve.

Classical ML

Logistic Regression

Connect a linear score to a sigmoid probability, a cross-entropy loss, and a decision threshold.

Classical ML

Principal Component Analysis (PCA)

Compare centered data with its projection onto a principal direction. Understand what variance retention does and does not preserve.