Gradient Boosting & XGBoost

Gradient boosting builds an additive predictor one weak learner at a time. Each new learner is fitted to a signal derived from the current model's loss, and a learning rate controls how much of that correction is added.

For squared error, that signal is the residual. For other losses, it is the negative gradient with respect to the current predictions. This is why residual fitting is a useful starting example but not a complete description of every boosting method.

Boosting is sequential error-correction. The first model makes a crude prediction; you compute its residuals (what it missed); a second small tree predicts those residuals; you add a fraction of it to the prediction; recompute residuals; repeat. Each step chips away at the remaining error. Contrast with bagging (random forests), which trains many trees independently in parallel and averages them to reduce variance: boosting reduces bias by building dependent, corrective trees.

Build an additive model stage by stage. Starting from a constant $F_0$, at step $m$:

where $\eta$ is the learning rate (shrinkage) and $h_m$ is the new weak learner. Gradient boosting fits $h_m$ to the negative gradient of the loss $L$ evaluated at the current predictions: the "pseudo-residuals":

For squared-error loss $L = \tfrac{1}{2}(y - F)^2$, this gradient is exactly $r_{im} = y_i - F_{m-1}(x_i)$: the ordinary residuals, which is why "fit the next tree to the residuals" is the intuition. Smaller $\eta$ means each tree contributes less, so you need more trees but generalize better. XGBoost extends this with a second-order (Newton) approximation using gradients and Hessians, plus explicit $L_1/L_2$ regularization on leaf weights and tree complexity, which is why it's both accurate and resistant to overfitting. Regularize with learning rate, tree depth, subsampling of rows/columns, and the number of trees (early stopping).

Key takeaway: Boosting fits each new tree to the negative gradient of the loss (the residuals, for squared error) and adds a shrunken copy: sequential error-correction that reduces bias, where bagging averages independent trees to reduce variance.

1. Conceptual: How does boosting differ from bagging / random forests? (Boosting builds trees sequentially, each correcting the last (reduces bias); bagging trains trees independently in parallel and averages them (reduces variance).)
2. Implementation: What does each new tree fit in gradient boosting? (The negative gradient of the loss at current predictions: the residuals, for squared-error loss.)
3. Applied: What does the learning rate control, and what's the tradeoff? (Shrinkage: how much each tree contributes. Smaller means slower learning, more trees needed, but better generalization.)
4. Systems-level: Why is XGBoost both accurate and fast? (Second-order (gradient + Hessian) optimization, regularization on leaf weights, and engineering: histogram-based splits, sparsity awareness, and parallelized tree construction.)
5. Failure modes: Why can gradient boosting overfit, and how do you prevent it? (It keeps reducing training error tree by tree; control with learning rate, shallow trees, row/column subsampling, L1/L2 regularization, and early stopping on a validation set.)

Without looking: write the additive update $F_m = F_{m-1} + \eta h_m$, state what each tree fits (the negative gradient / residuals), and contrast boosting with bagging in one sentence. Check against Stage 3.