XGBoost

XGBoost (Extreme Gradient Boosting) is a gradient boosting method built on decision trees. Unlike the bagging used in Random Forest, trees are trained sequentially: each new tree corrects part of the current model's errors.

Intuition

The goal is not to average independent trees, but to build an additive ensemble:

a first tree captures a coarse signal;
following trees learn residual errors;
the sum of trees produces a function more expressive than a single tree.

XGBoost adds explicit regularization, efficient handling of tabular data, and training optimizations that make it a practical reference on many structured supervised problems.

How it works

Initialization: start from a constant prediction (mean for regression, log-odds for classification).
Gradient computation: at each iteration, measure the direction in which the loss should be corrected.
Weak tree: train a small tree on these gradients / pseudo-residuals.
Additive update: add this tree to the ensemble with a weight controlled by learning_rate.
Repeat: continue until n_estimators is reached or a stopping rule is triggered, ideally with early_stopping.

In practice, XGBoost optimizes an objective made of:

a fit loss (prediction error),
tree complexity control (depth, leaves),
shrinkage through learning_rate.

This explains why XGBoost can reach very strong performance, but also why it usually requires more tuning than a random forest.

Key hyperparameters

Parameter	Role	Main effect
`n_estimators`	Number of added trees	Larger = more expressive, but slower and riskier without early stopping
`learning_rate`	Weight of each tree	Smaller = more stable learning, often needs more trees
`max_depth`	Maximum depth	Controls local complexity and overfitting
`min_child_weight`	Minimum child node weight / size	Prevents overly specific splits
`subsample`	Row subsampling	Reduces variance and cost; too low can add noise
`colsample_bytree`	Feature subsampling	Partly decorrelates trees and regularizes
`reg_alpha`	penalty	Encourages sparser contributions
`reg_lambda`	penalty	Stabilizes weights and limits extremes
`gamma`	Minimum gain required to split	Prevents low-value splits

Strengths and limitations

Strengths

Very strong performance on structured tabular data
Captures nonlinearities and complex feature interactions
Rich regularization and fine control of the bias-variance trade-off
Handles heterogeneous tabular features well
Often outperforms Random Forest in raw predictive accuracy

Limitations

More tuning-sensitive and more prone to overfitting
Less simple to explain and stabilize than a linear baseline or a forest
Can behave poorly if validation is poorly designed
Higher training cost, especially with hyperparameter search
Like other tabular models, it does not solve concept drift or non-iid issues by itself

Model	Principle	vs XGBoost
Random Forest	Bagging of independent trees	More robust and simpler; often less accurate but less demanding to tune
LightGBM	Histogram-based boosting optimized for speed	Often faster on large tabular datasets; different behavior on smaller ones
CatBoost	Boosting optimized for categorical variables	Often more natural when many categorical features are present
Linear / logistic regression	Linear baseline	Less expressive, but more interpretable and stable

Evaluation and tuning

Step	Recommendation
Data splitting	Train / validation / test; in time settings: walk-forward or purged CV — see cross-validation
Starting point	Start with `learning_rate`, `n_estimators`, `max_depth`, then refine `subsample` and `colsample_bytree`
Early stopping	Practically essential to avoid adding too many trees
Metrics	Classification: log loss, ROC AUC, PR AUC, F1 depending on the problem; regression: RMSE, MAE
Main risk	Over-tuning on the validation set or leakage if splits are poorly designed

Feature importance

Method	Principle	Limits
Gain	Average feature contribution to loss reduction	May overvalue some correlated variables
Cover / weight	Frequency or coverage of splits involving the feature	Less informative about actual predictive impact
Permutation	Performance drop after random permutation	More expensive, but often more robust
SHAP	Local decomposition of each feature contribution	Richer interpretation; see ML for markets

Applications

I will add concrete, precise examples in the future.

References

• XGBoost — general documentation

• XGBoost Python API —

XGBClassifier

XGBRegressor

• cuML FIL — Forest Inference Library for GPU inference on models trained with XGBoost