Random Forest

A random forest is an ensemble of decision trees trained on bootstrap samples of the data, with random feature subsets at each split. Final predictions aggregate trees by majority vote (classification) or averaging (regression).

Intuition

A single tree captures nonlinear interactions but overfits easily (high variance). Random forest applies bagging: train many trees on different subsamples and average them to reduce variance without increasing bias as much.

Randomizing features at each split decorrelates trees — a necessary condition for aggregation to genuinely improve generalization over one deep tree.

How it works

Bootstrap
: for each tree , draw observations with replacement from (about 63% unique observations per tree).
Unpruned tree: grow until depth or stopping criteria (max_depth, min_samples_leaf, etc.).
Random subspace
: at each node, consider only a subset of max_features variables for the best split.
Aggregation: classification → majority class; regression → mean of tree predictions.

Out-of-bag (OOB) score: observations not in a tree's bootstrap sample act as an implicit validation set. With oob_score=True, you get a performance estimate without a dedicated validation set — useful for prototyping, but not a substitute for rigorous cross-validation.

Key hyperparameters

Parameter	Role	Typical values
`n_estimators`	Number of trees in the forest	100–500; diminishing returns beyond
`max_depth`	Maximum depth of each tree	`None` (unlimited) or 10–30 depending on noise
`min_samples_leaf`	Minimum leaf size	1–10; higher = more regularization
`min_samples_split`	Minimum samples to split a node	2 (default) to 20+ on noisy data
`max_features`	Candidate variables per split	`sqrt` (classif.), `1/3` (regression); decorrelates trees
`bootstrap`	Sampling with replacement	`True` (default); `False` → each tree sees all data
`oob_score`	Out-of-bag evaluation	`True` when `bootstrap=True`
`class_weight`	Class imbalance weighting	`balanced` or weight dictionary
`n_jobs`	Parallel training	`-1` for all cores

Strengths and limitations

Strengths

Robust to outliers and feature scales (no mandatory normalization)
Captures interactions and nonlinearities without explicit feature engineering
Built-in feature importance; free OOB score
Parallelizable; relatively few critical hyperparameters

Limitations

Less interpretable than a single tree
Memory and latency grow with n_estimators
Regression: poor extrapolation outside the training domain
Can be outperformed by gradient boosting on structured tabular data
Sensitive to concept drift and non-iid data without adaptation

Model	Principle	vs Random Forest
Decision tree	Single tree, deterministic splits	More interpretable, much higher variance
Extra Trees	Random splits (random thresholds among candidates)	More decorrelation, often faster, sometimes slightly less accurate
Gradient Boosting (XGBoost…)	Sequential trees fitting residuals	Often better on tabular competitions; more tuning-sensitive and overfit-prone
Logistic / linear regression	Linear model	Fast interpretable baseline; misses complex interactions
k-NN	Local neighborhood	Metric- and dimension-sensitive; high prediction latency

Evaluation and tuning

Step	Recommendation
Data splitting	Train / validation / test; for time series: walk-forward or purged CV — see cross-validation
Metrics	Classification: accuracy, F1, ROC AUC, log loss by business cost; regression: RMSE, MAE,
Hyperparameter search	Start with `n_estimators` and `max_depth`; refine `min_samples_leaf` and `max_features` via grid or random search
OOB	Quick indicator during prototyping; validate on hold-out or CV before deployment
Overfitting	Increase `min_samples_leaf`, reduce `max_depth`; check train/validation gap

Feature importance

Method	Principle	Limits
Impurity (Gini / MSE)	Mean impurity decrease per feature across trees (`feature_importances_`)	Biased toward high-cardinality features; correlated features
Permutation importance	Performance drop when a feature is randomly permuted	More reliable; higher compute cost
SHAP	Shapley values per observation and feature	Local and global interpretation — see ML for markets

Applications

I will add concrete, precise examples in the future.

References

• scikit-learn —

RandomForestClassifier

RandomForestRegressor

• cuML —