Metrics

Condensed inventory of supervised ML evaluation metrics. Test performance metrics are usually model-agnostic; diagnostics (OOB, importances…) depend on the family. Protocol: train / validation / test, stratified CV when needed, baseline, uncertainty (bootstrap or CV), hold-out test once.

Master table

Metric	Measures	Prob	Threshold	Problem
Accuracy, F1, MCC	Errors @ τ	—	yes	Binary, multiclass
AUC-ROC, Gini (model)	Ranking	no	no	Binary
AUC-PR, AP	Ranking (rare class)	no	no	Binary
Log loss, Brier, ECE	Probs & calibration	yes	no	Binary, multiclass
Lift, quantile buckets, KS, PSI	Score ranking / stability	score	optional	Binary
MAE, RMSE, R²	Continuous error	—	—	Regression
OOB, Gini MDI, XGB gain	Model diagnostic	—	—	RF, XGBoost

Threshold & confusion matrix

Require a predicted class ŷ (threshold τ on p̂ in binary). Define the positive class (1).

Metric	Definition (binary)	Note
TP, TN, FP, FN	Counts @ τ	Threshold block basis
Accuracy	(TP+TN)/N	Weak under imbalance
Precision (PPV), Recall (TPR), Specificity (TNR)	TP/(TP+FP), TP/(TP+FN), TN/(TN+FP)	—
NPV, FDR, FOR	TN, FP variants	Medical, niche
F1, Fβ	Harmonic P/R	β > 1 → recall
Balanced accuracy, MCC, Cohen κ	½(TPR+TNR), Matthews, agreement	κ extends to multiclass
Expected cost, optimal τ	C_FP·FP + C_FN·FN	τ chosen on validation

Discrimination & ranking

Input: score or p̂ — no fixed τ. Gini (model) = 2×AUC−1 ≠ Gini MDI (RF importance).

Metric	Idea	Note
ROC, AUC-ROC	TPR vs FPR	Global ranking
Gini (model)	2×AUC−1	Credit scoring
PR curve, AUC-PR, AP	Precision vs recall	Rare class 1
Partial AUC, KS	FPR band; max\|CDF₁−CDF₀\|	Business / banking
Lift, cumulative gains	Positives in top k%	Campaigns, triage
Quantile buckets (deciles)	Observed rate, lift per bucket	Score monotonicity

Probabilistic & calibration

Require interpretable p̂. High AUC does not imply calibrated probabilities.

Metric	Role
Log loss	Cross-entropy; penalizes confident wrong probs
Brier score	MSE(y, p̂); decomposable (Murphy)
Reliability diagram, ECE, MCE	Observed vs predicted gap per bin
Sharpness	Spread of p̂ (≠ calibration)
Hosmer–Lemeshow	Goodness-of-fit test by bins
Recalibration (Platt, isotonic)	Post-hoc on validation

Multiclass classification

Y ∈ , K > 2.

Metric	Variant	Note
Accuracy, K×K matrix	—	Global
Precision / Recall / F1	macro, micro, weighted	Macro = equal classes
Log loss	multiclass	K-way probabilities
AUC	OvR, OvO	Macro-average
Top-k accuracy	—	Label in top k logits
Hamming loss	multi-label	Wrong labels

Regression

Y ∈ ℝ — distinct metric set.

Metric	Idea	Note
MAE, MedAE	\|y−ŷ\|, median	Robust (MedAE)
MSE, RMSE	(y−ŷ)²	Outlier-sensitive
MAPE, SMAPE	Relative % error	Unstable if y≈0
R², adjusted R²	Explained variance	Adjusted penalizes complexity
Pinball loss	Quantile τ	Quantile models
Interval score	Coverage + width	Prediction intervals

Comparison & robustness

Item	Use
Bootstrap CI, CV std dev	Uncertainty on AUC, F1…
McNemar, Delong, paired bootstrap	Compare A vs B (same examples)
Learning curves	Bias / variance vs train size
Out-of-time, FP/FN analysis	Temporal drift, error segments
PSI, calibration drift	p̂ distribution stability over time

Diagnostics — Random Forest

≠ test performance. See Random Forest.

Item	Note
OOB score / error	Internal perf; not a test/CV substitute
Gini importance (MDI)	Impurity decrease; biased if correlated
Permutation importance	More reliable out-of-sample
Gini impurity (node)	Tree build ≠ model Gini

Diagnostics — XGBoost

See XGBoost.

Item	Note
Gain / cover / weight importance	Weight favors high cardinality
Early stopping (eval metric)	logloss, auc, rmse… → # rounds
Train vs val curve	Overfitting by iteration
SHAP (optional)	Local interpretation

sklearn references

confusion_matrix, classification_report, roc_auc_score, average_precision_score, log_loss, brier_score_loss, CalibrationDisplay, CalibratedClassifierCV, f1_score, mean_absolute_error, r2_score, permutation_importance, StratifiedKFold