Linear regression
Linear regression models the relationship between a target variable and explanatory variables through a linear combination of coefficients.
In the multiple case:
where contains the features, the coefficients, and the error term.
Intuition
Linear regression searches for the best hyperplane approximating a continuous target from input variables. The standard criterion is the residual sum of squares.
In the simple case:
The coefficient measures the average change in associated with a one-unit increase in , holding other variables fixed in the multiple case.
Ordinary least squares estimation
The OLS estimator minimizes:
In matrix form, if is invertible:
This formula immediately shows why collinearity is problematic: if the columns of are linearly dependent, the inverse does not exist.
| Object | Formula | Role |
|---|---|---|
| Prediction | Fitted values from the model | |
| Residuals | Observed in-sample error | |
| RSS | Residual sum of squares | |
| Variance estimator | Estimated residual variance |
Model assumptions
| Assumption | Idea | Why it matters |
|---|---|---|
| Linearity | The model must be correctly specified in mean | |
| Exogeneity | Key condition for unbiasedness | |
| Independence | Errors are uncorrelated across observations | Important for classical inference |
| Homoscedasticity | Otherwise OLS standard errors are invalid | |
| No perfect collinearity | invertible | Condition for existence of |
| Normality of errors | Useful for exact small-sample tests, not required to compute OLS |
Theoretical properties
Under the standard assumptions, the OLS estimator is:
- unbiased:
- consistent as sample size increases under regularity conditions
- BLUE (Best Linear Unbiased Estimator) by the Gauss-Markov theorem
Its conditional variance is:
Under Gaussian errors, OLS also coincides with the maximum likelihood estimator.
Statistical inference
| Concept | Formula | Interpretation |
|---|---|---|
| Standard error | Estimated uncertainty on one coefficient | |
| t-statistic | Tests whether a coefficient is zero or equals a target value | |
| 95% CI | Plausible interval for the true coefficient | |
| Global F-test | Compare full and restricted models | Tests joint significance of several coefficients |
| p-value | Probability under of observing as extreme or more | Measures compatibility with the null, not effect size |
p-values are not enough to judge a model. You should also look at coefficient magnitude and sign, out-of-sample stability, business or economic coherence, and sensitivity to variables being added or removed.
Goodness of fit
| Metric | Formula / idea | Limitation |
|---|---|---|
| Share of explained variance | Mechanically increases when adding variables | |
Adjusted | Penalizes useless variables | Still an in-sample measure |
| RMSE | Root mean squared error | Sensitive to large errors |
| MAE | Mean absolute error | Less sensitive to outliers, but less smooth |
Diagnostics
| Diagnostic | Warning sign | Consequence |
|---|---|---|
| Residuals vs fitted | Systematic structure, curvature | Nonlinearity or misspecification |
| Heteroskedasticity | Residual spread grows with fitted values | Biased standard errors; prefer HC-robust estimates |
| Multicollinearity | High VIF, unstable coefficients | Harder interpretation, inflated variance |
| Influential points | High leverage, large Cook's distance | The model depends too much on a few observations |
| Autocorrelation | Time-correlated residuals | Frequent issue in time series settings |
Limits
- Causality does not follow from a significant linear correlation. - Outliers can strongly move coefficients. - The true relationship may be nonlinear or depend on omitted interactions. - With leakage or wrong temporal ordering, inference becomes misleading. - Linear regression extrapolates as a line or hyperplane, which can be economically absurd outside the observed region.
Common extensions
| Extension | Idea |
|---|---|
| Polynomial regression | Add powers of variables to capture curvature |
| Interactions | Allow the effect of one variable to depend on another |
| Ridge / Lasso / Elastic Net | Regularization to reduce variance and select variables |
| WLS / GLS | Handle heteroskedasticity or correlated errors |
| Logistic regression | Binary classification counterpart with a logit link instead of a continuous target |
Linear regression as an ML baseline
In tabular machine learning, linear regression remains an essential baseline:
- fast to train
- highly interpretable
- strong comparison point before nonlinear models
- useful to detect whether extra complexity (trees, boosting, neural nets) really adds value
It is especially relevant when data are scarce, noise is moderate, and coefficient interpretation matters.