Ridge & Lasso Regularization theory

When the number of predictors p exceeds the sample size n (p > n), the design matrix X has rank at most n < p. Consequently, the p × p Gram matrix X^T X is rank-deficient and singular, with at least p - n zero eigenvalues. The normal equations (X^T X) β̂ = X^T Y have infinitely many solutions that fit training data perfectly with zero residuals (SSE = 0), but exhibit catastrophic variance and completely fail on new test data. Regularization is mathematically required to restore full rank and bound estimator variance.

Introduced by Arthur Hoerl and Robert Kennard in 1970, Ridge Regression adds an L2 norm penalty on the coefficient vector to the OLS objective: L_Ridge(β) = ∑_{i=1}^n (Y_i - x_i^T β)^2 + λ ∑_{j=1}^p β_j^2 = ||Y - X β||^2 + λ ||β||_2^2, where λ ≥ 0 is the tuning penalty. Setting the gradient to zero yields the closed-form analytical solution: β̂_Ridge = (X^T X + λ I_p)^(-1) X^T Y. Adding λ > 0 to the diagonal shifts all eigenvalues of X^T X upward by λ, guaranteeing that X^T X + λ I is strictly positive-definite and well-conditioned, even when p > n.

Let X = U D V^T be the SVD of the centered design matrix X, where D = diag(d_j). The OLS fitted values are Ŷ_OLS = ∑ u_j u_j^T Y. In Ridge regression, the fitted values are Ŷ_Ridge = ∑ [d_j^2 / (d_j^2 + λ)] u_j u_j^T Y. The shrinkage factor f_j = d_j^2 / (d_j^2 + λ) is close to 1 for large singular values (principal directions with high variance), but shrinks aggressively toward zero for small singular values (directions with small variance and high noise). Ridge stabilizes estimation by filtering out low-variance noise directions.

Introduced by Robert Tibshirani in 1996, the Lasso (Least Absolute Shrinkage and Selection Operator) penalizes the L1 norm of the coefficients: L_Lasso(β) = ||Y - X β||^2 + λ ∑_{j=1}^p |β_j| = ||Y - X β||^2 + λ ||β||_1. Geometrically, minimizing sum of squares subject to an L1 constraint ||β||_1 ≤ t constrains solutions to a cross-polytope (rhombus in 2D) whose corners lie on the coordinate axes. The elliptical contours of the OLS loss function hit the sharp corners of the L1 diamond first, forcing many coefficients to become exactly zero. Thus, Lasso performs automated feature selection.

For an orthogonal design matrix X^T X = I, the Lasso optimization decouples into p independent one-dimensional problems. The exact solution is given by the soft-thresholding operator S(β̂_OLS, λ) = sign(β̂_OLS) · max(0, |β̂_OLS| - λ/2). If the OLS estimate has magnitude |β̂_OLS| ≤ λ/2, it is set exactly to zero; if |β̂_OLS| > λ/2, it is shrunk toward zero by the constant amount λ/2. Modern algorithms (like glmnet by Friedman, Hastie, and Tibshirani) solve general Lasso problems efficiently via cyclical coordinate descent using this soft-thresholding operator.

Hui Zou and Trevor Hastie (2005) introduced Elastic Net to resolve three key Lasso limitations: (1) When p > n, Lasso selects at most n predictors; (2) In groups of highly correlated variables, Lasso tends to arbitrarily pick one and discard the rest; (3) When n >> p and predictors are correlated, Ridge outperforms Lasso. Elastic Net combines both penalties: L_EN(β) = ||Y - X β||^2 + λ_1 ||β||_1 + λ_2 ||β||_2^2. The L1 penalty produces sparsity, while the L2 penalty stabilizes the matrix and creates a 'grouping effect' where correlated features enter the model together.

Because the penalty parameter λ controls the Bias-Variance tradeoff (λ = 0 is unbiased OLS with high variance; λ → ∞ shrinks all coefficients to zero with high bias), it is selected empirically. In K-fold cross-validation (typically K = 10), data is partitioned into K folds. The model is trained on K - 1 folds across a grid of λ values and evaluated on the remaining fold. Standard practice identifies: (1) λ_min: the penalty achieving minimum mean cross-validated error; and (2) λ_1se: the largest penalty whose error is within one standard error of the minimum, providing the most parsimonious model.