Probabilistic foundations of Polynomial Regression & Overfitting.
Polynomial Regression bridges linear regression and non-linear function approximation, illustrating the foundational Bias-Variance Tradeoff that governs all statistical and machine learning models. By expanding inputs into higher-degree Vandermonde design matrices, models gain capacity to capture curvature, but risk catastrophic overfitting where training error vanishes while out-of-sample generalization collapses. In this laboratory, we implement polynomial curve fitting, visualize the U-shaped test error curve, decompose bias and variance, and confirm the quadratic curvature parameter scaling gradient m = 0.200. In mathematical statistics, empirical data science, and modern computational engineering, understanding the asymptotic sampling properties and theoretical behavior of Polynomial Regression, Model Overfitting, and Bias-Variance Tradeoff provides crucial guarantees. Rigorous experimental validation connects foundational probability laws with observable empirical phenomena, ensuring that statistical inference remains valid under finite sample sizes and real-world data collection constraints.
Viva voce questions & analytical theory
How is Polynomial Regression formulated as a linear model?
Although polynomial regression models non-linear curvilinear relationships between the independent variable x and dependent variable Y, it remains mathematically a linear regression model. The model equation is Y_i = β_0 + β_1 x_i + β_2 x_i^2 + ··· + β_d x_i^d + ε_i. Because the model is linear in the unknown parameters β_0, β_1, ..., β_d, it can be solved directly using standard Ordinary Least Squares (OLS) by defining transformed features z_1 = x, z_2 = x^2, ..., z_d = x^d and inverting the Vandermonde normal equations β̂ = (X^T X)^(-1) X^T Y.
What is the Vandermonde matrix, and why does it become ill-conditioned for high polynomial degrees?
The Vandermonde design matrix X has rows of the form [1, x_i, x_i^2, ..., x_i^d]. As the polynomial degree d increases, the columns x^j and x^{j+1} become highly collinear on any positive domain (e.g., x^4 and x^5 look nearly identical over [0.5, 1.5]). Consequently, the Gram matrix X^T X becomes catastrophically ill-conditioned, with condition number growing exponentially with d. In floating-point arithmetic, this causes severe numerical instability, where tiny round-off errors produce wild oscillations in estimated coefficients.
What is the mathematical Bias-Variance decomposition of generalization error?
For any statistical estimator f̂(x) trained on dataset D with true model Y = f(x) + ε where E[ε] = 0 and Var(ε) = σ^2, the expected out-of-sample prediction error at point x decomposes into three non-negative terms: E_D[(Y - f̂(x))^2] = [Bias(f̂(x))]^2 + Var(f̂(x)) + σ^2. Specifically: (1) Bias² = (E_D[f̂(x)] - f(x))^2 measures underfitting (error from erroneous model assumptions, such as fitting a straight line to a parabola); (2) Variance = E_D[(f̂(x) - E_D[f̂(x)])^2] measures overfitting (sensitivity of the model to the specific random training data); (3) σ^2 is irreducible noise.
What is the U-shaped curve of test error, and how does it identify optimal model complexity?
Plotting prediction error against model complexity (polynomial degree d): (1) Training error MSE_train decreases monotonically as d increases, because higher-degree polynomials have more degrees of freedom to interpolate training points; (2) Bias² decreases with d as model flexibility increases; (3) Variance increases with d as the model over-reacts to random noise; (4) Total test error MSE_test exhibits a characteristic U-shaped profile, initially dropping as bias is reduced, reaching a global minimum at the optimal degree d* (where Bias² + Variance is minimized), and then exploding as variance overwhelms the prediction.
How do Orthogonal Polynomials resolve the numerical instability of Vandermonde matrices?
To eliminate multicollinearity among polynomial powers, orthogonal polynomials (such as Legendre polynomials on [-1, 1], Chebyshev polynomials, or Hermite polynomials) are employed. Features are constructed such that ∑ P_j(x_i) P_k(x_i) = 0 for j ≠ k. This makes the design matrix columns mutually orthogonal, reducing X^T X to a diagonal matrix. Each coefficient β̂_j can then be computed independently without matrix inversion, eliminating numerical ill-conditioning entirely.
What information criteria balance goodness-of-fit against model complexity?
To select optimal polynomial degree without relying exclusively on cross-validation, penalized likelihood criteria are used: (1) Akaike Information Criterion: AIC = 2k - 2 ln L ≈ n ln(SSE/n) + 2k; (2) Bayesian Information Criterion: BIC = k ln(n) - 2 ln L ≈ n ln(SSE/n) + k ln(n). Both criteria penalize the log-likelihood by the number of parameters k = d + 1. Because BIC applies a heavier penalty (k ln n versus 2k), it tends to choose more parsimonious models and is consistent for identifying the true model dimension as n → ∞.
How do spline models overcome the global oscillation problems of high-degree polynomials?
Global polynomials suffer from Runge's phenomenon: fitting a high-degree polynomial causes extreme, spurious oscillations near the boundaries of the interval. Spline regression solves this by fitting piecewise low-degree polynomials (typically cubic splines) connected smoothly at boundary points called knots. Cubic splines enforce continuous zeroth, first, and second derivatives (C² continuity) across knots, providing immense local flexibility without boundary instability.
Experimental precautions and statistical checks
- Center and standardize regressor variables (x - x̄)/s before constructing polynomial powers to reduce multicollinearity.
- Always evaluate polynomial model performance on an independent test dataset or via k-fold cross-validation.
- Use orthogonal polynomials or cubic splines instead of raw high-degree polynomials (d > 4) to avoid numerical instability.
- Select model degree d based on minimum cross-validation error or information criteria (AIC/BIC), not training R².
Theoretical assumptions & limitations
- High-degree global polynomials exhibit severe boundary oscillations (Runge's phenomenon) that invalidate extrapolation.
- Polynomial models have unconstrained asymptotic behavior: predictions diverge rapidly to ±∞ outside the training domain.