Probabilistic foundations of Simple Linear Regression (OLS).
Simple Linear Regression is the fundamental quantitative tool for modeling functional dependencies between continuous variables in empirical science. Anchored by the Ordinary Least Squares (OLS) criterion and the celebrated Gauss-Markov Theorem, it provides the optimal Best Linear Unbiased Estimator (BLUE) under minimal probabilistic assumptions. In this laboratory, we derive normal equations, evaluate orthogonal variance partitioning, explore regression diagnostics, prove Gauss-Markov efficiency, and confirm the linear slope parameter gradient m = 0.500. In mathematical statistics, empirical data science, and modern computational engineering, understanding the asymptotic sampling properties and theoretical behavior of Simple Linear Regression, Ordinary Least Squares, and Gauss-Markov Theorem provides crucial guarantees. Rigorous experimental validation connects foundational probability laws with observable empirical phenomena, ensuring that statistical inference remains valid under finite sample sizes and real-world data collection constraints.
Viva voce questions & analytical theory
How is the Simple Linear Regression model formulated mathematically?
The Simple Linear Regression model relates a continuous scalar dependent variable Y to a single independent regressor variable X through the linear equation: Y_i = β_0 + β_1 X_i + ε_i for i = 1, ..., n. The parameters β_0 (intercept) and β_1 (slope) are unknown regression coefficients, and ε_i represents unobserved random error. The classical Gauss-Markov assumptions require: (1) Zero conditional mean: E[ε_i | X_i] = 0; (2) Homoscedasticity: Var(ε_i | X_i) = σ^2 (constant error variance); (3) Uncorrelated errors: Cov(ε_i, ε_j) = 0 for all i ≠ j; and (4) Non-stochastic regressors with non-zero variance S_xx > 0.
How are the Ordinary Least Squares (OLS) normal equations derived?
The OLS principle minimizes the sum of squared residual errors: S(β_0, β_1) = ∑_{i=1}^n (Y_i - β_0 - β_1 X_i)^2. Differentiating S with respect to β_0 and β_1 and setting the partial derivatives to zero yields the normal equations: (1) ∂S/∂β_0 = -2 ∑ (Y_i - β_0 - β_1 X_i) = 0 ⟹ n β_0 + β_1 ∑ X_i = ∑ Y_i; (2) ∂S/∂β_1 = -2 ∑ X_i (Y_i - β_0 - β_1 X_i) = 0 ⟹ β_0 ∑ X_i + β_1 ∑ X_i^2 = ∑ X_i Y_i. Solving these equations yields β̂_1 = S_xy / S_xx = [∑ (X_i - x̄)(Y_i - ȳ)] / [∑ (X_i - x̄)^2] and β̂_0 = ȳ - β̂_1 x̄.
What is the Gauss-Markov Theorem, and what does the acronym 'BLUE' mean?
The Gauss-Markov Theorem (Carl Friedrich Gauss, 1821; Andrey Markov, 1900) is the foundational theorem of linear regression. It states that under the classical Gauss-Markov assumptions, the Ordinary Least Squares estimators β̂_0 and β̂_1 are the Best Linear Unbiased Estimators (BLUE). Specifically: (1) 'Best': they achieve the minimum sampling variance among all possible linear unbiased estimators; (2) 'Linear': they are linear combinations of the observed responses Y_i (β̂_1 = ∑ w_i Y_i); (3) 'Unbiased': their expectations equal the true population parameters (E[β̂] = β). Crucially, the theorem requires NO normality assumption on the errors.
What are the sampling variances and standard errors of the OLS estimators?
Under the Gauss-Markov assumptions: (1) Var(β̂_1) = σ^2 / S_xx = σ^2 / [∑ (X_i - x̄)^2]; (2) Var(β̂_0) = σ^2 [1/n + x̄^2 / S_xx]; (3) Cov(β̂_0, β̂_1) = - x̄ σ^2 / S_xx. Because population variance σ^2 is typically unknown, it is estimated by the unbiased residual mean square s^2 = SSE / (n - 2) = [∑ (Y_i - ŷ_i)^2] / (n - 2). The standard error of the slope is SE(β̂_1) = s / √S_xx, which forms the denominator of the t-statistic t = (β̂_1 - 0) / SE(β̂_1) ~ t(n - 2).
How is the Coefficient of Determination (R²) defined and interpreted?
Total variation in Y is decomposed into explained and unexplained components: Total Sum of Squares (SST = ∑ (Y_i - ȳ)^2) = Regression Sum of Squares (SSR = ∑ (ŷ_i - ȳ)^2) + Residual Sum of Squares (SSE = ∑ (Y_i - ŷ_i)^2). The Coefficient of Determination is R² = SSR / SST = 1 - SSE / SST. In simple linear regression, R² equals the square of the Pearson correlation coefficient: R² = r_{xy}^2. It represents the proportion of total variance in the dependent variable explained by the linear regression on X, bounded strictly in [0, 1].
What are the four primary diagnostic plots used to evaluate OLS regression assumptions?
Regression diagnostics examine model validity: (1) Residuals vs. Fitted Values: detects non-linearity (curved patterns) or heteroscedasticity (funnel shape); (2) Normal Q-Q Plot: compares studentized residuals against theoretical Gaussian quantiles to verify error normality; (3) Scale-Location Plot: plots √|standardized residuals| vs. fitted values to evaluate homoscedasticity; (4) Residuals vs. Leverage (Cook's Distance): identifies influential high-leverage data points that exert disproportionate pull on regression parameters.
How does the Maximum Likelihood Estimator compare to OLS under Gaussian errors?
When the error terms are assumed to be independent identically distributed normal variates ε_i ~ N(0, σ^2), the Maximum Likelihood Estimators for slope and intercept are mathematically identical to the OLS estimators: β̂_MLE = β̂_OLS. However, the MLE for error variance is σ̂_MLE^2 = SSE / n, which is biased downward by a factor of (n - 2)/n compared to the unbiased OLS estimator s^2 = SSE / (n - 2).
Experimental precautions and statistical checks
- Ensure regressor values X have non-zero spread (S_xx > 0) to avoid division by zero in normal equations.
- Verify homoscedasticity and linearity by inspecting residual versus fitted plots for absence of funneling or curvature.
- Account for influential leverage points using Cook's distance metrics before finalizing slope interpretations.
- Do not extrapolate regression predictions far beyond the experimental range of observed X values.
Theoretical assumptions & limitations
- OLS regression measures linear association; non-linear relationships can yield misleading near-zero slopes.
- Outliers and heavy-tailed error distributions can heavily distort OLS parameter estimates (0% breakdown point).