Probabilistic foundations of Logistic Regression.
Logistic Regression is the foundational classification and probabilistic modeling framework for binary discrete data across biostatistics, medicine, epidemiology, and machine learning. Overcoming the severe mathematical limitations of ordinary linear models for bounded probabilities, it employs the logit link function to linearize the log-odds of success, estimating parameters via maximum likelihood and Iteratively Reweighted Least Squares (IRLS). In this laboratory, we simulate Bernoulli response streams, perform IRLS optimization, compute Odds Ratios, and confirm the linear affine log-odds response model y = 0.800x - 2.000. In mathematical statistics, empirical data science, and modern computational engineering, understanding the asymptotic sampling properties and theoretical behavior of Logistic Regression, Logit Link Function, and Maximum Likelihood Odds Ratios provides crucial guarantees. Rigorous experimental validation connects foundational probability laws with observable empirical phenomena, ensuring that statistical inference remains valid under finite sample sizes and real-world data collection constraints.
Viva voce questions & analytical theory
Why is Ordinary Least Squares inappropriate for binary response variables?
Applying linear regression to binary Y ∈ {0, 1} (the Linear Probability Model Y = β_0 + β_1 X + ε) suffers from four fatal theoretical defects: (1) Unbounded probabilities: Linear models can predict probabilities p̂ < 0 or p̂ > 1 for extreme X values; (2) Inherent heteroscedasticity: Since Var(Y | X) = p(X)(1 - p(X)), error variance depends directly on X, violating homoscedasticity; (3) Non-normal errors: The residuals ε_i can only take two discrete values (1 - p_i or -p_i), invalidating t and F tests; (4) Non-linear relationship: Most physical and biological probabilities exhibit sigmoidal S-shaped saturation curves.
How is the Logistic Regression model and logit link function formulated?
Logistic regression models the success probability p(x) = P(Y = 1 | X = x) using the standard logistic sigmoid function: p(x) = exp(β_0 + β_1 x) / [1 + exp(β_0 + β_1 x)] = 1 / [1 + exp(-(β_0 + β_1 x))]. The odds of success is defined as Odds = p / (1 - p). Applying the logit link function (the natural logarithm of the odds) yields a strictly linear relationship: logit(p) = ln(p / (1 - p)) = ln(exp(β_0 + β_1 x)) = β_0 + β_1 x. This maps bounded probabilities p ∈ (0, 1) onto the entire real line (-∞, ∞).
How are the regression parameters estimated via Maximum Likelihood and IRLS?
For independent binary observations Y_i ~ Bernoulli(p_i), the likelihood function is L(β) = ∏ p_i^(y_i) (1 - p_i)^(1 - y_i). The log-likelihood is ℓ(β) = ∑ [y_i ln p_i + (1 - y_i) ln(1 - p_i)] = ∑ [y_i (x_i^T β) - ln(1 + exp(x_i^T β))]. The gradient (score vector) is S(β) = X^T (Y - p), and the Hessian is H = - X^T W X, where W = diag(p_i(1 - p_i)) is a diagonal weight matrix. Because no closed-form solution exists, Newton-Raphson evaluates β_{t+1} = β_t - H^(-1) S(β_t) = (X^T W X)^(-1) X^T W z, where z is an adjusted response vector. This is known as Iteratively Reweighted Least Squares (IRLS).
How is the regression slope β_1 interpreted in terms of the Odds Ratio?
Consider increasing the predictor by one unit from x to x + 1. The new log-odds is ln(Odds_{x+1}) = β_0 + β_1(x + 1) = ln(Odds_x) + β_1. Subtracting the two equations gives: ln(Odds_{x+1}) - ln(Odds_x) = ln(Odds_{x+1} / Odds_x) = β_1. Exponentiating both sides yields the Odds Ratio (OR): OR = Odds_{x+1} / Odds_x = exp(β_1). Therefore, for every unit increase in predictor X, the odds of success multiply by a constant factor of exp(β_1). If β_1 = 0.800, OR = e^(0.800) ≈ 2.226, meaning odds increase by 122.6% per unit increase in x.
What is the deviance, and how is the Likelihood Ratio Test used in logistic models?
In Generalized Linear Models, deviance D = -2 [ℓ(β̂) - ℓ(saturated)] measures the goodness-of-fit relative to a perfect saturated model. The Null Deviance D_0 measures fit with only the intercept, while Residual Deviance D_1 measures fit with all predictors included. The omnibus Model Chi-Square statistic is G = D_0 - D_1 = 2 [ℓ(full) - ℓ(null)] ~ χ²(p - 1). It tests the null hypothesis that all slope coefficients are simultaneously zero, serving as the logistic analogue of the overall F-test in linear regression.
What is the Hosmer-Lemeshow goodness-of-fit test?
David Hosmer and Stanley Lemeshow (1980) developed a test to evaluate how well predicted probabilities calibrate with empirical frequencies. The sample is partitioned into g groups (typically deciles, g = 10) based on ascending predicted probabilities p̂_i. The test statistic H = ∑_{k=1}^g [(O_k - n_k p̄_k)^2 / (n_k p̄_k (1 - p̄_k))] compares observed event counts O_k against expected counts n_k p̄_k. Under H_0 (model fits well), H follows a Chi-Squared distribution with g - 2 degrees of freedom (typically df = 8). A non-significant p-value (p > 0.05) indicates good calibration.
What is complete separation (quasi-complete separation), and what estimation failure does it cause?
Complete separation occurs when a linear decision boundary perfectly separates the binary classes (e.g., all Y = 0 have X < c and all Y = 1 have X > c). In this regime, the likelihood function monotonically increases as β_1 → ∞, and the maximum likelihood estimate does not exist on the finite real line. Standard software will produce astronomical coefficients and colossal standard errors. David Firth (1993) introduced penalized likelihood estimation (adding Jeffreys invariant prior penalty 0.5 ln |I(β)|) to guarantee finite, consistent estimates under separation.
Experimental precautions and statistical checks
- Ensure predictor data does not exhibit complete separation, which causes MLE parameters to diverge to infinity.
- Verify that binary outcomes are coded strictly as integers 0 and 1 before initiating IRLS iteration.
- Inspect predicted probabilities to verify that they calibrate with empirical frequencies across risk deciles.
- Report Odds Ratios alongside 95% confidence intervals exp(β̂_1 ± 1.96 SE) for clear physical interpretation.
Theoretical assumptions & limitations
- Logistic regression assumes that the log-odds of the outcome is a linear function of the continuous predictors.
- Extreme imbalance in binary classes (e.g., rare events with < 1% positive outcomes) requires specialized Firth penalized likelihood methods.