Jackknife & Bias Reduction theory

The jackknife was introduced by Maurice Quenouille in 1949 as a non-parametric method for reducing the bias of statistical estimators, and was later expanded by John W. Tukey in 1958 to estimate standard errors and construct robust confidence intervals. Tukey named it the 'jackknife' because, like a versatile Swiss Army pocketknife, it provides a rough-and-ready, dependable tool capable of handling almost any statistical problem without requiring complex analytical calculations. It operates deterministically by systematically recalculating the estimator on leave-one-out subsamples.

Suppose an estimator θ̂ based on N observations has expectation E[θ̂] = θ + a_1 / N + a_2 / N^2 + O(1/N^3) for some unknown constants a_1, a_2. When evaluated on a leave-one-out subsample of size N - 1, its expectation is E[θ̂_{(-i)}] = θ + a_1 / (N - 1) + a_2 / (N - 1)^2 + O(1/N^3). Taking the average over all N subsamples θ̄_{(-·)} = (1/N) ∑ θ̂_{(-i)}, we have E[θ̄_{(-·)}] = θ + a_1 / (N - 1) + O(1/N^2). Quenouille's jackknife estimator is defined as θ̂_jack = N θ̂ - (N - 1) θ̄_{(-·)}. Taking expectations: E[θ̂_jack] = N [θ + a_1/N + O(1/N^2)] - (N - 1) [θ + a_1/(N-1) + O(1/N^2)] = Nθ + a_1 - (N - 1)θ - a_1 + O(1/N^2) = θ + O(1/N^2). The leading bias term a_1/N has completely cancelled out!

John Tukey defined the i-th pseudo-value as J_i = N θ̂ - (N - 1) θ̂_{(-i)}. Notice that the arithmetic mean of the pseudo-values is J̄ = (1/N) ∑ J_i = N θ̂ - (N - 1) (1/N) ∑ θ̂_{(-i)} = N θ̂ - (N - 1) θ̄_{(-·)} = θ̂_jack. Tukey proposed treating the pseudo-values J_1, ..., J_N as approximately independent, identically distributed random variables. The jackknife variance estimator is then simply the sample variance of the pseudo-values divided by N: Var_jack(θ̂_jack) = (1 / [N (N - 1)]) ∑_{i=1}^N (J_i - J̄)^2 = ((N - 1) / N) ∑_{i=1}^N (θ̂_{(-i)} - θ̄_{(-·)})^2. This enables the construction of t-based confidence intervals: θ̂_jack ± t_{N-1, α/2} √Var_jack.

Both methods are non-parametric resampling techniques, but they differ fundamentally in execution: (1) Determinism: The leave-one-out jackknife is completely deterministic, requiring exactly N evaluations of the statistic. The bootstrap is a stochastic Monte Carlo technique requiring B random draws with replacement (typically B = 1,000 to 2,000). (2) Sampling scheme: The jackknife deletes 1 observation at a time, creating samples of size N - 1. The bootstrap samples with replacement, creating samples of size N with repeated and omitted points. (3) Approximation: Efron (1979) proved that the jackknife variance estimator is a linear Taylor-series approximation to the bootstrap variance estimator.

The jackknife requires the estimator θ̂ = T(F_n) to be mathematically 'smooth' (having a linear Fréchet derivative or bounded influence function). For non-smooth statistics like the sample median, deleting a single observation either shifts the median by a discrete step or leaves it completely unchanged. For example, in a sample of even size N = 2m, deleting an observation below the median changes the median to X_{(m+1)}, while deleting an observation above changes it to X_{(m)}. Consequently, the pseudo-values take only two discrete values, and the jackknife variance estimator fails to converge to the true variance (it is not consistent for the median). In contrast, the bootstrap is consistent for the median.

To overcome the failure of the leave-one-out jackknife for non-smooth statistics, C.F.J. Wu (1986) and Jun Shao (1989) developed the delete-d jackknife, which systematically deletes d > 1 observations at a time, evaluating the statistic on subsamples of size N - d. Shao proved that if d satisfies d → ∞ and N - d → ∞ such that d / N → 0 and √N / d → 0 (for example, d = √N · ln(N)), the delete-d jackknife variance estimator is asymptotically consistent even for non-smooth statistics like the sample median and quantiles.

The acceleration parameter a in Efron's BCa bootstrap confidence interval measures the rate of change of the estimator's standard error: a = (1/6) [∑_{i=1}^N U_i^3] / [∑_{i=1}^N U_i^2]^(3/2), where U_i is the empirical influence value of observation i. In practice, the influence values U_i are estimated directly using the leave-one-out jackknife: U_i = (N - 1)(θ̄_{(-·)} - θ̂_{(-i)}). Thus, the jackknife serves as an essential computational engine within advanced bootstrap inference pipelines.