Probabilistic foundations of Cramér-Rao Bound & Efficiency.
The Cramér-Rao Lower Bound (CRLB) is one of the crowning theoretical achievements of mathematical statistics. Established independently by Harald Cramér and C. R. Rao, it sets an unbreachable benchmark for the precision of unbiased statistical estimators based on Fisher information. Coupled with the Rao-Blackwell and Lehmann-Scheffé theorems, it provides the definitive criterion for identifying Minimum Variance Unbiased Estimators (MVUE). In this laboratory, we derive the information bound, benchmark competing estimators, verify the Cauchy-Schwarz equality condition, and confirm the exact unit efficiency gradient m = 1.000. In mathematical statistics, empirical data science, and modern computational engineering, understanding the asymptotic sampling properties and theoretical behavior of Cramér-Rao Lower Bound, Fisher Information Matrix, and Minimum Variance Unbiased Estimators (MVUE) provides crucial guarantees. Rigorous experimental validation connects foundational probability laws with observable empirical phenomena, ensuring that statistical inference remains valid under finite sample sizes and real-world data collection constraints.
Viva voce questions & analytical theory
What is the Cramér-Rao Lower Bound (CRLB) and who discovered it?
The Cramér-Rao Lower Bound was discovered independently by Harald Cramér (1946), C. Radhakrishna Rao (1945), and earlier by Maurice Fréchet (1943) and Georges Darmois (1945). It establishes a fundamental mathematical lower limit on the variance of any unbiased estimator T of a scalar parameter θ. For a sample of size n, the CRLB states that Var(T) ≥ 1 / I_n(θ) = 1 / [n I_1(θ)], where I_n(θ) is the Fisher information. No unbiased estimator can have a smaller variance than this bound.
How is the CRLB proven using the Cauchy-Schwarz inequality?
Let T be an unbiased estimator of θ, so E[T] = θ. Let S(θ) = ∇ ℓ(θ) be the score function with E[S(θ)] = 0 and Var(S(θ)) = I_n(θ). The covariance between T and S(θ) is Cov(T, S) = E[T S(θ)] = ∫ T [∇ f(x; θ) / f(x; θ)] f(x; θ) dx = ∇ ∫ T f(x; θ) dx = ∇ E[T] = ∇ θ = 1. Applying the Cauchy-Schwarz inequality [Cov(T, S)]^2 ≤ Var(T) Var(S) yields 1^2 ≤ Var(T) · I_n(θ). Dividing both sides by I_n(θ) gives Var(T) ≥ 1 / I_n(θ).
Under what condition does an unbiased estimator achieve the CRLB with equality?
By the Cauchy-Schwarz equality condition, [Cov(T, S)]^2 = Var(T) Var(S) if and only if T and S(θ) are linearly related: T - θ = a(θ) S(θ) with probability 1. This linear score condition holds if and only if the underlying probability density belongs to an exponential family of the form f(x; θ) = exp(A(θ) T(x) + B(x) + C(θ)). When this condition is met, T is said to be an efficient estimator, with statistical efficiency e(T) = CRLB / Var(T) = 1.0 (100%).
What is a Minimum Variance Unbiased Estimator (MVUE)?
An unbiased estimator T* of θ is a Minimum Variance Unbiased Estimator (MVUE) if for any other unbiased estimator T, Var(T*) ≤ Var(T) for all possible values of θ. If an unbiased estimator achieves the Cramér-Rao Lower Bound, it is automatically the MVUE. However, an MVUE can exist even when the CRLB is not attainable (e.g., when the CRLB bound is strictly lower than the variance of any unbiased estimator).
What is the Rao-Blackwell Theorem, and how does it improve estimators?
The Rao-Blackwell theorem states that if T is an unbiased estimator of θ and S is a sufficient statistic for θ, then the conditional expectation T* = E[T | S] is also an unbiased estimator of θ, and its variance is strictly less than or equal to the variance of T: Var(T*) = Var(E[T | S]) = Var(T) - E[Var(T | S)] ≤ Var(T). Conditioning on a sufficient statistic eliminates extraneous noise without introducing bias.
What does the Lehmann-Scheffé Theorem state regarding MVUE uniqueness?
The Lehmann-Scheffé theorem (1950) provides the ultimate tool for finding MVUEs. It states that if S is a complete sufficient statistic for parameter θ, and T* = g(S) is an unbiased estimator of θ that is a function of S, then T* is the unique Minimum Variance Unbiased Estimator (MVUE). No other unbiased estimator can match or beat its variance anywhere in the parameter space.
How is the CRLB generalized for biased estimators and vector parameters?
For an estimator with bias b(θ) = E[T] - θ, the bound generalizes to Var(T) ≥ [1 + b'(θ)]^2 / I_n(θ). For a parameter vector θ ∈ ℝ^p, the Cramér-Rao bound states that Cov(T) - I_n(θ)^(-1) is a positive semi-definite matrix, meaning the variance of any linear combination c^T T satisfies Var(c^T T) ≥ c^T I_n(θ)^(-1) c.
Experimental precautions and statistical checks
- Ensure the support of the distribution is independent of parameter θ; otherwise the regularity conditions for CRLB fail.
- Verify that the estimator is strictly unbiased before comparing its empirical variance against the standard CRLB.
- Account for derivative-of-bias terms [1 + b'(θ)]² if applying the generalized CRLB to biased estimators.
- Use sufficiently large Monte Carlo replications (N ≥ 50,000) to measure empirical estimator variances with sub-percent precision.
Theoretical assumptions & limitations
- The CRLB is not attainable for all parametric distributions; some models have MVUEs whose variance strictly exceeds 1/I_n.
- In non-regular estimation problems (e.g., U[0, θ]), super-efficient estimators can have variance scaling as 1/n², beating the 1/n bound.