Method of Moments Estimation theory

The Method of Moments was introduced by Russian mathematician Pafnuty Chebyshev in 1887 and popularized in Western statistics by Karl Pearson in 1894. It is an intuitive parameter estimation technique based on the Law of Large Numbers. By the Law of Large Numbers, the k-th sample raw moment m_k' = (1/n) ∑_{i=1}^n X_i^k converges in probability to the k-th population moment μ_k'(θ) = E[X^k]. The method equates the first p sample moments to their theoretical expressions in terms of p unknown parameters (m_k' = μ_k'(θ_1, ..., θ_p) for k = 1, ..., p) and solves the resulting system of equations for the estimators θ̃.

For Gamma(α, β) with rate β, the first two theoretical raw moments are E[X] = α / β and E[X^2] = Var(X) + (E[X])^2 = α/β^2 + α^2/β^2. The second central moment is population variance σ^2 = α / β^2. Equating sample mean x̄ to α/β and sample variance s^2 to α/β^2 yields the algebraic system: x̄ = α/β and s^2 = α/β^2. Dividing x̄ by s^2 gives β̃ = x̄ / s^2. Substituting β̃ into the mean equation yields α̃ = x̄ β̃ = x̄^2 / s^2. These closed-form estimators provide instant parameter estimates without iterative optimization.

Method of Moments estimators are generally consistent and asymptotically normal, but they are typically not asymptotically efficient (they do not achieve the Cramér-Rao Lower Bound). Karl Pearson's moment estimators often throw away higher-order probabilistic information captured by the full likelihood function. The asymptotic relative efficiency (ARE) is ARE = Var(θ̂_MLE) / Var(θ̃_MoM) ≤ 1. In Gamma models with large shape α, the ARE of the moment estimator can drop below 50% relative to MLE.

For X ~ U[0, θ], the theoretical mean is E[X] = θ / 2. Equating sample mean to expectation gives x̄ = θ / 2, yielding the MoM estimator θ̃ = 2 x̄. By the Central Limit Theorem, Var(θ̃) = 4 Var(x̄) = 4 (θ^2 / 12n) = θ^2 / (3n), converging at rate √n. In contrast, the MLE is the sample maximum θ̂ = X_{(n)}, whose variance is Var(θ̂) = θ^2 / [(n + 1)(n + 2)] ≈ θ^2 / n^2, converging at the super-fast rate n. Thus, MoM is dramatically inferior to MLE for uniform data.

Yes. Because MoM equates sample moments to theoretical functions without enforcing parameter domain constraints, it can yield inadmissible estimates in finite samples. For example, in a Beta(α, β) model, sample variance s^2 can occasionally exceed x̄(1 - x̄), which would mathematically require negative shape parameters (α < 0, β < 0), violating the definition of the Beta distribution. MLE, by contrast, automatically respects the parameter space.

Introduced by Lars Peter Hansen in 1982, GMM generalizes classical moment matching to econometrics where more moment conditions exist than unknown parameters (overidentified systems: E[g(X_i, θ)] = 0 with dim(g) = q > p = dim(θ)). GMM minimizes a weighted quadratic form J(θ) = ḡ(θ)^T W ḡ(θ). Hansen derived the optimal weighting matrix W = [Var(g)]^(-1), proving that optimal GMM achieves minimum asymptotic variance among all estimators based on those moment conditions.

Despite lower asymptotic efficiency, the Method of Moments possesses three major practical virtues: (1) Analytical tractability: MoM estimators frequently yield explicit closed-form formulas that require zero numerical iteration; (2) Starting values: MoM estimates serve as robust initial guesses to seed Newton-Raphson or EM algorithms for complex MLE models; (3) Robustness against certain distributional mis-specifications where lower moments are correctly specified but higher-order likelihood assumptions fail.