Multinomial Joint Sampling theory

The Multinomial distribution is the multi-category generalization of the Binomial distribution. Suppose n independent trials are conducted, where each trial results in exactly one of K mutually exclusive outcomes with probabilities p_1, p_2, ..., p_K satisfying p_i ≥ 0 and ∑_{i=1}^K p_i = 1. Let X_i denote the number of times outcome i occurs, such that ∑_{i=1}^K X_i = n. The joint probability mass function is P(X_1 = x_1, ..., X_K = x_K) = (n! / (x_1! x_2! ··· x_K!)) p_1^(x_1) p_2^(x_2) ··· p_K^(x_K) for non-negative integers x_i summing to n.

Because the total number of trials is fixed at n = ∑_{i=1}^K X_i, the category counts are constrained by a linear conservation law. If category i occurs more frequently in a given experiment, fewer trials remain available for all other categories. Mathematically, taking the variance of the constant sum yields Var(∑ X_i) = Var(n) = 0. Expanding this variance gives ∑ Var(X_i) + 2 ∑_{i < j} Cov(X_i, X_j) = 0. Because individual variances Var(X_i) = n p_i (1 - p_i) > 0 are strictly positive, the cross-covariances must be negative on average. Specifically, Cov(X_i, X_j) = -n p_i p_j for i ≠ j.

The K × K covariance matrix Σ of a Multinomial random vector has diagonal entries Σ_{ii} = Var(X_i) = n p_i (1 - p_i) and off-diagonal entries Σ_{ij} = Cov(X_i, X_j) = -n p_i p_j for i ≠ j. In matrix notation, Σ = n (diag(p) - p p^T). Because the vector of all ones 1 satisfies Σ 1 = n (p - p(p^T 1)) = n(p - p(1)) = 0, the matrix has an eigenvalue of 0, meaning Σ is singular with rank K - 1. This singularity reflects the constraint that the counts reside in a (K - 1)-dimensional simplex.

By collapsing categories, any single count X_i follows an exact Binomial distribution: X_i ~ Binomial(n, p_i). More generally, if we aggregate any subset of categories into a composite group with probability P_A = ∑_{i ∈ A} p_i and count X_A = ∑_{i ∈ A} X_i, the joint distribution of the remaining categories and X_A remains Multinomial. This aggregation invariance is a fundamental property of categorical processes.

Conditional on observing X_2 = x_2, the remaining n - x_2 trials must be distributed among the remaining K - 1 categories. The conditional distribution of X_1 given X_2 = x_2 is Binomial with trial count n - x_2 and renormalized success probability p_1 / (1 - p_2): X_1 | (X_2 = x_2) ~ Binomial(n - x_2, p_1 / (1 - p_2)). This conditional renormalization directly illustrates the negative dependency between categories.

If the total sample size N is not fixed, but is itself a random variable following a Poisson distribution N ~ Poisson(λ), and N items are subsequently sorted into K categories with probabilities p_1, ..., p_K, then the resulting category counts X_1, ..., X_K are mutually independent Poisson random variables with X_i ~ Poisson(λ p_i). This result, known as Poissonization, eliminates the negative covariance constraint and simplifies calculations in queuing networks and statistical physics.

Under the null hypothesis that observed counts X = (X_1, ..., X_K) arise from a multinomial distribution with specified probabilities p, the Pearson goodness-of-fit statistic is Q = ∑_{i=1}^K (X_i - n p_i)^2 / (n p_i). By the multivariate central limit theorem, the standardized vector converges to a degenerate multivariate normal distribution on the simplex. Karl Pearson (1900) proved that the quadratic form Q converges in distribution to a Chi-Squared distribution with K - 1 degrees of freedom.