Principal Component Analysis theory

Principal Component Analysis was invented by Karl Pearson in 1901 as an analogue of the principal axis theorem in classical mechanics, and independently expanded by Harold Hotelling in 1933. PCA is an unsupervised linear dimensionality reduction technique that transforms p possibly correlated variables into a smaller set of k < p linearly uncorrelated variables called Principal Components. The principal components are ordered such that the first component captures the largest possible variance in the data, the second captures the second largest variance orthogonal to the first, and so on.

Let X be an n × p centered data matrix with sample covariance matrix S = (1/(n-1)) X^T X. A linear combination of the variables is z_1 = X v_1, where v_1 is a p × 1 weight vector (loading vector). The variance of z_1 is Var(z_1) = v_1^T S v_1. To prevent variance from blowing up by scaling v_1, we constrain v_1 to have unit norm: ||v_1||^2 = v_1^T v_1 = 1. Applying Lagrange multipliers: L(v_1, λ) = v_1^T S v_1 - λ (v_1^T v_1 - 1). Differentiating with respect to v_1: ∂L/∂v_1 = 2 S v_1 - 2 λ v_1 = 0 ⟹ S v_1 = λ v_1. Thus, v_1 is the eigenvector of covariance matrix S corresponding to the largest eigenvalue λ_1 = max(λ).

Singular Value Decomposition (SVD) provides the most computationally stable and direct algorithm for PCA without explicitly calculating the covariance matrix X^T X. Let X̃ be the n × p centered data matrix with SVD: X̃ = U D V^T, where U is an n × p matrix with orthonormal columns, D is a p × p diagonal matrix of singular values d_1 ≥ d_2 ≥ ··· ≥ d_p ≥ 0, and V is a p × p orthogonal matrix of right singular vectors. The sample covariance matrix is S = (1/(n-1)) X̃^T X̃ = (1/(n-1)) V D U^T U D V^T = V [D^2 / (n - 1)] V^T. Thus: (1) The right singular vectors V are the principal component loading vectors; (2) The eigenvalues of S are λ_j = d_j^2 / (n - 1); (3) The principal component scores are Z = X̃ V = U D.

PCA on the raw Covariance matrix S is scale-dependent. If one variable is measured in millimeters and another in kilometers, the variable with the larger numerical scale will artificially dominate the first principal component, regardless of physical importance. Performing PCA on the Correlation matrix R is mathematically equivalent to first standardizing all variables to have mean zero and unit variance (z-scoring). As a universal rule, PCA should be performed on the correlation matrix whenever variables are measured in different physical units or display disparate variances.

A Scree Plot (Raymond Cattell, 1966) plots the eigenvalues λ_j in descending order against component index j. Common retention criteria include: (1) Elbow rule: identify the 'elbow' where the steep cliff flattens out into horizontal scree (retaining components before the bend); (2) Kaiser-Guttman criterion: retain components with eigenvalues λ_j > 1.0 (for correlation matrices, meaning the component explains more variance than a single original variable); (3) Cumulative variance threshold: retain enough components to explain 70% to 90% of total variance.

The Eckart-Young-Mirsky theorem (1936) is the fundamental mathematical theorem of dimensionality reduction. It states that for any rank-k approximation X̃_k of centered matrix X̃, the truncated SVD reconstruction X̃_k = ∑_{j=1}^k d_j u_j v_j^T minimizes the Frobenius reconstruction error ||X̃ - X̃_k||_F^2 = ∑_{j=k+1}^p d_j^2 among all possible matrices of rank at most k. Thus, PCA provides the optimal low-rank projection in the least-squares sense.

Invented by K. Ruben Gabriel in 1971, a Biplot is an elegant two-dimensional graphical display that plots both observations (scores) and variables (loadings) simultaneously. Observations are plotted as points in the space of the first two principal components (PC1 vs. PC2). Variables are plotted as directional vectors originating from the origin. The cosine of the angle between two variable vectors approximates their correlation; the length of a vector represents variance captured; and the orthogonal projection of an observation point onto a variable vector approximates its value on that variable.