Principal Component Analysis (PCA)

Principal component analysis finds orthogonal directions that explain the greatest variance in centered data. Keeping only the leading components gives a lower-dimensional representation.

PCA preserves variance under a linear projection. It does not automatically preserve the information most useful for a prediction task, and feature scaling can change the result. Inspect the data and the projection before deciding what to discard.

Imagine a cloud of points stretched mostly along one diagonal. PCA rotates the coordinate system so the first new axis points along the direction of greatest spread, the second along the next-greatest (orthogonal to the first), and so on. Projecting onto the first few axes keeps the structure that matters and discards directions where the data barely varies.

The first principal component is the single direction that, if you projected all points onto it, would preserve the most variance: equivalently, the line minimizing total squared reconstruction error.

Given centered data $X \in \mathbb{R}^{n \times d}$ (mean subtracted), the covariance matrix is

PCA finds the orthonormal directions $w$ that maximize projected variance $w^\top \Sigma\, w$ subject to $\lVert w \rVert = 1$. The solution is the eigenvectors of $\Sigma$, ordered by eigenvalue:

Each eigenvalue $\lambda_i$ is the variance captured by component $i$, so the fraction of variance explained by the top $k$ components is $\sum_{i=1}^{k}\lambda_i \big/ \sum_{j=1}^{d}\lambda_j$: the standard way to choose $k$. In practice PCA is computed via the SVD of $X = U S V^\top$: the principal components are the columns of $V$, and the singular values relate to variance by $\lambda_i = s_i^2 / (n-1)$. SVD is preferred for numerical stability (it avoids forming $X^\top X$). Scaling matters: PCA is variance-driven, so features on larger scales dominate: standardize features first unless they're already comparable.

Key takeaway: PCA = center, then project onto the top eigenvectors of the covariance matrix (computed stably via SVD), keeping enough components to hit your explained-variance target, and always scale features first, because PCA listens to variance, not to importance.

1. Conceptual: What does PCA maximize, and what are the principal components? (It finds orthogonal directions that maximize projected variance: the eigenvectors of the covariance matrix, ordered by eigenvalue.)
2. Implementation: Why compute PCA via SVD instead of eigendecomposition of the covariance? (SVD on the centered data is more numerically stable and avoids explicitly forming XᵀX, which can lose precision.)
3. Applied: How do you choose the number of components k? (By cumulative explained variance: keep enough components to reach a target, e.g. 95%, using the eigenvalue/singular-value ratios.)
4. Systems-level: Why must you scale features before PCA? (PCA is variance-driven, so a feature on a larger scale dominates the components regardless of importance: standardize first.)
5. Failure modes: When does PCA fail or mislead, e.g. vs. t-SNE? (It only captures linear structure and global variance; nonlinear manifolds or cluster structure may need t-SNE/UMAP, which preserve local neighborhoods instead.)

From memory: list the four PCA steps (center, covariance/SVD, top-k, project), state what eigenvalues represent, and explain why scaling matters. Check against Stage 3.