Principal Component Analysis (PCA)

Principal component analysis finds orthogonal directions that explain the greatest variance in centered data. Keeping only the leading components gives a lower-dimensional representation.

PCA preserves variance under a linear projection. It does not automatically preserve the information most useful for a prediction task, and feature scaling can change the result. Inspect the data and the projection before deciding what to discard.

Imagine a cloud of points stretched mostly along one diagonal. PCA rotates the coordinate system so the first new axis points along the direction of greatest spread, the second along the next-greatest (orthogonal to the first), and so on. Projecting onto the first few axes keeps the directions with greatest variance and discards lower-variance directions, which may still contain useful predictive information.

The first principal component is the single direction that, if you projected all points onto it, would preserve the most variance: equivalently, the line minimizing total squared reconstruction error.

Given centered data $X \in \mathbb{R}^{n \times d}$ (mean subtracted), the covariance matrix is

PCA finds the orthonormal directions $w$ that maximize projected variance $w^\top \Sigma\, w$ subject to $\lVert w \rVert = 1$. The solution is the eigenvectors of $\Sigma$, ordered by eigenvalue:

Each eigenvalue $\lambda_i$ is the variance captured by component $i$, so the fraction of variance explained by the top $k$ components is $\sum_{i=1}^{k}\lambda_i \big/ \sum_{j=1}^{d}\lambda_j$: the standard way to choose $k$. In practice PCA is computed via the SVD of $X = U S V^\top$: the principal components are the columns of $V$, and the singular values relate to variance by $\lambda_i = s_i^2 / (n-1)$. SVD is preferred for numerical stability (it avoids forming $X^\top X$). Scaling matters: PCA is variance-driven, so features on larger scales dominate: choose scaling based on units and the question. Standardization gives each feature equal marginal variance; it can also amplify low-variance noise.

Key takeaway: PCA = center, then project onto the top eigenvectors of the covariance matrix (computed stably via SVD), keeping enough components to hit your explained-variance target, with a deliberate scaling choice, because PCA listens to variance, not to predictive importance.

1. Conceptual: What does PCA maximize, and what are the principal components? (It finds orthogonal directions that maximize projected variance: the eigenvectors of the covariance matrix, ordered by eigenvalue.)
2. Implementation: Why compute PCA via SVD instead of eigendecomposition of the covariance? (SVD on the centered data is more numerically stable and avoids explicitly forming XᵀX, which can lose precision.)
3. Applied: How do you choose the number of components k? (By cumulative explained variance: keep enough components to reach a target, e.g. 95%, using the eigenvalue/singular-value ratios.)
4. Systems-level: When should you scale features before PCA? (When differences in units should not determine the components. Standardization is a modeling choice, not a requirement; preserving natural scales can be appropriate.)
5. Failure modes: When does PCA fail or mislead, e.g. vs. t-SNE? (It only captures linear structure and global variance; nonlinear manifolds or cluster structure may need t-SNE/UMAP, which preserve local neighborhoods instead.)

From memory: list the four PCA steps (center, covariance/SVD, top-k, project), state what eigenvalues represent, and explain why scaling matters. Check against Stage 3.

Sources and further reading

• scikit-learn, Principal component analysis: centering, SVD, and dimensionality reduction.
• scikit-learn, Importance of feature scaling: how scaling changes principal components.
• UMAP documentation, Transforming new data: using a fitted UMAP model on new samples.