Two thin matrices and the space their product can reach.
For a 4,096 × 4,096 projection: trainable values instead of 16,777,216 (0.10%)
LoRA learns the change to a weight matrix as the product of two thin matrices. Every row of BA is a mix of the same r rows of A, so the update can only move within an r-dimensional space while W stays frozen. Blue is negative, amber positive; the values are illustrative.