LoRA and PEFT

LoRA adapts a pretrained model by training a low-rank update to selected weight matrices while keeping the original weights frozen. Instead of updating a full matrix W, it learns two smaller matrices whose product is added to W.

The practical benefit is fewer trainable parameters and less optimizer state. Low rank is a constraint on the update, not a claim that every useful adaptation is exactly low-rank or that the base model becomes smaller.

A linear layer normally computes $Wx$. LoRA adds a parallel low-rank branch: $h = Wx + BAx$, where $A$ projects down to a small rank $r$ and $B$ projects back up. The base $W$ is frozen; only $A$ and $B$ get gradients. Because $r$ is small (often 8–64), $A$ and $B$ together are a rounding error in size next to $W$.

For compatible dense weights, merge the scaled update into $W$ before inference. This removes the separate adapter operations while retaining the original matrix shape. Keeping multiple adapters separate or using quantized weights can change that deployment choice.

For a frozen weight $W \in \mathbb{R}^{d \times k}$, LoRA reparameterizes the update as a low-rank product:

$\alpha$ is a scaling constant (the effective update is $\frac{\alpha}{r} BA$). Initialization matters: $A$ is random (e.g. Gaussian) and $B = 0$, so at the start $\Delta W = 0$ and the model is exactly the pretrained base: training begins from the known-good point and only departs as needed.

Parameter count: full fine-tuning trains $d \cdot k$ params per matrix; LoRA trains $r(d + k)$. For $d=k=4096$, $r=8$: that's $65{,}536$ vs $16.7$M: a ~256× reduction, per matrix. QLoRA pushes this further by quantizing the frozen base to 4-bit, so even the frozen weights cost little memory, enabling fine-tuning of very large models on a single GPU.

Key takeaway: LoRA bets that task adaptation lives in a low-rank subspace: freeze $W$, train $\Delta W = \frac{\alpha}{r}BA$ with $B=\mathbf{0}$ at init. The trainable fraction depends on rank and which layers are adapted. Merging into compatible weights removes the adapter branch, not the base model's inference cost.

1. Conceptual: Why can a low-rank update match full fine-tuning? (Low-rank updates worked well on the tasks studied in the LoRA paper. Their adequacy depends on the task, chosen layers, and rank; not every useful full update is low-rank.)
2. Implementation: Why initialize B to zero? (So ΔW = BA = 0 at the start: training begins exactly from the pretrained model and only deviates as needed, which is stable.)
3. Applied: Roughly how many parameters does LoRA train versus full fine-tuning for a d×k matrix? (r(d+k) vs d·k: often <1%, e.g. ~256× fewer for d=k=4096, r=8.)
4. Systems-level: What does QLoRA add, and why does it matter? (It 4-bit-quantizes the frozen base so even the frozen weights are cheap to hold: enabling fine-tuning of very large models on one GPU.)
5. Failure modes: Does LoRA add inference latency? (An unmerged adapter adds operations. Merging W' = W + (α/r)BA into compatible dense weights removes that extra branch; quantized deployment needs separate consideration.)

Without looking: write the LoRA reparameterization with shapes of A and B, explain why B is initialized to zero, and give the parameter count versus full fine-tuning. Then state how to remove the inference overhead. Check against Stage 3.

Sources and further reading

• Hu et al., LoRA: low-rank updates, initialization, and merging.
• Dettmers et al., QLoRA: quantized frozen weights and adapter training.