RLHF (Reinforcement Learning from Human Feedback)

Reinforcement learning from human feedback uses human preferences to shape model behavior. A common pipeline first trains on demonstrations, then fits a reward model to comparisons, and finally optimizes a policy against that reward with a constraint on how far it moves.

Keep the roles separate: SFT imitates demonstrations, a reward model scores outputs, and RL changes the policy using that signal. DPO offers a different preference-learning objective; it is not an extra stage of the same pipeline.

You can't directly backprop "be more helpful." So you build a proxy: show humans pairs of model outputs for the same prompt, ask which is better, and fit a reward model to those preferences. Then treat the LLM as a policy and use RL to produce outputs the reward model scores highly, but tether it to the original SFT model with a KL penalty so it doesn't drift into degenerate, reward-hacking text.

The KL term is the safety leash: high reward with low KL means "better, but still recognizably the same model."

Reward model. Given human preference data where response $y_w$ is preferred over $y_l$ for prompt $x$, fit a scalar reward $r_\phi$ with the Bradley-Terry loss:

i.e. push the preferred response's reward above the rejected one's.

Policy optimization. Optimize the policy $\pi_\theta$ to maximize reward while staying near the reference (SFT) policy $\pi_{\text{ref}}$:

PPO implements this with a clipped surrogate objective that limits how far the policy moves per update (stability). $\beta$ controls the leash: too small and the model reward-hacks; too large and it barely changes. DPO (Direct Preference Optimization) is the popular alternative: it derives a closed-form loss directly on the preference data that implicitly optimizes the same KL-regularized objective, skipping the separate reward model and RL loop entirely.

Key takeaway: Humans can't write perfect answers, but they can reliably rank pairs, so RLHF learns a scalar reward from comparisons (Bradley-Terry) and optimizes the policy against it, with a KL penalty as the leash that prevents reward hacking.

The reward-model loss and the KL-shaped reward: the two pieces unique to RLHF:

1import torch
2import torch.nn.functional as F
3from torch import Tensor
4
5
6def reward_model_loss(r_chosen: Tensor, r_rejected: Tensor) -> Tensor:
7    # Bradley-Terry: preferred reward should exceed rejected reward
8    return -F.logsigmoid(r_chosen - r_rejected).mean()
9
10
11def kl_shaped_reward(
12    reward: Tensor, logp_policy: Tensor, logp_ref: Tensor, beta: float = 0.1
13) -> Tensor:
14    # per-token reward used by PPO: task reward minus KL drift from the SFT model
15    kl = logp_policy - logp_ref               # sample estimate of KL
16    return reward - beta * kl
17
18
19r_chosen, r_rejected = torch.tensor([2.1]), torch.tensor([0.4])
20assert reward_model_loss(r_chosen, r_rejected) < reward_model_loss(r_rejected, r_chosen)

1. Conceptual: What are the three stages of RLHF? (SFT on demonstrations → reward model from human preference comparisons → RL (PPO) optimizing the policy against the reward model.)
2. Implementation: Why train a reward model from comparisons instead of absolute scores? (Humans are far more consistent ranking A vs B than assigning calibrated numeric scores; Bradley-Terry turns comparisons into a trainable reward.)
3. Applied: What is the KL penalty for? (It keeps the optimized policy close to the SFT model, preventing drift into degenerate, reward-hacking outputs.)
4. Systems-level: What is reward hacking and how does the setup mitigate it? (The policy exploits flaws in the imperfect reward model for high score / low quality; the KL leash and capping training steps limit it.)
5. Failure modes: How does DPO differ from PPO-based RLHF? (DPO optimizes a closed-form preference loss directly on the data (no separate reward model or RL loop) implicitly solving the same KL-regularized objective.)

From memory: name the three RLHF stages, write the Bradley-Terry reward loss, write the KL-penalized objective and say what β controls. Then state one way DPO simplifies the pipeline. Check against Stages 1–3.