RLHF (Reinforcement Learning from Human Feedback)

Reinforcement learning from human feedback uses human preferences to shape model behavior. A common pipeline first trains on demonstrations, then fits a reward model to comparisons, and finally optimizes a policy against that reward with a constraint on how far it moves.

Keep the roles separate: SFT imitates demonstrations, a reward model scores outputs, and RL changes the policy using that signal. DPO offers a different preference-learning objective; it is not an extra stage of the same pipeline.

You can't directly backprop "be more helpful." So you build a proxy: show humans pairs of model outputs for the same prompt, ask which is better, and fit a reward model to those preferences. Then treat the LLM as a policy and use RL to produce outputs the reward model scores highly, and regularize its distance from an SFT reference model with a KL penalty. This can limit drift, but cannot guarantee that the policy will not exploit the reward model.

The KL term penalizes distributional drift. High reward and low KL are optimization signals, not proof of truthfulness, safety, or agreement with every user.

Reward model. Given human preference data where response $y_w$ is preferred over $y_l$ for prompt $x$, fit a scalar reward $r_\phi$ with the Bradley-Terry loss:

i.e. push the preferred response's reward above the rejected one's.

Policy optimization. Optimize the policy $\pi_\theta$ to maximize reward while staying near the reference (SFT) policy $\pi_{\text{ref}}$:

PPO uses a clipped surrogate objective that discourages large policy-ratio changes; clipping is not a hard bound on the final policy update. $\beta$ controls the leash: smaller values permit more drift; larger values put more weight on staying near the reference. DPO (Direct Preference Optimization) is the popular alternative: it reparameterizes the reward using the optimal policy of a KL-regularized objective and trains a preference-classification loss. Standard offline DPO avoids a separately fitted reward model and online rollout loop. Its derivation does not guarantee the same fitted policy as PPO on finite data.

Key takeaway: Comparisons can be easier to collect than calibrated scores, but human labels can disagree and contain bias. The reward is a learned proxy; KL regularization limits drift without guaranteeing good behavior.

Two pieces of a PPO-based RLHF setup: a reward-model loss and a sequence-level sampled KL-shaped reward. This is not a full PPO implementation.

1import torch
2import torch.nn.functional as F
3from torch import Tensor
4
5
6def reward_model_loss(r_chosen: Tensor, r_rejected: Tensor) -> Tensor:
7    # Bradley-Terry: preferred reward should exceed rejected reward
8    return -F.logsigmoid(r_chosen - r_rejected).mean()
9
10
11def kl_shaped_reward(
12    reward: Tensor, logp_policy: Tensor, logp_ref: Tensor, beta: float = 0.1
13) -> Tensor:
14    # Inputs have shape (batch,): one reward and summed response log-prob per sequence.
15    # Sequences are sampled from the policy; the expectation of this log-ratio is KL.
16    kl = logp_policy - logp_ref               # an individual sample may be negative
17    return reward - beta * kl
18
19
20r_chosen, r_rejected = torch.tensor([2.1]), torch.tensor([0.4])
21assert reward_model_loss(r_chosen, r_rejected) < reward_model_loss(r_rejected, r_chosen)

1. Conceptual: What are the three stages of RLHF? (SFT on demonstrations → reward model from human preference comparisons → RL (PPO) optimizing the policy against the reward model.)
2. Implementation: Why train a reward model from comparisons instead of absolute scores? (Pairwise comparisons avoid requiring a shared numeric scale, although labels can disagree. Bradley-Terry models preference probabilities from reward differences.)
3. Applied: What is the KL penalty for? (It keeps the optimized policy close to the SFT model, discouraging drift without guaranteeing that reward hacking is eliminated.)
4. Systems-level: What is reward hacking and how does the setup mitigate it? (The policy exploits flaws in the imperfect reward model for high score / low quality; the KL leash and capping training steps limit it.)
5. Failure modes: How does DPO differ from PPO-based RLHF? (Standard DPO uses a preference-classification loss derived from a KL-regularized objective, without a separately trained reward model or online rollout loop. It still trains network weights numerically.)

From memory: name the three RLHF stages, write the Bradley-Terry reward loss, write the KL-penalized objective and say what β controls. Then state one way DPO simplifies the pipeline. Check against Stages 1–3.

Sources and further reading

• Ouyang et al., Training language models to follow instructions with human feedback: SFT, preference rewards, and PPO-based training.
• Rafailov et al., Direct Preference Optimization: preference-loss derivation and its assumptions.