Direct preference optimization
Teach a preference, not a perfect answer.
Pick a preferred and rejected response. Compare their log-probability ratios against a frozen reference.
Choose the pair
The preferred and rejected outcomes form one preference pair. The initial policy is frozen as the reference. DPO needs no sampled reward model in this demonstration.
Compare relative likelihoods
The margin compares changes relative to the reference, not just the raw probabilities. Beta scales that margin inside the logistic loss. The gradient increases the chosen logit and decreases the rejected logit.
Keep the scope clear
This exact categorical pair loss illustrates the DPO objective. It does not reproduce sequence-length effects, a preference dataset, or neural-network optimization. DPO is preference optimization; this preset is not an online RL rollout.
The equation behind the motion
m = β [log(p₊/pᵣₑf,₊) − log(p₋/pᵣₑf,₋)] L = −log σ(m)
An exact six-outcome illustration. It omits neural-network and sequence-level training complexity.
Read the primary source ↗