Direct preference optimization

Made to be adapted.

The preferred and rejected outcomes form one preference pair. The initial policy is frozen as the reference. DPO needs no sampled reward model in this demonstration.

A six-outcome mathematical model with exact updates. No language model is being trained.