preference-optimization-dpo
verified09f97a57-880f-4acb-a482-2790391137ad
Align an LLM to human preferences with DPO (and when to prefer RLHF) ā preference data quality, beta, reference model, and regression checking.
Metadata
Skill file
# Preference Optimization with DPO (and RLHF tradeoffs)
Use when your model produces correct-but-unpolished outputs and you have
**paired examples** of "this answer is better than that one" ā so you can teach it
*preference* (tone, helpfulness, safety, formatting choices) rather than a single
correct answer.
## DPO in one paragraph
Direct Preference Optimization (DPO) reparameterizes the RLHF objective into an
implicit reward derived from the policy itself (and a frozen reference policy),
then trains the model with plain supervised-style loss over a **static** dataset of
preference pairs. No separate reward model, no PPO rollout machinery ā dramatically
simpler, more stable to train, and cheaper. For most teams outside frontier labs,
DPO (or a variant) is the pragmatic default for preference/alignment work.
## When DPO vs when (classic) RLHF
- **Prefer DPO** when: you have a good static preference dataset, you want
stability and low compute, you don't have an RL engineering team, and you don't
need the model to explore at training time.
- **Prefer RLHF/PPO-based** when: you want *online* learning (the model generates
its own samples during training and a live reward judges them), or you already
have a reliable learned reward model. The downside is PPO's known instability
and heavy infrastructure.
- **Hybrid/"online DPO"**: re-generate preference pairs with the current model
during training ā get online-learning benefits without the PPO machinery.
## The data is 90% of the outcome
Preference tuning lives and dies on data quality ā spend more time here than on
hyperparameters.
- **Clean, validated pairs.** Each example is `(prompt, chosen, rejected)` where
`chosen` is *demonstrably* better. Garbage in ā you'll amplify the wrong style.
- **Balanced and diverse.** Avoid pairs where the distinction is only length (the
model will learn "longer = better"); mix in pairs that teach real judgment.
- **No leakage.** Keep eval/benchmark prompts out of training.
- **Pair count:** thousands of solid pairs matter more than tens of thousands of
noisy ones. Start with 1ā10k.
- **Synthetic construction** (e.g. have a judge model pick between two responses) is
common ā but validate with a held-out manual/LLM-judge review; synthetic noise
compounds.
## Key hyperparameters
- **`beta`**: the temperature/inverse-temperature of the implicit reward ā how hard
you push the model toward the preferred answers. Conventionally 0.1; typical
range ~0.05ā0.5. Too low ā mode collapse / overfit to chosen. Too high ā barely
changes. Start at 0.1 and adjust based on whether the model moves too much or too
little.
- **Epochs: train for ~1**, not 3ā5. DPO overfits fast; more epochs degrade quality
and cause reward hacking.
- **Learning rate**: smaller than SFT ā ~5e-6 to 5e-5 region for full FT; scale up
for LoRA-tuned DPO.
- **LoRA for DPO**: a low-rank adapter + `DPOTrainer` from TRL fits DPO into a single
GPU. Keep a frozen **reference policy** (`ref_model`) pinned ā DPO is
anchor/baseline-sensitive, so the reference must be the *pre-DPO* model.
## Evaluation (do not skip)
- Check the preference target DID change on held-out pairs.
- Then check **capability regression**: run standard benchmarks; a well-aligned
model that collapsed general ability is a failed experiment. Look at the data /
beta, not just the loss.
- Use both automatic metrics (preference accuracy on holdout) and *manual inspection*
of samples. Watch for **reward hacking** (model learns a cheap proxy ā verbosity,
hedging ā that scores well but isn't actually better).
## Common failure modes
- **Model over-tunes to the chosen answer verbatim** (memorization) ā usually too
many epochs or too-low beta / too-similar chosen/rejected.
- **Self-preference / length bias.** The implicit reward inflates with output length;
the model gets longer and longer. Add length-balanced pairs and/or cap length.
- **Reward hacking** ā model optimizes the proxy, not real quality. Keep a diverse
eval and inspect qualitatively.
- **Drift from base.** DPO on a small set can degrade general reasoning. Keep a
strong reference and rely on eval to catch it.
- **Dataset contamination** between preference pairs within one prompt ā duplicates
that teach contradictions.
## Verify
- Hold-out **pairwise accuracy** > baseline (chance ~50%) and improving.
- Generated outputs on fresh prompts visibly reflect the desired preference.
- Capability benchmarks (e.g. reasoning/code/general QA) do not regress above your
threshold.
- Manual review of a sample: preferred answers genuinely better, not just longer or
more sycophantic.