## DPO Hyperparameter Tuning Reference

### The three critical knobs

| Parameter | Typical range | What it does | Failure if wrong |
|-----------|--------------|-------------|------------------|
| `beta` | 0.05 - 0.5 (default 0.1) | Temperature of implicit reward | Too low: mode collapse. Too high: barely moves. |
| `epochs` | 1 (rarely 2) | Training passes | >1: overfits, reward hacks, length bias surges |
| `learning_rate` | 5e-6 to 5e-5 | Step size | Same as SFT: too fast diverges |

### DPO vs RLHF/PPO decision

| Criterion | DPO | RLHF/PPO |
|-----------|-----|----------|
| Requires reward model? | No | Yes |
| Training stability | High | Low (PPO is notoriously unstable) |
| Data | Static preference pairs | Can be online (model generates during training) |
| Compute | Lower (no PPO rollout) | Higher |
| Best for | Most teams, most use cases | Online learning, frontier labs |

### Data quality rules

- **Chosen must be DEMONSTRABLY better** — ambiguous pairs teach noise
- **Length-balanced pairs** — otherwise model learns "longer = better"
- **1-10k solid pairs** matter more than 50k noisy ones
- **No benchmark leakage** — keep eval prompts out of training

### Common failure modes

| Failure | Symptom | Fix |
|---------|---------|-----|
| Model over-tunes to chosen verbatim | Identical output regardless of prompt | Fewer epochs, higher beta |
| Self-preference / length bias | Outputs get longer every epoch | Add length-balanced pairs, cap length |
| Reward hacking | Verbosity/hedging scores well but quality drops | Diverse eval set, qualitative inspection |
| Drift from base | Reasoning/capability benchmarks regress | Strong reference model, eval on both preference AND capability |
| Dataset contamination | Duplicate prompts with contradictory pairs | Deduplicate, audit pairs within each prompt |

### Evaluation checklist
- [ ] Pairwise accuracy on holdout set > 50% (baseline chance)
- [ ] Generated outputs visibly reflect preferred style/quality
- [ ] Reasoning/code/general benchmarks do NOT regress
- [ ] Manual sample review: chosen answers genuinely better, not just longer

### LoRA for DPO (single GPU)
```python
from peft import LoraConfig
peft_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, peft_config)
# Then use DPOTrainer as normal — LoRA adapter fits on 1 GPU
```