# Bias Mitigation in LLM-as-Judge

## Known biases and countermeasures

### 1. Position bias

Judges tend to prefer the first (or sometimes last) presented answer.

**Mitigation:**
- Run comparison twice with swapped order; keep only agreeing verdicts
- Grade each candidate independently on a rubric instead of head-to-head
- Randomize presentation order across samples

### 2. Length bias

Judges systematically prefer longer, verboser responses.

**Mitigation:**
- Reward conciseness explicitly in the rubric
- Normalize length before scoring (use a length penalty)
- Include short + long examples in the rubric prompts

```python
# Example rubric level that penalizes verbosity
RubricLevel(2, "Verbose but correct", "Answer is correct but uses excessive words. Prefer concise answers.")
```

### 3. Self-preference bias

A judge model favors text that looks like its own style.

**Mitigation:**
- Use a different model family for judging (e.g., Claude judges GPT)
- Use multiple judges and aggregate
- Grade on rubrics divorced from style

### 4. Sycophancy / agreeableness

Judges may reward hedging, politeness, or user-aligned answers.

**Mitigation:**
- Anchor the rubric on facts and content coverage, not tone
- Include adversarial examples (wrong but polite vs right but blunt)

## Validation protocol

1. **Label 100–300 samples by hand** (or use expert annotations)
2. **Compute judge-vs-human agreement:** Cohen's kappa for categorical, Pearson/Spearman for continuous
3. **Iterate on the rubric**, not the judge model, if agreement is low
4. **Periodic re-check:** judges drift as the judge model is updated or output distribution shifts

## Agreement thresholds

| Kappa | Interpretation | Action |
|-------|---------------|--------|
| > 0.80 | Strong | Trust for screening; still spot-check |
| 0.60–0.80 | Moderate | Use for relative comparisons; don't use as sole metric |
| 0.40–0.60 | Weak | Refine rubric; add more levels |
| < 0.40 | Poor | Don't use this judge for this task |