## GGUF Quantization Level Guide

### Level reference (approx bits/weight)

| Quant | Effective bits | Quality | Use when |
|-------|---------------|---------|----------|
| F16/BF16 | 16 | Reference | Full quality, no size constraint |
| Q8_0 | ~8.5 | Near-lossless | Have room, want guaranteed quality |
| Q6_K | ~6.6 | Very high | Quality-conscious, some RAM headroom |
| Q5_K_M | ~5.7 | High | Good default for careful use |
| Q4_K_M | ~4.8 | Good (community default) | Limited RAM, best all-rounder |
| Q3_K_M | ~3.9 | Moderate | Tight hardware |
| Q2_K / IQ2 | ~2.5-3 | Noticeably degraded | Last resort, reasoning suffers |

### K-quant vs legacy naming
- `Q4_0` / `Q5_0` = legacy flat quantization (older)
- `Q4_K_M` / `Q5_K_M` = K-quants, block-based with per-tensor mixed precision
- `_S` = small, `_M` = medium, `_L` = large (size suffix, NOT a quality rating)

### Size estimator (7B/8B models)
| Quant | Approx file size |
|-------|-----------------|
| F16 | ~14-16 GB |
| Q8_0 | ~8 GB |
| Q4_K_M | ~4-5 GB |
| Q2_K | ~2.5-3 GB |

### RAM budgeting (critical!)
```
Total RAM needed = model_size + KV_cache + overhead

KV cache scales with --ctx-size:
  7B model: ~1MB/token of context
  8192 ctx  ~ 8 GB KV cache (FP16) — often MORE than the model itself
```

### Quantization vs alternatives
| Goal | Path |
|------|------|
| Run popular model locally | Download pre-quantized GGUF |
| Max throughput on beefy GPU | AWQ/GPTQ via vLLM/TensorRT-LLM |
| Convert your own fine-tune | convert_hf_to_gguf.py + llama-quantize |
| Extreme RAM constraint | Q2/IQ2 (accept quality loss) |

### Verification prompts
Always test with a *reasoning/code* task, not just casual chat - low quants pass
chat but fail on complex output:
```
"Write a Python function that finds the longest palindromic substring."
"Debug this logic error: [complex code]"
```