llm-fine-tuning-lora-qlora
verified5f5b4fce-65e6-4823-82b8-5f625fe734e6
Fine-tune large language models efficiently with LoRA/QLoRA and PEFT — rank/alpha/target-module choices, data prep, training config, and merge/export.
Metadata
Skill file
# Fine-Tuning LLMs with LoRA & QLoRA
Use when you need to adapt an open-weights model to your domain or task (style,
format, domain knowledge, custom behaviors) but cannot afford — or should not
attempt — full fine-tuning of all weights.
## Why LoRA instead of full fine-tune
Full fine-tuning of a 7B model needs ~100-120 GB of VRAM (several H100s).
LoRA freezes the base weights and inserts small **low-rank adapter matrices**
into the attention/MLP linear layers, training only ~0.1–1% of parameters. That
drops memory ~10–20× while retaining 90–95% of full-finetune quality on most
tasks. **QLoRA** additionally quantizes the frozen base to 4-bit (NF4), so you can
LoRA-tune a 7B on a single 24 GB consumer GPU (e.g. RTX 4090) — even a 70B with
655 GB of memory across ~2 such GPUs.
Key consequence: you keep a small adapter (tens of MB) on top of a frozen base.
Because adapters add **zero inference latency** once merged, you can train many
domain adapters and swap/swear them onto one base model.
## Data preparation (matters more than hyperparameters)
Fine-tuning quality is dominated by the dataset, not the config:
- **Format to chat template.** Convert each example to the model's chat template
(`tokenizer.apply_chat_template`) rather than raw concatenated text — this is
a top cause of broken fine-tunes.
- **Keep it small and clean.** 1–5k high-quality examples beat 100k noisy ones.
For style/format alignment, hundreds to low thousands is often enough.
- **Balanced labels.** For classification/following tasks, distribute classes so
the model doesn't just learn the majority.
- **No target leakage in output.** The *completion* portion must be what you want
the model to *generate*, not the instruction+answer repeated.
- **Dedup and de-contaminate.** Remove near-duplicate rows and any eval/benchmark
text from the training set or you'll see fake "amazing" scores.
## Choosing LoRA hyperparameters
The biggest lever is **which modules you target**, more than the exact rank.
- **`target_modules`**: attention projections `["q_proj","k_proj","v_proj","o_proj"]`
are the default and usually enough. Add MLP layers
(`gate_proj`,`up_proj`,`down_proj`) or just pass `"all-linear"` (what QLoRA does)
for more capacity when the task demands it.
- **`r` (rank)**: capacity of the adapter. Search powers of two: 8, 16, 32, 64.
Start at 16. Larger r for harder/more varied tasks or small base models; the
performance-vs-r curve usually has a "knee" — beyond it memory grows with no
quality gain.
- **`lora_alpha` (**scaling**):** controls update magnitude. Common practice is
`alpha = 2 × r` (e.g. alpha=32 when r=16) to keep the effective learning rate
stable; many guides also accept alpha = r. You can instead fix alpha and tune
LR.
- **`lora_dropout`**: 0.05–0.1 on small/noisy datasets for regularization; 0.0
on large clean datasets (dropout just slows convergence there).
- **`bias="none"`** (default) is correct; keep biases frozen.
## Training loop basics
Use `transformers.Trainer` (or TRL's `SFTTrainer`), `peft`, and `bitsandbytes`:
```python
from peft import LoraConfig, get_peft_model, TaskType
config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj","k_proj","v_proj","o_proj"],
bias="none",
)
model = get_peft_model(base_model, config) # or PeftModel.from_pretrained for QLoRA
```
- **Epochs**: 1–3. Quality typically plateaus fast; more epochs on a small set
overfits and the model repeats phrases.
- **Batch**: small batch (1–2) + `gradient_accumulation_steps` to reach an
effective batch of 8–32. Gradient checkpointing + `bf16` (or `fp16` on
non-Ampere+) to fit memory.
- **Learning rate**: 1e-4 to 2e-4 for LoRA (higher than full FT). Warmup ~3% and
cosine schedule.
- **Sequence length**: keep it near your real usage; absurdly long lengths blow
up memory for little gain.
- **QLoRA specifics**: `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)`.
4-bit base is slower to train than 16-bit LoRA but vastly cheaper memory-wise.
## Merge, export, serve, evaluate
- **Merge** adapter into base for deployment: `model = model.merge_and_unload()`,
then `model.save_pretrained(dir)` + `tokenizer.save_pretrained(dir)`.
- Export to GGUF (`llama.cpp/llama-quantize`) or ONNX if serving with those runtimes.
- **Always eval after fine-tuning**: check the target task AND capability
regression on general benchmarks (the tuned model often loses some general
ability). If it "forgot" base knowledge, the dataset skewed the distribution too
far or you overtrained.
- Use an unseen hold-out split to measure — never the training rows.
## Pitfalls
- **Wrong chat template** → model rambles or ignores instructions. Format data
with `apply_chat_template`, then inspect actual `tokenized` decoded output.
- **NF4 vs FP16 compute mismatch** → NaN/quality crash. Set `bnb_4bit_compute_dtype`
explicitly; keep base in 4-bit but matmuls in bf16/fp16.
- **Tuning all-linear when you only need style** → more overfitting, not more
quality. Start minimal, expand only if underfitting.
- **Learning rate too high on r=16 + alpha=32** → unstable loss. Start 1e-4.
- **Ignoring eval** → shipping a model that's great at the task, broken as an LLM.
- **Training on the eval set** → fake benchmark numbers that collapse in production.
## Verify
- Loss decreases but you *must* inspect generated samples on unseen prompts.
- Confirm trainable params are ~0.1–1% of total (`model.print_trainable_parameters()`).
- After merge, confirm the base unprompted behavior is largely preserved on a few
general questions while the target task shows the desired change.
- Check the adapter runs on the target runtime (GGUF/ONNX/transformers) end to end.