llm-quantization-gguf
verified57de2c11-e92c-412f-8bf0-2baf2ac82430
Run LLMs locally with quantization ā GGUF format, K-quant vs legacy levels, quality/size tradeoffs, and llama.cpp serving.
Metadata
Skill file
# Local LLM Inference with GGUF Quantization
Use when you need to fit a model into RAM/VRAM that's smaller than the FP16
checkpoint ā running on a laptop, edge box, or a cheap GPU ā and want to keep
quality as high as possible at the chosen size.
GGUF is the file format from the llama.cpp project (introduced Aug 2023) that
packs model tensors *and* rich metadata into a single file, memory-mapped
directly for fast loading. Quantization lowers each weight's precision to shrink
the file and speed inference, trading a little accuracy for a lot of size.
## Pick your precision level
- **Q8_0** (~8 bit): near-lossless; ~1.07 bytes/weight overhead plus scales.
Use when you have the room and want to guarantee quality.
- **Q5_K_M / Q6_K**: strong quality-to-size balance, good default for quality
conscious local use.
- **Q4_K_M** (K-quant, mixed): the community default for "quality is good, size
is small" ā around 4.5-4.85 bits/weight effective. Best all-rounder on limited
RAM.
- **Q2/Q3** and **IQ2/IQ3** (importance-quant): smallest sizes, most quality
loss; only for the tightest hardware. Q2 IQ variants can drop into the
"noticeably degraded" territory for reasoning.
- **F16/BF16**: full precision; the reference, big, slow-ish to load.
"K-quants" (Q4_K, Q5_K, Q6_K) use a block-based scheme that stores scale and
minimums per super-block, giving better quality at the same bit-width than the
older flat forms (Q4_0, Q5_0). The `_M` (medium) variants mix precisions ā
critical tensors get more bits, others fewer ā which is why they beat the plain
S-size at equal average bits.
The name's size suffix (e.g. a 7B/8B model's Q4_K_M ā 4-5 GB) tells you the file
size, not a quality rating ā same "Q4_K_M" across model families isn't a
measure of quality.
## Standard workflow
1. Download a pre-quantized GGUF from Hugging Face (TheBloke-style repos or the
official/community quantized releases) ā usually faster and better-tested
than converting yourself.
2. Alternatively convert your own: `convert_hf_to_gguf.py` from the HF checkpoint,
then quantize with `llama-quantize`. E.g.:
```bash
python llama.cpp/convert_hf_to_gguf.py ./merged-model --outfile model-f16.gguf --outtype f16
llama.cpp/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M
```
3. Serve with the llama.cpp server (OpenAI-compatible HTTP endpoint):
```bash
llama.cpp/build/bin/llama-server \
-m model-Q4_K_M.gguf \
--ctx-size 8192 \
-ngl 999 # offload layers to GPU if available
```
Then use it like any `/v1/chat/completions`-style endpoint.
## Choosing quantization vs the alternative
- If you just want to *run* a popular model locally, download a pre-quantized
GGUF. If you want max throughput and have a beefy GPU, consider a non-GGUF
path (AWQ/GPTQ via a serving engine) instead.
- Know that quantization is *not* cost-free for reasoning-heavy or code tasks ā
heavily reduced precision (Q2/IQ2) measurably degrades complex output even
when it passes casual chat.
## Pitfalls
- Picking the micro-quant to fit RAM exactly with zero headroom ā the context
window also consumes RAM/VRAM; `--ctx-size` plus the KV cache can blow the fit.
- Reading "4-bit" as one uniform precision when K-quants actually mix levels per
tensor for quality.
- Using a quant file for a different model architecture/family than your code
expects.
- Forgetting the metadata alignment lets the file be mmap'd ā keep the file
aligned in storage for fast load.
- Assuming quantization quality transfers across model families.
## Verify
- Load the GGUF in llama.cpp and run a few representative prompts (including a
reasoning/code task, not just chat).
- Check `llama-server` logs for layers offloaded vs on CPU and confirm memory
fits.
- Compare a Q4_K_M vs Q8_0 generation on a hard task and confirm the quality
loss is acceptable for your use.