vllm-model-serving
verifiedaf7ea4ff-7401-452f-ba05-3e08e78bf521
Serve open-weight LLMs in production with vLLM — continuous batching, PagedAttention, memory/KV-cache tuning, and OpenAI-compatible API.
Metadata
Skill file
# Serving LLMs in Production with vLLM
Use when you are moving an open-weight model (Llama family, Qwen, Mistral, etc.)
from "run a notebook" to a real API serving a bunch of concurrent users, and you
need high throughput on limited VRAM with an OpenAI-compatible endpoint.
vLLM is the de-facto production stack because it packages several researched
techniques into one server. You don't need to implement them, but you DO need to
understand them to set the right flags.
## The four big ideas (so the flags make sense)
1. **PagedAttention** — the KV cache is cut into fixed-size pages and allocated
like OS virtual memory instead of one giant contiguous buffer per request.
This slashes memory wasted on fragmentation and lets request slots be reused
the instant a sequence finishes. This is the core reason vLLM fits larger
batches on the same GPU.
2. **Continuous batching** (iteration-level scheduling) — instead of waiting for
the whole batch to finish, the scheduler admits/retires requests at every
decode step. When one request finishes, the next queued one takes its slot
immediately. This is what sustains throughput under mixed-length traffic.
3. **Flash Attention / optimized kernels** — fused attention kernels cut memory
bandwidth pressure in prefill.
4. **CUDA Graphs** — replays a recorded static-shape kernel graph to strip Python
overhead at small batch sizes.
## The few flags that matter
- `--model <repo-id>`, `--tensor-parallel-size N` (shard across N GPUs for big
models), optionally `--pipeline-parallel-size`.
- `--max-num-seqs`: cap on concurrent sequences per engine; raise toward your
peak concurrency (e.g. 256–512 for a hot API). Too low → queuing; too high →
memory pressure.
- `--max-model-len` / `--max-num-batched-tokens`: bounds the context window and
the tokens processed per step. A longer window costs more KV cache.
- `--gpu-memory-utilization` (default ~0.9): how much VRAM the KV cache pool may
consume. Lower it if you want reserve for other processes; too low hurts
batch size.
- `--served-model-name`: the name exposed by the OpenAI-compatible endpoint so
clients can `model=<that>`.
- Quantized checkpoints (AWQ/GPTQ/FP8) pass through `--quantization` for higher
throughput on the same GPU budget.
Start the OpenAI-compatible server:
```bash
vllm serve meta-llama/Llama-3.1-8B-Instruct \
--tensor-parallel-size 1 \
--max-num-seqs 128 \
--gpu-memory-utilization 0.9 \
--served-model-name my-llm
```
Then hit `/v1/chat/completions` with the usual OpenAI schema plus stream=true
for token streaming.
## Tuning notes
- Higher `--max-num-batched-tokens` raises throughput but can increase
time-to-first-token (TTFT) if long prefills block decode steps.
- For latency (chat), favor more `max-num-seqs` headroom and moderate batching;
for raw throughput, maximize batched tokens.
- Watch TTFT, tokens/sec, and queued-request latency separately — global
"throughput" hides a bad p50 for interactive users.
## Pitfalls
- Loading a model whose total weights exceed VRAM and watching OOMs you can't
explain — check weights fit with room for KV cache.
- Ignoring `max-model-len` so the KV cache pool has no room for batch depth.
- Blindly copying someone's H100 flags to a single 24GB card.
- Forgetting to enable streaming on the client, so TTFT feels broken.
## Verify
- `curl` a `/v1/chat/completions` request; confirm a well-formed response.
- Send 20–50 concurrent requests; report throughput, TTFT, p50/p95 latency.
- Confirm `nvidia-smi` shows memory reserved per `--gpu-memory-utilization` and
no OOM under load.