llm-gateway-model-routing
verifiedbde313d0-8150-4317-aa1e-e48bc893e606
Add an LLM gateway for one API across providers — model routing (cost/latency/capability), fallbacks, budgets, and observability with tools like LiteLLM.
Metadata
Skill file
# LLM Gateway & Model Routing
Use when your app calls multiple LLM providers/models and you need **one API**, plus
automatic **fallback**, **model routing**, **budget/rate controls**, and **unified
cost observability** — instead of hard-coding provider SDKs and duct-taping retries.
## What an LLM gateway does
A gateway is a routing + control layer that sits between your app and many LLM
backends:
- **Unified API** — your code talks to one OpenAI-compatible endpoint regardless of
provider (OpenAI, Anthropic, Gemini, local vLLM…).
- **Model routing** — send each request to the cheapest/fastest/most-able model that
can handle it (cost-based, latency-based, capability-based).
- **Fallbacks** — if provider A is down/rate-limited/context-overflow, transparently
retry provider B or a fallback model.
- **Budgets & rate limits** — per-team/customer virtual keys, spend caps, concurrency.
- **Observability** — centralized cost per model, latency, error rates, token usage.
**Migration is cheap:** point your SDK's `base_url` at the gateway; no app-code
rewrite.
## Options landscape
- **LiteLLM (open-source, self-host):** most production-ready general-purpose choice
for teams that want control and no per-request markup. Handles routing, fallback,
budgets, load balancing. Run as a proxy; config in YAML. ~15–30 ms added latency
per request.
- **OpenRouter (hosted):** zero infra, 300+ models, one key, but add a fee on
credits and no built-in fallback routing.
- **Managed gateways (Portkey and similar) / other OSS (Bifrost):** trade control for
managed observability (Portkey) or raw performance (Bifrost, high-throughput
Go-based).
Pick LiteLLM-style when you want full control and cost optimization at scale; hosted
when you want convenience and no server to run.
## Routing strategies
- **Priority/fallback (baseline — everyone should have this):** ordered list — try
model A, on failure model B, then C. LiteLLM calls these `fallbacks` and
`context_window_fallbacks` (for out-of-context-window errors).
- **Cost-based:** cheapest deployment by price/token for the request. Combine with a
hard **spend cap** per deployment/model: soft preference for cheap + hard ceiling
prevents surprise bills.
- **Latency/least-busy:** route to the fastest / least-loaded backend for interactive
apps.
- **Capability routing:** simple queries → cheap fast model; hard/edge queries →
frontier model. Studies (e.g. RouteLLM, FrugalGPT-style cascades) report cutting
cost **40–85%** while keeping near-frontier quality by sending only a fraction of
queries to the expensive model. Implement with a classifier or scoring heuristic on
prompt difficulty.
## LiteLLM proxy sketch
```yaml
# config.yaml
model_list:
- model_name: primary # public alias your app calls
litellm_params: { model: "gpt-4o", api_key: os.environ/OPENAI_API_KEY }
- model_name: primary
litellm_params: { model: "claude-sonnet", api_key: os.environ/ANTHROPIC_API_KEY }
router_settings:
fallbacks: [{ "primary": ["claude-sonnet"] }] # fail over gracefully
# cost-based-routing, budget/rate-limit settings, etc.
```
Then point your app at `http://gateway:4000` with a gateway key. The router picks a
deployment under the `primary` alias, applies fallbacks, and logs spend to a backend
(e.g. Postgres) asynchronously after each response.
## Monitoring & guardrails
- Track **cost per model per day** and **fallback activation rate** — alert if
fallbacks fire > ~5% of requests (a provider problem or a routing bug).
- Watch **spend per request**; alert on spikes above baseline.
- Virtual keys let you cut off a single team/customer without touching provider keys.
- Keep provider keys only in gateway config/secrets — never in app code.
## Pitfalls
- **Fallback ≠transparent:** a model switch silently changes behavior/latency.
Log *which* model served each request (`x-litellm-response-cost` / model headers)
and alert on high fallback rates.
- **Cost routing without a cap:** "always cheapest" can chase a bad deployment's
quality; add a quality/score gate and a hard budget.
- **Single point of failure:** the gateway itself must be HA if you rely on it (run
replicas, share Redis for usage counters across instances).
- **Not testing context-window fallback:** out-of-context errors need their own
fallback path, not the generic connectivity one.
- **Latency overhead creep:** every routing decision adds ms; keep the hot path lean
and benchmark end-to-end.
## Verify
- A request with the primary model "down" falls through to the fallback and returns a
coherent response.
- Cost analytics show per-model spend and per-key budgets correctly enforced.
- An over-budget virtual key is actually rejected/blocked (hard ceiling).
- End-to-end latency with routing/fallback stays within your SLO.