embedding-model-selection
verified0d3c3e49-31ca-4b3c-8c01-18256c9578c9
Choose the right embedding model for RAG — MTEB scores, dimensions vs cost/size, max context, and self-host vs API tradeoffs.
Metadata
Skill file
# Embedding Model Selection for RAG
Use when you're choosing the embedding model that powers retrieval and need to
trade accuracy against cost, latency, storage, and context limits — a decision
that's annoying to reverse after your corpus is embedded.
## Dimensions: the cost lever everyone under-plans
Vector dimensionality drives storage and search cost almost directly.
- **384** dims: lightweight, fast, low-cost — good for simple/toy tasks, weakest
ceiling.
- **768** dims: the common "sweet spot" — meaningful quality without the storage
bill. ~3 KB/vector; 1M docs ≈ ~3 GB raw (float32).
- **1536** dims: the OpenAI default-style tier; high quality; 6 KB/vector.
- **3072** dims: max quality, but roughly **6× the storage cost** of 768 for
typically marginal recall gains.
The retrieval-quality curve flattens quickly after ~768 dims for most tasks:
going 256→768 buys real recall, but 1536→3072 buys little while multiplying
storage. For most RAG, **768–1024 dims is the practical sweet spot.** Note some
models (e.g. OpenAI text-embedding-3) are trained with **Matryoshka
representation learning**, letting you truncate to a shorter dimension with
graceful degradation — you can benchmark 768 vs 1536 from the same model/API
before committing.
## What to check before picking
- **MTEB score**: a general aggregate benchmark (retrieval subset included). A
useful sanity check, NOT your truth — your domain distribution differs, and a
strong retrieval score in MTEB can still underperform a model tuned for your
data or languages.
- **Max input context**: e.g. some models cap at 512 tokens (≈ 380 words), others
at 8K. If your docs are long and you embed whole sections, a 512-token cap means
you're chunking whether you like it or not.
- **Multilingual need**: embedding quality degrades sharply for languages the
model wasn't trained on. If you need 10+ languages, pick a multilingual model
(e.g. BGE-M3, multilingual OpenAI variants) — an English-tuned model will not
serve Polish/Spanish/etc. well.
- **Self-host vs API**: an API (OpenAI, Cohere, Voyage) is zero-ops and per-1M-token
priced; a self-hosted Sentence-Transformers/BGE model runs on CPU for small
scale and costs nothing per token but you own the infra + serving. Approximate
per-query API cost is negligible until millions of queries/yr, where it
becomes a real line item.
## Benchmark on YOUR data
Never trust the leaderboard alone. Build a small set of real queries with known
relevant docs, embed with 2–3 candidate models, and measure retrieval recall@k.
Because embeddings are cheap to compute and the corpus may be small, this is fast
and it's the only honest comparison. If you self-host, also measure latency and
memory under expected load (a single mid-range GPU can embed ~30M tokens/day).
## Pitfalls
- Picking by MTEB total score without checking your domain/language and dimensions.
- Choosing max dimensions "for quality" and paying ~6× storage for ~no recall.
- Embedding documents longer than the model's context silently truncating them —
chunk or split first.
- Not measuring recall on real queries; a fancy score on a public benchmark can
mask a model that fits your corpus poorly.
- Ignoring consistency: switching embedding models mid-project means
re-embedding the whole corpus (vectors are model-specific).
## Verify
- Report recall@k / precision@k of the chosen model on your golden query set.
- Confirm storage/latency of the chosen dimension fits your infra budget at your
corpus size.
- Confirm the model's context limit covers (or you chunk for) your longest real
documents.