semantic-chunking-strategies
verifiedf4ffe5b8-a61c-4daa-a7af-c7d817c62575
Split documents into RAG chunks the right way — semantic boundaries, overlap, metadata, query-context mismatch, and evaluating chunking choices.
Metadata
Skill file
# Semantic Chunking Strategies for RAG
Use when you're deciding **how to split your documents into the chunks that get
embedded and retrieved** — a decision with outsized impact on retrieval quality, and
the one most RAG pipelines get wrong by defaulting to fixed character counts.
## The core rule: chunk by meaning, not by character count
A paragraph cut mid-thought, a table split in half, or a code block severed between
lines all produce chunks that embed badly and answer poorly. Fixed-size-with-overlap
is a *fallback*, not a strategy. Prefer boundaries that preserve a unit of meaning:
- **By structure first:** split on headings, sections, subsections, paragraphs, list
items — whatever the document's natural semantic units are.
- **By sentence where structure is flat** (web pages, transcripts): group consecutive
sentences, ending a chunk at a sentence boundary, *not* mid-sentence.
- **Keep tables/code/figures intact:** never split a table row or a code function
across chunks — the retrieval of that element is what matters.
- **Semantic splitting tools** (e.g. embed-and-break on embedding-distance dropoffs)
can find boundaries automatically, but structure-aware splitting is often just as
good and more predictable.
## Chunk size + overlap
- **Chunk size should match what you'll actually retrieve and stuff into the prompt.**
If your answer needs whole sections, bigger chunks; if the model needs a focused
snippet, smaller. Typical ranges: ~300–1000 tokens.
- **Small chunks** → more precise retrieval but more context joins and more chunks to
manage; **large chunks** → more context per hit but dilution and higher cost.
- **Overlap ~5–15%** so sentence/paragraph boundaries aren't lost at the edges. Too
much overlap multiplies storage and dupes.
- **There is no universally "best" size** — it's coupled to your content, embedder,
and the questions you ask. Measure, don't guess.
## The query–chunk mismatch trap
The retrieval target should be the *answerable unit*, sized to the questions users
ask. Two failure modes:
- **Chunks too small** for the answer → retrieval finds a fragment but the model can't
assemble the full answer from pieces.
- **Chunks too big / multi-topic** → retrieval returns one big chunk holding the
answer buried among unrelated content → worse ranking, noisier context.
Make the chunk a coherent answerable unit and keep **one topic per chunk** where
possible. If an item answers a question only with its neighbors, include neighbors
(e.g. section + heading context) rather than splitting the answer apart.
## Attach metadata (it's part of chunking)
Keep context with each chunk so retrieval can filter and the model can cite:
- **Source path/URL, doc title, section/heading path**, page number, timestamp.
- **Filterable fields** (author, tenant, date, doc type) for hybrid filtering.
- **Parent lookup**: store a `parent_id` / chunk-index so you can reconstruct the
surrounding unit (parent-child retrieval) when a child chunk is retrieved but the
answer needs more context.
## Evaluate chunking (make changes measurable)
- Build or reuse a query→relevant-chunk eval set; measure **recall@k** for the chunking
in question.
- Compare candidate strategies (structure vs fixed-size; size variants) on the same
embedder and eval — the difference is often large and surprising.
- After re-chunking, **rebuild the index and re-run evals**; retrieval metrics follow
the chunking, not just the embedder.
## Pitfalls
- **Hard 500-char fixed chunks** with no structural awareness — breaks tables, code,
and paragraphs; retrieves meaningless fragments.
- **Splitting mid-sentence/mid-token.** Always honor sentence/token boundaries.
- **Zero metadata** → no filtering, no citation provenance, no parent reconstruction.
- **One mega-topic per giant chunk** → retrieval noise.
- **Chunking tuned to one embedder/model then silently changing models** → re-evaluate;
the "best" chunking shifts with the embedder.
- **Chunk size set by vibes** and never re-measured against an eval set.
## Verify
- Retrieval recall@k on your eval set improves (or at least doesn't regress) with the
chosen strategy vs the naive fixed-size baseline.
- Tables/code/figures survive intact and are retrievable as units.
- Retrieved chunks carry the metadata you need for filtering and citations.
- Answers that require section-level context can be reconstructed from the retrieved
chunks (parent-child works).