multimodal-rag-vision
verified65b67d6e-a38a-4a4b-b455-0441bcab3eb3
Build RAG over images, PDFs, and mixed documents — native multimodal embeddings vs text summarization, VLM understanding (ColPali-style), and image retrieval.
Metadata
Skill file
# Multimodal RAG (Images, PDFs, Charts)
Use when your documents contain meaning in **images, figures, charts, tables, or
scanned pages** that plain text extraction discards — invoices, manuals, papers,
slides. A text-only RAG pipeline that OCRs and drops the visuals loses the very
information users ask about ("what does the chart in Figure 3 show?").
## The core decision: text-summarize vs native multimodal embeddings
Two main strategies, with a real accuracy/cost tradeoff:
1. **Text-summarize first (classic):** a vision-language model (VLM) generates a
text description of each image/PDF page; only that text is embedded and stored.
- Cheap to store/query, works with any text vector store and any LLM.
- **Loses visual fidelity** — a caption can't capture fine detail, exact
coordinates, or subtle differences; retrieval answers then miss the specifics.
2. **Native multimodal embeddings (modern):** a multimodal embedding model encodes
text *and* images into one shared vector space so a text query retrieves images
directly. Examples: CLIP-style models, Jina v4, Cohere Embed / voyage-multimodal,
ColPali-style patch-level embeddings for PDFs.
- Preserves visual detail; cross-modal similarity (text→image) works.
- Heavier, costlier to store/run; some models are multi-vector (ColPali).
## ColPali-style PDF retrieval (skip the parser)
ColPali turns the classic "OCR → parse → embed" pipeline upside down: **render each
PDF page as an image at high DPI** and feed it to a vision-language transformer,
producing **patch-level embeddings** directly. There's no OCR step to fail, no layout
detector to confuse a multi-column abstract with a sidebar, no table extractor to
drop a row. Great for documents where layout *is* the meaning (reports, slides,
manuals). Trained as a document-retrieval model (ViDoRe benchmark), it wins on hard
visual reasoning; tradeoff is multiple vectors per page (more storage) and the need
for a compatible retrieval + optional rerank/parse pass.
## Choosing your approach
- **Text is the real content, images are illustrative** → text-summarize is fine and
cheapest; correctness mostly lives in the text.
- **Figures/charts carry the answer** (finance, science, specs) → native multimodal
embeddings or ColPali; test the difference on your own eval.
- **Single-vector multimodal models** (e.g. Cohere Embed 4, voyage-multimodal-3.x)
can now handle raw PDF pages in one embedding call, support long context
(e.g. ~128k tokens ≈ large documents), and Matryoshka dimensions (256–1536) so
you can trim storage without re-embedding. Often the best cost/quality balance on
typical enterprise corpora; ColPali-style still wins the hardest visual tasks.
- **Hybrid:** use a VLM caption *and* keep the image retrieval path — feed both the
retrieved image and its caption to the answer LLM.
## The multimodal RAG pipeline
1. Ingest varied media (PDF pages, images, tables), possibly with OCR for embedded
text.
2. **Embed** per your chosen strategy (native text+image embeddings, or VLM captions),
storing `path` + `media_type` + `embedding` per item.
3. Store in a vector DB that handles your vector dimensionality.
4. User query → embed with the *same* model (don't mix a text-only and multimodal
embedder — vector spaces won't align) → similarity search across chunks/images.
5. Retrieved text + images + (captions) → **vision-language model** that can accept
image inputs → grounded generation.
6. Return the response with references and the actual images the model used.
**Key correctness rule:** the query embedder and the index embedder must be the same
model family. Mixing a text-only embedder for queries with a multimodal index (or
vice-versa) produces broken cross-modal ranking.
## Pitfalls
- **OCR-only text loss:** charts/tables become meaningless token soup; verify what
your parser actually retained.
- **Information loss in captions:** summaries hide the details users ask about —
confirm on a figure-heavy eval set.
- **Mismatched embedders** (text vs multimodal) → garbage cross-modal retrieval.
- **Truncation/context:** raw PDF pages may exceed a model's token budget; use a
model with long context or chunk pages.
- **Storage blowup:** patch-level/multi-vector models multiply vectors per page — size
your vector store and budget accordingly.
- **Benchmark chasing:** ViDoRe V1 is saturated (>90 nDCG@5 for top models); evaluate
on your own corpus with V2-style harder cases in mind.
## Verify
- Build a figure-heavy eval set (questions whose answers live in images/charts) and
measure **recall@k** for text-summarize vs native-multimodal vs ColPali on it.
- Confirm retrieval actually surfaces the right image/page for a "what does the
figure show" query.
- End-to-end: the VLM answer correctly references the retrieved visual (faithfulness).
- Check storage/latency budget vs the quality gain before committing to the heaviest
approach.