## Multimodal Embedding Model Comparison

### Strategy decision table

| Criterion | Text-summarize (VLM) | Native multimodal | ColPali |
|-----------|---------------------|-------------------|---------|
| Visual fidelity | Low (captions lose detail) | High | Highest |
| Storage cost | Low (text vectors only) | Medium | High (multi-vector/page) |
| Query speed | Fast | Fast | Slower (max-sim scoring) |
| Setup complexity | Simple (any text vector store) | Medium | Higher (patch-level) |
| Best for | Text-first docs + illustrative images | Figures/charts carry answers | Layout-critical docs |

### Model options (check current versions)

| Model | Type | Notes |
|-------|------|-------|
| CLIP (OpenAI) | Multimodal | Original, good baseline |
| Jina CLIP v2 | Multimodal | Text+image, good zero-shot |
| Cohere Embed 4 | Multimodal | Single-vector, ~128k context, Matryoshka |
| voyage-multimodal-3 | Multimodal | Long context, competitive quality |
| ColPali (Vidore) | Patch-level PDF | Best for document retrieval; multi-vector |

### The embedder-matching rule (CRITICAL)
```
Query embedder == Index embedder (same model family)
```
Mixing a text-only query embedder with a multimodal index (or vice-versa)
produces broken cross-modal ranking. If you index with CLIP, you query with
CLIP. Period.

### PDF pipeline comparison

```
Classic: PDF -> OCR -> text extraction -> text chunk -> text embedder -> search
ColPali: PDF -> render page as image -> vision transformer -> patch embeddings -> search
```

Classic drops tables, charts, and layout. ColPali preserves everything but
generates more vectors and needs more storage.

### Hybrid approach (best of both)
1. Generate VLM caption for each image/page (for text search)
2. Keep native multimodal embedding for the image (for visual search)
3. At query time: feed BOTH retrieved image AND its caption to the answer VLM
4. This way you catch keyword matches AND visual matches

### Evaluation baseline
Build a figure-heavy eval set (questions whose answers live in charts/images),
measure recall@k for each strategy against that set. A "what does Figure 3 show"
query that retrieves the right image/page is a win.