## Semantic Cache Threshold Tuning

### The one knob that matters

The cosine similarity threshold trades correctness against hit rate:

```
threshold low  -> more hits, more WRONG answers (false positives)
threshold high -> fewer hits, safer answers
```

### Starting points

| Embedding model type | Suggested start | Notes |
|---------------------|-----------------|-------|
| Sentence transformers (all-MiniLM-L6-v2) | 0.75-0.85 | Cheap, coarse |
| OpenAI text-embedding-3 | 0.92-0.95 | Finer-grained |
| Cohere embed | 0.85-0.90 | Domain-dependent |

### Tuning procedure
1. Build a validation set: near-duplicates (should HIT) + distinct queries (should MISS)
2. Sweep threshold from 0.70 to 0.98 in 0.02 steps
3. At each step, measure: hit rate on near-dups, false-hit rate on distinct
4. Pick the threshold where false-hit rate is within tolerance (often <1-2%)
5. Grade a 1-5% sample of real hits with LLM-as-judge vs fresh LLM answer

### Cache key design
```python
# WRONG: key on question only -> cross-context collision
key = hash(question)

# RIGHT: key on question + context + tenant
key = hash(f"{context}||{tenant}||{question}")
```

### TTL guidance by content type
| Content | TTL |
|---------|-----|
| Static docs / FAQs | Long (days) |
| Prices / inventory | Short (minutes) or skip caching |
| User-specific data | Never cache (scope, don't share) |
| Agent tool results | Short (session-length) |

### What to measure in telemetry
- Hit rate (exact vs semantic, separate!)
- False-positive rate (from graded sample)
- Latency: cache hit vs LLM call
- Cost saved = (misses_avoided * input_tokens * price)

### Known-good baseline
Literature reports ~60-70% hit rates at >95% precision, but YOUR corpus
determines YOUR numbers - measure, don't copy.