## Observability Dashboard Design

### RED metrics (service health)
| Metric | What | Alert on |
|--------|------|----------|
| Request rate | req/s per endpoint | Sudden drop or spike |
| Error rate | 5xx as % of total | >1% (or above error budget) |
| Duration | p50, p95, p99 | p95 above SLO |

### USE metrics (resource health)
| Metric | What | Alert on |
|--------|------|----------|
| CPU utilization | % used | >80% sustained |
| Memory saturation | % used | >85% or swap usage |
| Disk I/O | ops/s + latency | Latency spikes |

### Dashboard layout (single screen)
```
┌─────────────────────────────────────────────────────┐
│  RED: Rate  │  RED: Errors  │  RED: Latency (p95)  │
├─────────────────────────────────────────────────────┤
│  Traces: Top 5 slow spans (breakdown)               │
├─────────────────────────────────────────────────────┤
│  Log volume by level │  Error frequency by endpoint  │
└─────────────────────────────────────────────────────┘
```

### SLO-based alerting (not threshold-on-every-spike)
- **SLO**: 99.9% of requests complete in <200ms
- **Error budget**: the remaining 0.1% allowed
- **Alert**: when error budget burn rate exceeds 5x over 1h
- **Don't alert**: on individual blips within budget

### Correlation ID wiring
Every log line, every span, every metric counter should carry:
```
trace_id (or request_id for log-only services)
```
This is how you go from "alert: error rate up" to "show me the failing trace"
to "show me the logs for that exact request" in seconds.

### Dashboards checklist
- [ ] Single screen shows RED per service
- [ ] Drill down: click latency spike -> see trace waterfall
- [ ] Drill down: click error -> see correlated logs
- [ ] Owner assigned per dashboard (who fixes it?)
- [ ] New deploy is rollback-decidable from dashboard alone