## ONNX Runtime Execution Providers Reference

### Provider priority chain
ORT tries providers in the order you specify and falls back silently.
**Always print `get_providers()` to confirm what actually loaded.**

### Provider table

| Provider | Where | Best for | Gotcha |
|----------|-------|----------|--------|
| CPUExecutionProvider | Everywhere | Fallback, small models | Default when nothing else loads |
| CUDAExecutionProvider | NVIDIA GPU | GPU inference | Needs `onnxruntime-gpu` package |
| TensorRTExecutionProvider | NVIDIA GPU | Max throughput | Needs TensorRT package; disable ORT graph optims |
| OpenVINOExecutionProvider | Intel CPU/GPU | Intel hardware | Model compatibility varies |
| CoreMLExecutionProvider | macOS | Apple Silicon | Limited op coverage |

### Silent fallback (the #1 gotcha)
```python
session = ort.InferenceSession(
    "model.onnx",
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"]
)
print(session.get_providers())
# If ['CPUExecutionProvider'] — your GPU code is actually running on CPU
```

### Graph optimization levels

| Level | What it does | When to use |
|-------|-------------|-------------|
| ORT_DISABLE_ALL | No optimization | Debugging |
| ORT_ENABLE_BASIC | Node fusions, constant folding | Minimal latency |
| ORT_ENABLE_EXTENDED | More aggressive fusions | Production (good default) |
| ORT_ENABLE_ALL | Maximum optimization | Production (preferred) |

### Online vs offline optimization
```python
# Online: optimizes at session init (every time)
session = ort.InferenceSession("model.onnx", sess_options=so)

# Offline: run once, save, load the optimized version
# Use onnxruntime.transformers.optimizer for transformer-specific fusions
```

### Quantization notes
| Method | Setup | Speedup | Accuracy risk |
|--------|-------|---------|---------------|
| Dynamic INT8 | Minimal (no calibration data) | Good | Low |
| Static INT8 | Needs calibration dataset | Better | Low (if calibrated well) |
| Full INT8 | Most setup | Best | Higher |

### IOBinding (skip the copy tax)
When decoding tokens in a loop (autoregressive generation), ORT copies
inputs from CPU -> GPU and outputs GPU -> CPU every step. IOBinding
pins buffers on device to avoid this:
```python
io_binding = session.io_binding()
io_binding.bind_cpu_input("input_ids", input_ids_np)
io_binding.bind_output("logits", "cuda")
session.run_with_iobinding(io_binding)
```