flaky-test-triage
verified6b06fc9c-2e76-4922-b6d9-1c7d0bf57078
Use when a test passes sometimes and fails sometimes — classify the flakiness cause (state, time, order, network, parallelism), reproduce in a loop, and fix at the root instead of retrying.
Metadata
Skill file
# Flaky Test Triage
**Use when** a test passes sometimes and fails sometimes — the worst kind of bug because it erodes trust in your entire CI. "Just retry it" is not a fix.
## The Flakiness Taxonomy
| Category | Signature | Example |
|----------|-----------|---------|
| **Shared state** | Fails after a specific other test ran | Test B fails only when Test A runs first |
| **Time dependence** | Fails at specific times/durations | Test checks `datetime.now()`, sleeps 1s, races a timeout |
| **Ordering** | Fails only in a certain order | Test relies on alphabetical/insertion order |
| **Network** | Fails on slow/loaded CI | External API call, no timeout |
| **Parallelism** | Fails only when tests run in parallel | Shared fixture, global variable, same file/port |
| **Hidden real-time** | Fails near boundaries | Clock crosses midnight mid-test, TZ mismatch |
## Reproduce Tactics
### 1. Run in a loop
```bash
# Run the failing test 100 times
pytest tests/test_thing.py::test_flaky --count=100 -x
# Or with a shell loop
for i in $(seq 1 50); do
pytest tests/test_thing.py::test_flaky -x -q || echo "FAILED on iteration $i" && break
done
# Install pytest-repeat for --count
pip install pytest-repeat
```
### 2. Control time & randomness
```python
# Freeze time — never let datetime.now() leak into tests
from freezegun import freeze_time
@freeze_time("2026-08-15 12:00:00")
def test_time_dependent():
assert get_discount() == 0.2 # "summer sale" — deterministic now
# Pin randomness
import random
random.seed(42)
```
### 3. Run in isolation to detect ordering
```bash
# Run JUST this test (no siblings):
pytest tests/test_thing.py::test_flaky -x
# If it passes alone but fails in the full run -> ordering/shared-state issue
# Run the full file (ordering matters):
pytest tests/test_thing.py -x
```
## Fixes Per Category
### Shared State → Reset fixtures
```python
# BAD: module-level state leaks between tests
_cache = {}
def test_a():
_cache["key"] = "value" # pollutes for test_b
def test_b():
assert _cache == {} # FAILS if test_a ran first
# FIX: autouse fixture that resets
@pytest.fixture(autouse=True)
def clear_cache():
_cache.clear()
yield
```
### Time Dependence → Inject the clock
```python
# BAD: real time
def test_expiry():
token = create_token(expires_in=1) # expires in 1 second
sleep(1.1) # race!
assert is_expired(token)
# FIX: inject a controllable clock
def test_expiry(clock):
clock.now = datetime(2026, 8, 15, 12, 0, 0)
token = create_token(expires_in=60, clock=clock)
clock.advance(61)
assert is_expired(token, clock=clock)
```
### Network → Wait-for-condition, not sleep
```python
# BAD: fixed sleep races under load
def test_service_ready():
start_service()
sleep(2) # sometimes not ready in 2s -> flaky
assert ping() == "ok"
# FIX: poll with timeout
def wait_for(condition, timeout=10, interval=0.1):
import time
deadline = time.time() + timeout
while time.time() < deadline:
if condition():
return
time.sleep(interval)
raise TimeoutError("condition not met")
def test_service_ready():
start_service()
wait_for(lambda: ping() == "ok")
assert ping() == "ok"
```
### Ordering → Sort / normalize
```python
# BAD: relies on dict insertion order or filesystem order
def test_list_users():
assert [u.name for u in get_users()] == ["Alice", "Bob"]
# FAILS if DB returns them in a different order
# FIX: sort before comparing
def test_list_users():
names = sorted(u.name for u in get_users())
assert names == ["Alice", "Bob"]
```
## Decision Table: Reproduce → Fix
| Can you reproduce? | Action |
|---|---|
| In a 50-run loop | Great — now isolate the cause and fix at root |
| Only in full suite | Ordering/shared-state — run in isolation, check fixtures |
| Only on CI (not locally) | Env diff: Python version, OS, resources, parallelism |
| Never (rare) | Add logging to capture state at failure, re-run CI with it |
## Guardrails
- **Never "just retry" a flaky test.** It masks real bugs and erodes trust. Fix the root cause or quarantine with a tracking issue.
- A flaky suite is worse than a slow suite. Prioritize fixing flakiness over adding features.
- Don't add `sleep()` as a "fix" — it's a race condition in disguise. Use `wait_for`.
- One flaky test can hide another — fix them one at a time.
## Pitfalls
| Pitfall | Fix |
|---------|-----|
| "Just retry it" (pytest-rerunfailures) | Masks bugs; use only as a TEMPORARY quarantine with a tracking issue |
| Adding fixed `sleep(2)` | Races under load; use polling with timeout |
| Ignoring flaky tests because they "usually pass" | They'll fail exactly when you need them (a release) |
| Fixing the wrong category | Reproduce FIRST, classify, then fix |
| Not tracking flaky tests | File an issue, tag it "flaky", link the test |
## Verify / Checklist
- [ ] Flakiness classified into a category (state/time/order/network/parallelism/real-time)
- [ ] Reproduced in a loop (at least one failure in 50 runs)
- [ ] Root cause identified (not just "works now")
- [ ] Fix applied at the root (inject clock, reset fixture, sort, wait_for)
- [ ] Test now passes 50x in a loop without failure
- [ ] No `sleep()` or blind retry introducedAttached files
No attached files.