property-based-testing
verifiedb7920c29-67fa-408c-b62a-505282539d0d
Use when example-based tests aren't enough — test invariants with randomly generated inputs, shrinking to minimal counterexamples. Use Hypothesis (Python) or fast-check (JS).
Metadata
Skill file
# Property-Based Testing
**Use when** example-based tests cover only a handful of cases and you want automated discovery of edge cases. Write properties (invariants that hold for all inputs), and let the framework generate counterexamples.
## What is a Property?
A property is a statement that must be true for ALL valid inputs:
```python
# Example-based (tests 3 specific inputs):
def test_reverse_examples():
assert reverse([1,2,3]) == [3,2,1]
assert reverse([]) == []
assert reverse([5]) == [5]
# Property-based (tested for hundreds of random inputs):
from hypothesis import given, strategies as st
@given(st.lists(st.integers()))
def test_reverse_roundtrip(xs):
"""Reversing twice returns the original list."""
assert reverse(reverse(xs)) == xs
@given(st.lists(st.integers()))
def test_reverse_preserves_length(xs):
"""Reversing doesn't change length."""
assert len(reverse(xs)) == len(xs)
```
## Common Properties (Invariants)
| Domain | Property | Hypothesis test |
|--------|----------|-----------------|
| Encode/Decode | `decode(encode(x)) == x` | Round-trip identity |
| Sort | `is_sorted(sort(xs))` and `multiset(sort(xs)) == multiset(xs)` | Sort idempotence + element preservation |
| Serialize | `deserialize(serialize(obj)) == obj` | Round-trip |
| Arithmetic | `add(a, b) == add(b, a)` | Commutativity |
| Transform | `transform(a + b) == transform(a) + transform(b)` | Linearity |
| Merge | `merge(xs, []) == xs` | Identity element |
## Strategies for Generating Inputs
```python
from hypothesis import strategies as st
# Primitive strategies
@given(st.integers(min_value=0, max_value=10000))
def test_positive_integers(n): ...
@given(st.text(alphabet="abcdef", min_size=1, max_size=100))
def test_non_empty_strings(s): ...
@given(st.floats(allow_nan=False, allow_infinity=False))
def test_finite_floats(f): ...
# Composite strategies
@given(st.lists(st.integers(), min_size=0, max_size=100))
def test_lists(xs): ...
@given(st.dictionaries(st.text(), st.integers()))
def test_dicts(d): ...
@given(st.datetimes())
def test_dates(dt): ...
# Custom composite
@st.composite
def order_payloads(draw):
return {
"id": draw(st.uuids()),
"items": draw(st.lists(st.text(), max_size=5)),
"total": draw(st.floats(min_value=0, max_value=1000)),
}
@given(order_payloads())
def test_process_order(payload):
result = process(payload)
assert result["status"] in ("ok", "error")
```
## Shrinking to Minimal Counterexamples
When a property fails, Hypothesis shrinks the failing input to the simplest reproduction:
```python
@given(st.lists(st.integers()))
def test_sort_idempotent(xs):
assert sort(sort(xs)) == sort(xs)
# Hypothesis output:
# Falsifying example: test_sort_idempotent(xs=[0, 0])
# (Discovered [5, 3, 1, 5, 3, 1, ...] and shrank to [0, 0])
# This tells you: sort fails when there are duplicates!
```
## Combining with Example-Based Tests
Properties are powerful but cryptic. Use both:
```python
# Example-based: readable, specific, documents intent
def test_sort_empty():
assert sort([]) == []
def test_sort_single():
assert sort([42]) == [42]
def test_sort_already_sorted():
assert sort([1, 2, 3]) == [1, 2, 3]
# Property-based: catches edge cases you didn't think of
@given(st.lists(st.integers()))
def test_sort_preserves_length(xs):
assert len(sort(xs)) == len(xs)
@given(st.lists(st.integers()))
def test_sort_elements_preserved(xs):
assert sorted(sort(xs)) == sorted(xs)
```
## Hypothesis Settings
```python
from hypothesis import given, settings, strategies as st
@given(st.lists(st.integers()))
@settings(max_examples=500, deadline=2000) # 500 cases, 2s each max
def test_expensive_property(xs):
...
# Or configure globally in conftest.py:
# from hypothesis import settings
# settings.register_profile("ci", max_examples=1000, deadline=None)
# settings.load_profile("ci")
```
## Guardrails
- Properties must be **falsifiable but true**. "sort(xs) is a list" is too weak (vacuously true). "sort(xs) == xs" is too strong (false for unsorted input).
- Set a `deadline` — properties that run too slow will be flagged.
- Avoid non-deterministic properties (random, time, network). Properties must be deterministic.
- Don't generate unbounded inputs — constrain with `min_value`/`max_value`/`max_size`.
## Pitfalls
| Pitfall | Fix |
|---------|-----|
| Properties that are always true | Strengthen: add `and len(result) == len(input)` |
| Properties that are sometimes false | Check: is your generator producing invalid inputs? Filter with `assume()` |
| Slow generators | Constrain sizes; use `@settings(max_examples=50)` |
| Non-deterministic property (flaky) | Don't use `datetime.now()` or `random` inside the property |
| Confusing failure output | Read the "Falsifying example" — it's the minimal repro |
## Verify / Checklist
- [ ] At least one property per new function (round-trip, idempotence, length preservation)
- [ ] Example-based tests still exist for readability and documentation
- [ ] Strategies are constrained (no unbounded integers, no infinite strings)
- [ ] Hypothesis settings configured for CI (deadline, max_examples)
- [ ] `assume()` used to filter invalid inputs (not as a crutch for weak properties)
- [ ] Shrinking produces readable counterexamplesAttached files
No attached files.