review-ai-generated-code
verifieda9125704-0eb2-4058-aeb8-60384587bd3a
Review LLM/agent-generated code with extra skepticism — verify claims, check for subtle bugs and invented APIs, and confirm it actually runs. Use for any AI-authored diff before trusting it.
Metadata
Skill file
# Review AI-Generated Code
Use for any code authored by an LLM or coding agent before trusting it. AI code
can be syntactically perfect and semantically wrong — the fluency is a placebo.
Review it with extra skepticism at every rung of the trust ladder.
## 1. The trust ladder — climb only when the rung holds
Start at the bottom and climb. Never skip a rung.
| Rung | Check | Command / action |
|---|---|---|
| 1. Syntax | Does it parse? | `python -m py_compile file.py`, `cargo check`, `go build` |
| 2. Imports real | Do imported modules/libraries actually exist? | `python -c "import x; print(x.__version__)"` for every import |
| 3. APIs real | Do the functions/classes/flags it calls actually exist? | Read the library docs; verify each top-level API signature |
| 4. Logic correct | Is the algorithm sound, edge cases covered? | Manual trace, test with known inputs |
| 5. Verified | Does it pass tests and produce correct output? | `pytest`, run the script, check output |
Most AI bugs live at rungs 2–4: the code looks right but the library does not have
that function, or the flag is spelled wrong, or the logic is subtly inverted.
## 2. Common AI failure modes — a cheat sheet
| Failure mode | What it looks like | How to catch it |
|---|---|---|
| **Invented libraries** | `from magiclib import solve` — library does not exist | Try the import; pip install will fail or install the wrong thing |
| **Invented flags** | `json.dumps(data, escape_slashes=True)` — flag is `escape_slash=False` | Read the actual API docs for every flag |
| **Swapped operators** | `if a > b:` when it should be `a < b` | Test with boundary values |
| **Off-by-one** | `range(1, n)` when it should be `range(1, n+1)` | Test with `n=0`, `n=1`, `n=max` |
| **Swallowed errors** | `try: ... except Exception: pass` | Grep for silent `except` blocks |
| **Plausible-but-wrong SQL** | `SELECT * FROM users WHERE is_active = 'true'` — boolean column compared as string | Run against a test DB |
| **Wrong time/date math** | Mixing naive and aware datetimes, mismatched timezones | Test at DST boundaries, UTC midnight |
## 3. "Run it before you believe it"
The single most important rule for AI code: **run it**. Code that was generated
but never executed is unverified by definition.
```bash
# Minimum: run the main path and one error path
python -c "from new_module import main; print(main(test_input))"
python -c "from new_module import main; print(main(bad_input))" # expects error
# Run the AI's own tests, if it wrote any
pytest -q tests/
# If tests don't exist, write a smoke test yourself before accepting
```
## 4. Red-flag phrases from AI output
When the AI says any of these, stop and manually verify:
- "This should work..."
- "Assuming the API is available..."
- "In most cases..."
- "The standard approach is..." (but no source cited)
- Any API/import/link you cannot confirm in 30 seconds of searching
## 5. Worked example: catching a hallucinated API
An agent generates this line in a Python file:
```python
import requests
resp = requests.fetch("https://api.example.com/data")
data = resp.body() # not json(), not content — "body()" does not exist
```
The review walks the trust ladder:
1. **Syntax**: parses cleanly. ✓ (rung 1 holds)
2. **Imports real**: `requests` exists. ✓ (rung 2 holds)
3. **APIs real**: `requests.fetch()` does not exist (the real call is `requests.get()`
or `requests.post()`). `resp.body()` does not exist (the real attr is `.content`
or `.json()`). ✗ (rung 3 fails)
Two invented APIs in three lines. The rung-3 check — reading the actual library
docs for every top-level call — catches both immediately. Fluency without
correctness is typical of AI output and is exactly why rung 3 is not optional.
## 6. Prompting for verifiable output
When *you* are the one asking an AI to generate code, include these constraints
to reduce review burden:
```text
- Use only standard-library modules (or <specific, named libs>).
- After generating, write a short pytest that exercises the main path.
- Return the exact Python code block, no explanatory prose interleaved.
```
The first two constraints reduce hallucination surface; the third makes it
possible to copy-paste and run without manual extraction. You will still need to
review run-by-run (rung 5), but rungs 2 and 3 become faster.
## Guardrails
- Do **not** assume fluency equals correctness. Fluently wrong code is more
dangerous than obviously broken code.
- Do **not** accept an AI's explanation without checking the diff. The explanation
may describe what the AI *intended*, not what it *wrote*.
- Do **not** rubber-stamp because it "looks reasonable." Every rung on the trust
ladder must actually hold.
- Do **not** skip the "run it" step. AI code that was never executed is unsafe
by definition.
- Do **not** trust an AI-generated test that "passes" — the test may be testing
the wrong thing, or itself be buggy.
## Pitfalls
- **Assuming fluency = correctness** — the AI writes like a senior engineer but
invents APIs and inverts logic.
- **Accepting an explanation without checking the diff** — the AI's prose summary
may not match the actual code change.
- **Rubber-stamping** — "it's AI, must be right" or the opposite: "it's AI, I'll
just approve to be done."
- **Not running it** — the #1 cause of merging broken AI code; it passed its
own hallucinated tests.
- **Missing invented APIs** — an LLM will confidently generate `lib.never_existed()`
and you will not catch it without running the import.
## Verify / Checklist
- [ ] All imports resolve (no invented libraries or modules).
- [ ] Every API call, flag, and keyword arg was cross-checked against real docs.
- [ ] The code was run at least once with real input and real output.
- [ ] Tests were run; if the AI wrote them, they were inspected for correctness.
- [ ] No silent `except: pass` or swallowed errors.
- [ ] No red-flag phrases remain unresolved.
- [ ] The actual diff matches the intended change (did not creep or add unrelated edits).Attached files
No attached files.