grpo-reasoning-finetune
verifiedbc7f6b15-202f-412d-b18d-3f97567f2670
Fine-tune a base model to reason with Reinforcement Learning from Verifiable Rewards (RLVR) using GRPO — reward function design, group sampling, KL control, and when RL beats SFT for math/code/agentic tasks.
Metadata
Skill file
# Reinforcement Fine-Tuning with GRPO (RLVR)
Use when supervised fine-tuning (SFT) has stalled and you want a model to **get
better at tasks with objectively checkable outcomes** — math, code that compiles
and passes tests, exact-format answers, tool/agent rollouts with a verifiable end
state. This is the technique family behind DeepSeek-R1 and current reasoning
models.
## Why RL instead of more SFT
SFT teaches the model to *imitate* good answers from the training set. RL teaches
it to *search and improve*: the model explores many candidate answers, and we
reward the ones that are *correct*, even if the training set never contained
them. For tasks with verifiable outcomes this is often the biggest single jump
in capability-per-training-dollar you can get.
**Reinforcement Learning from Verifiable Rewards (RLVR)** is the specific setup:
the reward comes from a checkable function (exact-match, unit tests passing,
format valid) rather than a learned reward model. This removes the reward-model
bottleneck and lets you scale cheaply.
## The algorithm: GRPO
**Group Relative Policy Optimization (GRPO)** (Shao et al. 2024, used for R1):
for each prompt, sample a **group** of G candidate responses (typical G=8–64)
from the current policy. The reward for each response is normalized by the mean
and std of the *group's* rewards. Advantages computed from that relative baseline
drive the policy update, with a KL penalty keeping the policy near the reference
model.
Why GRPO over PPO: PPO needs a value/critic network that predicts the expected
return (memory-hungry, fiddly to train). GRPO **drops the critic** entirely and
uses the group mean as the baseline. That means far lower GPU memory and much
simpler training — which is why it became the default for reasoning models.
Key concept — **KL control**: you can't let the model blow away from the
reference into degenerate output. The config has either a hard `kl_coef`
(per-token KL penalty) or a dynamic adaptive KL controller that raises/lowers the
coefficient to stay inside a target range. Too loose -> reward hacking; too tight
-> no learning.
## Writing reward functions
This is where the skill lives. Reward functions are Python callables taking a
list of completions (and the prompt / ground truth) and returning a float per
completion. Good practice:
- **Return continuous, shaped signals when possible** — a binary 0/1 is hard to
learn from. For code, use the *fraction of test cases that pass* rather than
"all or nothing". For math, grade partial steps or at least exact-match vs
empty.
- **Compose multiple rewards and weight them.** A standard pairing (used in the
AWS SageMaker RLVR recipe) is:
- a *format* reward for producing the required answer envelope (e.g. wraps
final answer in `\boxed{...}` or a `<answer>` tag), and
- an *accuracy* reward for the content being correct.
You need the format reward to keep output parseable; you need the accuracy
reward to improve. Reward hacking often shows up as the format reward
succeeding while accuracy stays flat.
- **Be ruthlessly exact about what "correct" means.** For math, normalize the
answer (strip whitespace, evaluate `1/2` vs `0.5` before comparing), otherwise
identical-right answers get different rewards and gradient noise kills you.
- **Watch for reward hacking:** if a reward can be satisfied without doing the
task (e.g. always outputting a well-formed envelope), the policy will find it.
Sanity-check by tracking each reward component live during training.
## Training setup with TRL
The `TRL` library (`GRPOTrainer`) is the standard path. Minimal shape:
```python
from trl import GRPOCConfig, GRPOTrainer
config = GRPOCConfig(
model_name_or_path="Qwen/Qwen2.5-0.5B-Instruct",
num_generations=8, # group size G
max_completion_length=512,
beta=0.04, # KL coefficient
learning_rate=1e-6, # RL lr, much lower than SFT
reward_funcs=[format_reward, accuracy_reward],
output_dir="grpo_out",
logging_steps=1,
)
trainer = GRPOTrainer(config=config, train_dataset=dataset)
trainer.train()
```
The `dataset` is a `datasets.Dataset` of `{"prompt": ..., "answer": ...}` rows.
Start from a model that is already *instruct-tuned* and *reasonably competent at
the task* — RL sharpens and extends, it does not teach from zero.
## Hyperparameter levers (get these right)
- **Group size G (num_generations):** more samples per prompt = better baseline,
more exploration, but more cost. 8 is a reasonable start; scale up for harder
tasks where you need rare successes to appear in the group.
- **Learning rate:** RL updates are small — `1e-6` to `1e-5` typically, a
fraction of SFT's lr. Too high -> instability/oscillation.
- **beta (KL coefficient):** tune by watching pass-rate vs KL divergence. If the
pass rate plateaus while KL explodes, your reward is gameable.
- **max_completion_length:** reasoning models emit long CoT. Undersize it and
correct solutions get truncated and misrewarded.
- **Epochs / steps:** RL can run far fewer steps than SFT because each step does
a lot of generation. Watch reward-per-step curves, not epochs.
- **Multiple reward functions:** keep them in a list; TRL sums the *weighted*
rewards. Weigh components so no single easy reward dominates.
## When GRPO/RLVR is NOT the right tool
- Task answers can't be checked automatically (open-ended creative writing).
Prefer preference methods (DPO) or learned reward models there.
- You have a small, clean labeled set and just need to imitate it — SFT first.
- No verifiable signal and no budget for group sampling — RL is sample-hungry.
## Pitfalls
- **Reward hacking** (the #1 failure): policy games the easy reward. Mitigate by
shaping rewards, composing multiple, and watching component curves.
- **KL collapse / mode collapse:** model repeats one high-reward template. Tighten
KL or add diversity to the group.
- **Format vs accuracy divergence:** if format reward hits 0.99 and accuracy is
flat, the model learned phrasing, not reasoning.
- **Instruct-tuning a raw base model mid-RL:** start from an instruct model; raw
bases produce garbage that the rewards then reinforce.
- **Overly sparse 0/1 rewards** -> slow, unstable learning. Shape them.
- **Comparing across group sizes unfairly:** normalize rewards within the group
(GRPO does this automatically as its baseline) — don't feed raw rewards as if
comparable across runs.
## Verify
- Log per-reward-component curves and confirm *both* format and accuracy rise.
- Run the trained model on a held-out eval set (e.g. GSM8K / your test split)
and compare pass@1 against the SFT baseline and the starting instruct model —
a 2-3x lift on a hard eval is the expected headline result for math.
- Confirm KL stayed bounded (no runaway divergence from the reference).
- Red-team the output: generate many samples and confirm format compliance and
no reward-hacking templates (all-identical answers, empty reasoning that scores).