No supervised fine-tuning, no human preference labels, no learned reward model. The model proposes answers, a Python function grades them, and the ones that score above their group's average get reinforced. That is the whole loop.
For each problem the model samples a group of 8 completions. Every completion is scored by a reward function, and each one's advantage is simply how far its reward sits from the group's mean. Completions above the mean get pushed up, completions below it get pushed down, and a KL penalty keeps the whole thing tethered to the base model so it can't drift into nonsense that happens to score well.
The interesting part is what's missing. Classic PPO needs a second network (a learned critic) to estimate how good a state is, and that critic costs as much memory as the policy. GRPO throws it away: the group average is the baseline. That single substitution is what lets this run on one consumer card instead of a cluster, and it's the technique behind DeepSeek-R1.
It also means the reward has to be verifiable. There's no human in the loop and no model judging taste: just answers_match(parsed, gold), a deterministic check. Grade-school math is a good fit precisely because correctness is decidable.
Correctness dominates by design. Everything else is scaffolding that keeps the gradient alive while the model is still learning to answer in a parseable shape.
Source: train/rewards.py. Mean group reward climbed 1.23 → 2.80 of 3.25 over 750 steps.
The first smoke run produced 0 tagged completions out of 80. Not 0% accuracy: the base model was often doing the arithmetic correctly. It just ignored the format instruction and answered in prose.
The reward parser looks for an <answer> block. No block means no parsed answer, which means reward 0, for every completion in the group. And when every member of a group scores identically, every advantage is zero, so the gradient is zero. The run wasn't learning slowly. It was a no-op burning GPU hours.
The fix had two halves, and it needed both:
What is 2 + 3? turn showing the exact tag structure. Tagged completions appear at step 0 instead of never.tag_presence_reward pays +0.125 per required tag present exactly once, instead of all-or-nothing. Now a completion with 3 of 4 tags outscores one with 2, so there is a gradient to climb toward the format even before any answer is correct.This is also why the playground sends that same one-shot: the live prompt has to be the prompt the model was trained and evaluated under, or the comparison would be measuring something else.
Stack: TRL GRPOTrainer + Unsloth FastLanguageModel, with vLLM colocated for rollout generation, on an RTX 5060. Peak memory landed at just over half the usable budget. The ceiling here was patience, not capacity.
A 1.5B model at 70% on GSM8K is not competitive with frontier models, and it isn't meant to be. The claim is narrower and, I think, more interesting: a measurable capability gain from reinforcement learning alone, on hardware anyone can buy, with every number reproducible from a seeded script in the repo. The forgetting check on ARC-Challenge is there for the same reason: it's the result that would have embarrassed the project if it had gone the other way.