Every number, and where it came from
All evaluation is greedy (temperature=0) at seed 3407, on data the model never trained on. Each figure below is read off a JSON file committed under eval/results/. The scripts that produced them are in the repo and rerunnable.
GSM8K pass@1, full 1,319-problem held-out set
Lenient scoring (last number in the output)
+11.2 points from reinforcement learning alone, no supervised fine-tuning, no human labels.
| Model | Strict | Lenient | Gap |
|---|---|---|---|
| Base | 54.36% | 58.83% | 4.47 pts |
| GRPO-tuned | 69.98% | 69.98% | 0.00 |
The gap column is the quiet result. Strict requires a parseable <answer> block; lenient falls back to grabbing the last number in the output. The base model loses 4.47 points to formatting alone: problems it solved but couldn't present. The tuned model scores identically under both, which is a direct measurement of 100% format compliance. Quoted honestly, the improvement is +11.2 points lenient, the harder of the two comparisons. Strict-to-strict it is +15.6.
Did math RL damage anything else?
ARC-Challenge, 200 questions of general reasoning, never trained on
−1.0 point, within noise at n=200. Narrow RL on grade-school math did not measurably degrade general reasoning. This is the result that would have sunk the project had it gone the other way, which is exactly why it's on the page.
750 steps, 86.2 minutes, one 8GB card
Mean group reward 1.23 → 2.80 of a possible 3.25.
- Reward
- 1.23 → 2.80 (max 3.25)
- KL to base
- ~0.05 (stayed close)
- Completion length
- flat ~180 tok (no length hacking)
- Peak VRAM
- 3.64 GiB of ~6.6 usable
Flat completion length matters more than it looks. The most common way an RL run cheats a reward is by padding: rambling until something scoreable falls out. Length held steady while reward more than doubled, so the gain came from better reasoning, not more of it.
Training figures come from the run's log on the training machine; the plot above is the committed docs/reward_curve.png.
What quantization costs
| Model | Backend | pass@1 | gen tok/s |
|---|---|---|---|
| fp16 merged | vLLM (GPU) | 0.76 | 673.9 |
| f16 GGUF | llama.cpp (CPU) | 0.76 | 12.6 |
| Q4_K_M GGUF | llama.cpp (CPU) | 0.69 | 21.1 |
Q4 costs about 7 points of accuracy and buys 3.15× smaller weights (0.93 GB). The f16 GGUF row is the control: it scores identically to fp16/vLLM, which proves the drop is quantization and not the CPU backend. At n=100 the 95% confidence interval is roughly ±9 points, so treat the magnitude loosely. The direction is what's solid.
Latency vs throughput, single stream
| Stack | TTFT | Decode |
|---|---|---|
| vLLM fp16 (GPU) | 19.2 ms | 105 tok/s 674 batched |
| Ollama Q4 (GPU) | 139 ms | 227.8 tok/s |
These measure different things and the winner flips depending on which you need. vLLM answers 7× faster on first token and scales to 674 tok/s under batching, the right call for serving many users. Ollama Q4 decodes a single stream more than twice as fast. The live playground runs vLLM, because first-token latency is what a visitor actually feels.
Seed 3407 throughout, greedy decoding for every evaluation, held-out splits only. The raw JSON behind each figure lives in eval/results/, and the scripts that wrote it sit next to it in eval/. Where a number here is rounded, the file has the full precision.