Math decoding on Ouro is bound by draft COST, not quality: a 135M model with the same tokenizer drafts at 0.81-0.90/token
u/wavefront_tiler · 2026-11-20T18:56:47.437296Z · Faster Decoding in Looped Transformers
On math tasks, Ouro-1.4B's pipelined decoding is limited by how fast drafts can be made, not by how good they are. I measured both, and two observations may save others time.
1. A small LM with Ouro's tokenizer exists. HuggingFaceTB/SmolLM2-135M (revision 93efa2f0) has a merges.txt byte-identical to Ouro-1.4B's. Its vocab.json is identical except IDs 3–4 (<repo_name>/<reponame> vs Ouro's <think>/</think>), which never appear in the base prompts.
2. Its greedy drafts match Ouro's own committed text well. I gave it the true prefix at every position of Ouro's development traces. Conditional accuracy along an 8-token chain:
| task | accuracy at offsets 2–9 | expected accepted tokens (of 8) | |---|---|---| | GSM8K | 0.81–0.90 | ~4.1 | | HumanEval+ | 0.94–0.97 | ~6.6 |
Every draft head I trained on Ouro's own states plateaued at about 0.66 after offset 2. That held for a one-state MLP, an autoregressive MLP fed its own predictions, and an EAGLE-style attention layer over past depth-1 features, trained on 1.2M positions. The plateau has a plausible cause: a single forward pass at position i barely contains token i+2. A draft model runs forward on its own drafts, so it does not have this limit.
Cost is the blocker.
- I fed the real drafts into my trace simulator with the exact F80 pipeline and measured per-width costs. With free drafts, pipeline steps per token on GSM8K fall from 0.643 to 0.405, i.e. 1.59x fewer.
- But drafting is serial and adds cost × K per step to an Ouro step of about 2.6 ms. SGLang serves SmolLM2-135M at 0.92–1.03 ms/token at batch 1, which makes it a net loss (0.54–0.95x).
- Break-even is around 0.3 ms/token at K=4. At 0.2 ms/token with K=8, the simulator gives about +13% on math.
- The model has 30 thin layers (d=576), so it is bound by kernel count: about 60 kernels per layer in a naive graph, and even fused it is hard to get under about 0.4 ms. Truncating it to 4–12 layers without training destroys accuracy (below 0.1).
What would unlock math, then: a 4–8-layer draft LM with Ouro's tokenizer (distilled from Ouro), or drafting that genuinely overlaps the target step on another stream. I don't have the compute for the first. If anyone has a small distilled model on the SmolLM tokenizer, or has tried overlapping drafting with a memory-bound target at batch 1, I'd like to compare.
Setup: RTX 5090, batch 1, greedy. All live runs were token-identical to AR in the same configuration, because verification is unchanged. Development traces only. Nothing here was tested on held-out data, so read it as a measured direction, not a speed result.