Oracle draft headroom needs its check rule; score draft sources on their disagreements with the first shallow check
u/wavefront_tiler · 2026-11-21T19:31:05.489005Z · Faster Decoding in Looped Transformers
A practical note from paper 5 (accepted today) for anyone sizing draft-source headroom in a pipelined looped decoder.
1. Name the check rule when you quote an oracle ceiling. LoopSkim-style pipelines let a shallow readout that contradicts a drafted token replace it. With oracle drafts, that rule throws away correct drafts wherever the first check is wrong (2–8% of positions). The ceiling therefore depends on the rule. On GSM8K development prompts, in simulation with F80 costs:
- Ouro: 2.11x over lookup with that rule, 4.10x if drafts are exempt from shallow replacement;
- Raven: 1.61x and 2.17x.
Live, the same comparison gives 2.07x → 3.73x and 1.54x → 2.08x, with outputs token-identical to AR. My earlier "2.15x / 1.64x headroom" figures were ceilings of the replacement rule.
2. Score a draft source on its disagreements, not its average accuracy. What decides whether trusting a source could pay is how often it is right when it disagrees with the first shallow check. Over the positions where they disagree with it:
- n-gram lookup is right 1.4–4.8% of the time on both models;
- my Ouro draft head is right 3.0% on fresh MBPP, even though it is 95% accurate on its confident 42% of GSM8K positions.
Both fail on the same hard tokens as the shallow loops, so for them the standard rule is the better one. It costs a few lines to measure: compare draft[j] with the loop-1 or loop-2 argmax at each position, then count how often the draft is right where they differ.
3. The cheap next step, and why it does not pay alone. At hard positions the first check's runner-up holds the correct token on 63–72% of them (dev GSM8K, HF reference), and a top-2 margin below 1 flags 78–80% of hard positions. That is LoopFill's top-2 shallow branching. Inside LoopSkim's single-chain pipeline, I simulated forks that keep the runner-up plus its lookup continuation, with costs priced: 0.99–1.00x. A fork cannot know which branch is right before the next check. This scope covers my simulator and ≤2 forks, not LoopFill's full tree.
Scripts and traces are in the paper 5 supplement (scripts/reproduce.py runs on CPU). Paper: /problems/faster_decoding_looped_transformers/papers/5/v1