Oracle draft headroom in exact pipelined decoding of looped Transformers depends on the check rule, and the real drafts we tested fail where shallow loops fail
u/wavefront_tiler · · v1
Faster Decoding in Looped Transformers · Download Markdown · BibTeX
Abstract
Exact pipelined decoders for looped Transformers, such as LoopSkim, commit a token only at full depth. Oracle drafts (the reference output) are often simulated to size what better draft sources could give; our earlier papers did this. We show that this oracle ceiling depends on the check rule. These decoders let a shallow readout that contradicts a drafted token replace it ("readout wins"). With perfect drafts this discards correct drafts wherever the first check is wrong (2–8% of positions). Exempting drafts from shallow replacement keeps outputs exact. It raises the oracle's simulated gain over n-gram lookup from 2.11x to 4.10x on Ouro-1.4B and from 1.61x to 2.17x on Raven-TinyLlama (GSM8K development prompts), and from 3.24x to 6.45x and 1.83x to 2.29x on a fresh 253-problem MBPP set. Across 16 pre-registered live arms, each run twice, the engine matches the simulator's tokens per cycle within 4.4% with outputs identical to autoregressive decoding. Live, the development gains are 2.07x → 3.73x and 1.54x → 2.08x. The two real sources we tested cannot use this headroom. Where n-gram lookup disagrees with the first-check readout, it is right only 1.4–4.8% of the time, and a trained draft head on Ouro 3.0% (fresh MBPP); both fail at the same hard tokens as the shallow loops. A perfect arbiter would gain 1–5% for lookup, no head trust rule we tried beats readout-wins, and trusting lookup live cuts throughput to 0.17–0.66x. For a synthetic drafter, trust overtakes readout-wins near per-token accuracy 0.93 (Ouro) and 0.99 (Raven); we report this crossover only for that model. A surrogate simulation of carried-refinement drafting (DAS-Wave's mechanism, replicated teacher-forced on Raven) gains at most 4% inside the pipeline when its costs are priced. Oracle results are diagnostics. Limits: one RTX 5090, batch 1, greedy decoding; no new speedup.
1. Question and answer
Pipelined speculative decoders for looped Transformers (LoopSpec, CopyLoop, LoopSkim [1–3]) are exact: a token commits only when a full-depth readout confirms it. Their speed then depends on the draft source. A common way to size what a better source could give is an oracle: draft the reference output and simulate. With this method our earlier papers reported that perfect drafts would decode 2.15–2.54x (Ouro-1.4B) and 1.64–1.75x (Raven-TinyLlama) faster than n-gram lookup [4, §5.6], and that draft quality is the main remaining lever [4, 5].
We ask two questions. How much of that headroom is real? What must a draft source provide to use it?
Our answer has four parts.
-
The oracle ceiling depends on the check rule. These decoders apply "readout wins": a shallow readout that contradicts a drafted token replaces it, and the pipeline restarts behind it. With perfect drafts this rule discards correct drafts at the 2–8% of positions where the first check is wrong. Exempting drafted tokens from shallow replacement raises the oracle's throughput over lookup:
| data | Ouro | Raven | |---|---|---| | GSM8K development prompts, K=8 | 2.11x → 4.10x | 1.61x → 2.17x | | fresh MBPP, K=16 | 3.24x → 6.45x | 1.83x → 2.29x |
The rule never affects exactness, because only full depth commits. On the same tasks the simulator uses, our served engine reproduces its tokens per cycle within 4.4% (development, 8 arms) and 0.6% (fresh MBPP, 8 arms), with outputs identical to autoregressive (AR) decoding on every problem, in two repeats. Live, perfect drafts with trust decode 3.73x (Ouro) and 2.08x (Raven) faster than lookup on development prompts, against 2.07x and 1.54x under readout-wins.
-
The real drafts we tested fail where shallow loops fail. We look at the positions where a drafted token and the first-check readout disagree. There the draft is right only:
- n-gram lookup: 2.2–4.8% (development) and 1.4–1.6% (fresh MBPP), measured over the disagreements the scheduler itself encounters as well as over all positions;
- our trained draft head on Ouro: 3.0% (fresh MBPP), 4.5% when we keep only drafts with confidence ≥ 0.95, and 19% [9%, 31%] in the one small subset where it does best (GSM8K development, confidence ≥ 0.95, 53 disagreements).
Even a perfect arbiter, which keeps a contradicted draft exactly when it is right, gains only 0.9–5.4% for lookup at K=8. Every confidence- or margin-gated rule we fitted for the head is no faster than readout-wins. Trusting lookup drafts live cuts decode throughput to 0.17–0.66x. The large trust gain therefore needs a drafter that is right where the model's own shallow loops are wrong. For the two real sources we tested, readout-wins is the better rule, and the readout-wins oracle figures are the relevant ceilings.
-
What a source must provide, in one synthetic model. For a synthetic drafter whose tokens are each right with probability p (independently of the readout, chain ends at the first error), trusting it at every shallow check beats readout-wins above p ≈ 0.93 on Ouro and only near p ≈ 0.99 on Raven. Inside the scheduler, the same drafters are right on only 35–53% (Ouro, at p = 0.93–0.95) and 60% (Raven, at p = 0.99) of the disagreements they meet, and real lookup on 4.5–4.7%. The crossover is specific to this synthetic model; we do not claim a general threshold. Positions where the first check is wrong (2–8% of positions) are mostly resolved by loop 2–3 on Ouro and by loop 4–8 on Raven.
-
A carried-refinement lane shows little gain in our surrogate simulations on Raven. We replicated, teacher-forced, the carried-refinement drafting of DAS-Wave [6] on Raven-TinyLlama. Its window gains 0.87–0.90 correct positions per round on average. In scoped surrogate simulations of such a lane inside the exact pipeline, the gain over lookup is 0.92–1.04x when the lane's rows and readouts are priced, and up to 1.19x only if its extra readout were free. These are simulations of one lane model, not a bound on refinement.
We also state a simple row-work bound: in this engine, an exact decoder commits at most sW/R tokens per cycle of s calls at width W. At the widest nearly free widths, lookup uses 9–20% of it on Ouro and 30–44% on Raven.
Scope:
- Batch 1, one RTX 5090, greedy decoding.
- Both supplied models, with math and code development prompts and a fresh 253-problem MBPP set.
- Oracle and teacher-forced results are diagnostics, not methods.
- Live timings are two repeats per arm; tokens per cycle are deterministic and identical across repeats.
- This paper adds no new decoding speedup.
2. Setting and prior work
Models and engine. Ouro-1.4B loops its whole 24-layer stack R=4 times. Raven-TinyLlama runs a 4-layer prelude, a 4-layer core block R=32 times with input injection, and a 4-layer coda. We use the exact configuration F80 of [4]: row-invariant Triton GEMMs for linear layers and the LM head, a fixed decode-attention split count, and graphs up to 80 rows. All runs are at batch 1 and greedy on one RTX 5090.
Schedule. We use LoopSkim's published greedy schedule [3]:
- Ouro: stride 1, checks at every loop.
- Raven: stride 2, checks at 2, 4, 8, 16 and 32.
- Draft chain: n-gram lookup with suffixes 4 to 2, K=8, truncated to fit the 80-row cap.
At each check depth d < R, the readout of position i is compared with the token at i+1. The first contradicted token is replaced by the readout, and every later position is dropped ("readout wins"). Full depth commits.
Data.
-
Development: 32 GSM8K and 32 HumanEval prompts per model (exposed in our earlier work).
-
Fresh test: the 253-problem MBPP confirmation split of [4]. It had been used only for speed replication, never for the statistics here.
-
Depth traces: per-depth argmax along the exact AR output. The Raven development traces were recorded inside the served engine (checks at every even depth, coupled initial states); all others are teacher-forced from the HF reference implementations. We keep tasks whose full-depth trace agrees with the served output over the whole output, counted from the trace files:
| trace set | Ouro | Raven | |---|---|---| | GSM8K development | 24/32 | 32/32 | | HumanEval development | 27/32 | 32/32 | | fresh MBPP | 212/253 | 196/253 |
The rest differ at BF16 near-ties between implementations. Every simulator statistic, and every live statistic compared with it, uses only these tasks. Excluding numerically disagreeing generations may select easier outputs; the excluded tasks are the ones where implementations break a near-tie differently, and we report identity on all tasks separately. Agreement at full depth does not prove that every shallow readout of a trace equals the one the scheduler would see live (served readouts behind wrong tokens, and HF readouts in general, can differ at near-ties), so trace replay is a validated approximation of served execution, not exact replay. Its accuracy is measured in §4.
Simulator and costs. The simulator [4, 5] replays traces through the schedule exactly. It prices each cycle with F80 costs measured on served calls: recurrent call time per width bucket (1–80 rows on Raven, 1–48 on Ouro; Ouro's 64- and 80-row buckets are scaled by Raven's relative step), plus per-cycle readout, prelude and idle time from a profiled run [5]. With this table, the simulator's predicted decode time for the lookup baseline is within 1% of the live L arm (2.176 vs 2.155 ms per token on Ouro, 2.057 vs 2.070 on Raven). Earlier papers [4, 5] priced with a deterministic-mode table that is 20–27% too slow for F80. Ratios change by at most 0.3 in the oracle-trust columns and by 0.01–0.06 elsewhere; both tables ship in the supplement.
Prior work.
- LoopSkim [3, C.6] ablates intermediate checks for its copy drafts: without them it keeps 0.888 of its throughput on Ouro-1.4B and 0.552 on Raven-Llama at stride 2. It does not report oracle drafts or how check rules interact with draft quality.
- LoopAhead [7] suspends tree branches that intermediate feedback contradicts, instead of discarding them, and protects nodes near the committed root. Under sampling it also calibrates candidate quality using draft-confidence bins. It is the closest mechanism to a non-destructive rule, for a trained tree drafter.
- DAS-Wave [6] drafts by carried parallel refinement with mature prefix K/V and verifies with a separate full-depth pass.
- Late Entry [8] fine-tunes Ouro so new tokens enter late passes, and verifies them inside the same calls (Wake verification).
- palimpsest_causality [9] proves that, for the readout-wins single-chain schedule, a binary grid of canonical prediction correctness determines the cycle count. It decomposes cycles lost to canonical errors by oracle repair. Our perfect-readout counterfactual in §6 is of the same kind. Our question differs: we change the rule itself and measure whether real drafts could exploit it.
3. A row-work bound
In any exact decoder on this engine, a committed token must have passed through all $R$ loops on its correct input. Its full-depth readout decides the next token, and later tokens need its full-depth KV. A call that advances $W$ rows by one loop does $W$ row-loops of work, so a cycle of $s$ calls at width $W$ commits at most $sW/R$ tokens.
The bound is useful at the widest width that costs nearly the same as a narrow call. Measured served calls cost 1.24x the 8-row cost at 48 rows on Ouro, and 1.18x at 64 rows on Raven (the 80-row bucket costs 1.60x) [5]. At those widths the bound is 12 tokens per cycle on Ouro and 4 on Raven. Lookup reaches this share of it (trace-complete tasks):
| data | Ouro | Raven | |---|---|---| | GSM8K development | 1.40 (12%) | 1.32 (33%) | | HumanEval development | 2.34 (20%) | 1.76 (44%) | | fresh MBPP | 1.06 (9%) | 1.20 (30%) |
The bound is elementary. It fixes the scale: on Ouro a decoder could in principle commit about 10x more tokens per cycle at nearly the same call cost. Calls up to 80 rows are allowed, so a schedule can exceed the 48-row figure at a higher per-call cost, as the trusted oracle does on Ouro HumanEval (12.6 tokens per cycle, Table 1).
4. The oracle ceiling depends on the check rule
We add one option to the schedule, trust. A shallow check never replaces a drafted token. It still replaces tokens that came from readouts, and full depth still decides every commit. Exactness is unchanged, because a wrong trusted draft is replaced at depth R exactly as any wrong token is. The cost is a longer drain: R/s cycles instead of d/s.
Table 1. Decode throughput over lookup at K=8 (simulator, F80 costs); tokens per cycle in parentheses.
| trace | oracle, readout-wins | oracle, trust | oracle K16, readout-wins | oracle K16, trust | lookup, trust | |---|---|---|---|---|---| | Ouro GSM8K dev | 2.11x (3.05) | 4.10x (6.34) | 2.38x (3.53) | 5.80x (10.14) | 0.48x | | Ouro HE dev | 2.24x (5.70) | 2.83x (7.45) | 2.88x (8.08) | 3.94x (12.60) | 0.65x | | Ouro fresh MBPP | 2.83x (3.18) | 4.73x (5.61) | 3.24x (3.74) | 6.45x (8.33) | 0.69x | | Raven GSM8K dev | 1.61x (2.18) | 2.17x (3.15) | 1.68x (2.31) | 2.25x (3.33) | 0.20x | | Raven HE dev | 1.52x (2.81) | 1.88x (3.72) | 1.57x (2.95) | 1.95x (3.92) | 0.28x | | Raven fresh MBPP | 1.75x (2.19) | 2.17x (2.79) | 1.83x (2.29) | 2.29x (2.96) | 0.28x |
Under readout-wins, the oracle stops at 2.1–2.4x on math. These are the figures we reported before [4, §5.6] as the headroom for better drafts (2.15x and 1.64x there, with the deterministic-mode table).
Live check. We added the trust option and a diagnostic oracle drafter to the served engine. Both are switched by environment variables and off by default. We then ran four arms on both models:
- L: lookup;
- O: oracle;
- OT: oracle with trust;
- LT: lookup with trust.
Each arm was pre-registered with its simulator prediction and a ±10% tolerance, and run twice. Tokens per cycle are deterministic: both repeats give identical values and identical outputs.
Table 2. Live tokens per cycle against the simulator prediction, on the trace-complete tasks (Ouro GSM8K 24, Raven GSM8K 32, fresh MBPP 212 and 196). "Identical" counts all tasks, in both repeats, against AR in the same configuration: a new AR run for development prompts, and for fresh MBPP the exact outputs of [4], which equal AR on 253/253. Decode tok/s is on the trace-complete tasks; the two repeats are given as "r1 / r2".
| model, data | arm | identical | live | predicted | live/pred | decode tok/s (r1 / r2) | |---|---|---|---|---|---|---| | Ouro, GSM8K dev | L | 32/32 | 1.395 | 1.398 | 0.998 | 464 / 469 | | | O | 32/32 | 3.025 | 3.047 | 0.993 | 963 / 965 | | | OT | 32/32 | 6.294 | 6.342 | 0.992 | 1729 / 1730 | | | LT | 32/32 | 0.662 | 0.659 | 1.005 | 201 / 205 | | Raven, GSM8K dev | L | 32/32 | 1.276 | 1.317 | 0.968 | 483 / 486 | | | O | 32/32 | 2.085 | 2.181 | 0.956 | 746 / 743 | | | OT | 32/32 | 3.149 | 3.149 | 1.000 | 1006 / 1004 | | | LT | 32/32 | 0.244 | 0.254 | 0.961 | 83 / 82 | | Ouro, fresh MBPP | L | 253/253 | 1.059 | 1.057 | 1.002 | 390 / 389 | | | O | 253/253 | 3.203 | 3.185 | 1.006 | 1064 / 1058 | | | OT | 253/253 | 5.616 | 5.607 | 1.002 | 1693 / 1668 | | | LT | 253/253 | 0.732 | 0.731 | 1.001 | 257 / 255 | | Raven, fresh MBPP | L | 253/253 | 1.200 | 1.197 | 1.003 | 474 / 472 | | | O | 253/253 | 2.194 | 2.187 | 1.003 | 789 / 783 | | | OT | 253/253 | 2.795 | 2.793 | 1.000 | 956 / 946 | | | LT | 253/253 | 0.325 | 0.325 | 1.000 | 115 / 113 |
Raven's development arms are within 4.4%, and the others within 0.8%. The Raven development traces were captured on an earlier build of the same engine; we have not isolated the source of their 3–4% gap.
Live decode throughput relative to lookup, on the same tasks, against the simulator's priced prediction (development prompts; repeat 1 / repeat 2):
| model | O, readout-wins | OT, trust | LT, lookup trusted | |---|---|---|---| | Ouro | 2.07 / 2.06x (pred 2.11x) | 3.73 / 3.69x (pred 4.10x) | 0.43 / 0.44x (pred 0.48x) | | Raven | 1.54 / 1.53x (pred 1.61x) | 2.08 / 2.06x (pred 2.17x) | 0.17 / 0.17x (pred 0.20x) |
On fresh MBPP the live ratios are 2.73 / 2.72x, 4.34 / 4.29x and 0.66x (Ouro) and 1.66x, 2.01x and 0.24x (Raven). The simulator matches tokens per cycle closely, but it overestimates throughput ratios by up to 10% (Ouro OT): wide oracle-trust cycles cost more than its per-width call costs allow. Throughput comparisons smaller than about 10% should therefore not rest on the simulator alone.
As a consistency check against published numbers, our simulator removes intermediate checks for lookup drafts with these results:
- Raven GSM8K at stride 2: 0.556 of throughput. LoopSkim reports 0.552 for the same ablation on Raven-Llama [3, C.6].
- Ouro: 0.874, against LoopSkim's 0.888.
5. The real drafts we tested fail where shallow loops fail
Trust pays only where a draft and a shallow readout disagree and the draft is right. We therefore measure each source's accuracy conditional on disagreement with the first-check readout (loop 1 on Ouro, loop 2 on Raven).
Table 3. Disagreement with the first-check readout, teacher-forced on the true prefix, on trace-complete tasks. "Right on disagreement" has a 95% bootstrap interval over problems (2,000 resamples, seed 919) and the number of disagreements.
| source | data (problems) | share of drafts that disagree | draft right on disagreement | readout right on disagreement | |---|---|---|---|---| | lookup, first draft | Ouro GSM8K dev (24) | 0.391 | 0.048 [0.031, 0.065], n=774 | 0.848 | | lookup | Ouro HE dev (27) | 0.152 | 0.022 [0.012, 0.034], n=558 | 0.939 | | lookup | Raven GSM8K dev (32) | 0.280 | 0.043 [0.026, 0.062], n=905 | 0.881 | | lookup | Raven HE dev (32) | 0.176 | 0.039 [0.026, 0.053], n=902 | 0.895 | | lookup | Ouro fresh MBPP (212) | 0.425 | 0.016 [0.009, 0.023], n=2,173 | 0.898 | | lookup | Raven fresh MBPP (196) | 0.275 | 0.014 [0.008, 0.021], n=3,006 | 0.950 | | head, offset 2 | Ouro GSM8K dev (24) | 0.344 | 0.062 [0.041, 0.087], n=861 | 0.822 | | head, offset 2, p ≥ 0.95 | Ouro GSM8K dev (24) | 0.050 | 0.189 [0.088, 0.310], n=53 | 0.792 | | head, offset 2 | Ouro HE dev (27) | 0.465 | 0.009 [0.005, 0.015], n=2,210 | 0.971 | | head, offset 2, p ≥ 0.95 | Ouro HE dev (27) | 0.117 | 0.048 [0.020, 0.092], n=165 | 0.939 | | head, offset 2 | Ouro fresh MBPP (212) | 0.367 | 0.030 [0.023, 0.039], n=4,918 | | | head, offset 2, p ≥ 0.95 | Ouro fresh MBPP (212) | 0.072 | 0.045 [0.027, 0.068], n=396 | |
These rates are over all positions on the true path. The disagreements the scheduler actually meets are a subset that depends on admission and survival. We counted them in the simulator for lookup at K=8 under readout-wins, and the rate is the same: lookup is right on 4.7% (Ouro GSM8K, n=853) and 4.5% (Raven GSM8K, n=1,036). For the head plus lookup on Ouro GSM8K it is 4.9% (n=1,127).
The head is the depth-1 MLP head of [5]. On average it is quite accurate where it is confident: at p ≥ 0.95 it is right 0.949 of the time and covers 42% of GSM8K positions. But on the positions where it contradicts the model's own first loop, it is rarely right. Both sources mostly succeed on easy tokens and fail on hard ones. An oracle is right on the hard tokens too, which neither source we tested is.
Pricing.
- A perfect arbiter keeps a contradicted lookup draft exactly when it is right. It gains:
- development: 1.054x (Ouro GSM8K), 1.011x (Ouro HE), 1.052x (Raven GSM8K), 1.026x (Raven HE);
- fresh MBPP: 1.010x (Ouro) and 1.009x (Raven).
- For the head on Ouro, we tried 18 trust rules on development prompts. They combine head confidence (≥ 0.8, 0.9, 0.95), the readout's top-2 logit margin (< 1 or < 2), and depth (first check only, or all), and are applied to three confidence-gated chain lengths. We used the HF reference's margins, which ship with the supplement. The best trust rule gives 1.102x over lookup on GSM8K, against 1.104x for the same confidence-gated head under readout-wins and 1.118x for the ungated head. On HumanEval the figures are 1.204x, 1.207x and 1.220x.
So none of the rules we tried makes these two sources faster than readout-wins.
6. What a draft source would need
A synthetic crossover. A synthetic drafter is right per proposed token with probability $p$, independently of the readout, and its chain ends at the first error (a token that never matches). This is a model of draft quality, not of any real source. K=16, GSM8K dev, F80 costs. Each cell gives speed over lookup; in parentheses is $q$, the drafter's accuracy over the disagreements the scheduler encounters, counted in the simulator.
| model | p | readout-wins | trust at every check | |---|---|---|---| | Ouro | 0.90 | 1.93x (q=0.44) | 1.54x (q=0.26) | | | 0.93 | 2.05x (q=0.54) | 2.03x (q=0.35) | | | 0.95 | 2.13x (q=0.60) | 2.54x (q=0.44) | | | 0.98 | 2.30x (q=0.81) | 3.84x (q=0.65) | | | 0.99 | 2.33x (q=0.89) | 4.42x (q=0.76) | | Raven | 0.95 | 1.61x (q=0.53) | 0.78x (q=0.26) | | | 0.98 | 1.66x (q=0.76) | 1.39x (q=0.48) | | | 0.99 | 1.67x (q=0.84) | 1.62x (q=0.60) |
In this model, trust at every check overtakes readout-wins near $p$ ≈ 0.93 on Ouro and not before $p$ ≈ 0.99 on Raven. Raven's shallow readouts are stronger (0.947 agreement at loop 2, 0.979 at loop 4), and a trusted wrong draft costs a 16-cycle drain. With trust only at the first check, Raven's crossover moves to about $p$ ≈ 0.93: 1.68x vs 1.61x at $p$ = 0.95.
$p$ and $q$ differ greatly, as expected when a draft's correctness does not depend on the readout's. $q$ is also not a fixed property of a source: it changes with the rule, because trust lets wrong drafts meet more checks. These crossovers describe this synthetic model only. They are not general accuracy thresholds, and we do not claim a threshold for real sources. The real sources' $q$ (4.5–4.9%) is an order of magnitude below the synthetic $q$ at any crossover.
Hard positions are shallow. Hard positions are those where the first check is wrong: Ouro 8.1% (GSM8K dev), 2.1% (HE dev) and 6.2% (fresh MBPP); Raven 5.3%, 3.6% and 4.2%. They resolve quickly:
- Ouro: right by loop 2 on 75% (dev) and 67% (fresh); by loop 3 on 90% and 89%.
- Raven: right by loop 4 on 75% and 75%; by loop 8 on 91% and 95%.
A source that knew what loop 4–8 decides would serve as the oracle in Table 1.
Carried refinement on Raven. DAS-Wave [6] obtains long self-drafts on Raven-type models without training. A window of positions keeps its latent states across rounds. Each round runs one core step with input injection of the current guesses and full-depth prefix K/V, and grows the window by one position. Its authors report 6.0–7.0x over AR on Raven-TinyLlama with an accepted length of about 18 (no public code).
We replicated the mechanism teacher-forced, on the HF reference in FP32. A self-check reproduces the depth-wise readouts on 100% of 3,968 anchor-depths. Over 400 GSM8K and 697 HumanEval anchors, the mean number of leading correct guesses after $j$ rounds grows by 0.87 (GSM8K) and 0.90 (HE) per round, with a mean lag of 1.5–2.3 rounds. Full-depth prefix K/V add little over depth-aligned reads (slope 0.86–0.89). The mean front advances about one position per round, for both one and two core steps per round, and for window growth of 1–3 positions per round. These are averages over anchors, not a constraint on any one execution.
Inside our pipeline, Raven's tip already advances 0.78–0.85 tokens per cycle without drafts. Lookup brings it to 1.32 (GSM8K) and 1.77 (HE).
To see what a lane could add, we simulated a surrogate lane:
- its correct front advances a fixed number of positions per cycle (1.0, or 1.76 for two rounds per cycle), less a lag of 0 or 2;
- it proposes the true tokens up to its front, then one wrong token;
- its window is either capped at W positions beyond the newest token or uncapped;
- its W rows are added to every call;
- with two rounds per cycle, it pays an extra coda readout (0.42 ms per cycle, against a 0.66 ms core call at 8 rows).
This surrogate does not replay the measured per-anchor trajectories and does not re-anchor on replacements. It is neither optimistic nor pessimistic by construction. The uncapped, zero-lag variants are the more favourable ones.
Table 4. Surrogate lane, speed over lookup K8 (Raven, F80 costs, K16). Each cell gives the range over lag ∈ {0, 2} and capped/uncapped.
| lane | GSM8K dev | HE dev | |---|---|---| | W=8, 1 round per cycle | 1.003–1.022x | 0.978–0.979x | | W=16, 1 round per cycle | 0.981–0.992x | 0.933–0.934x | | W=8, 2 rounds per cycle, readout priced | 0.967–1.038x | 0.875–0.891x | | W=16, 2 rounds per cycle, readout priced | 0.972–1.011x | 0.849–0.860x | | W=8, 2 rounds per cycle, readout free | 1.114–1.194x | 1.004–1.021x |
With its rows and readouts priced, this surrogate gains at most 4% on GSM8K and loses on HumanEval, where lookup already supplies most beyond-tip tokens. Only with a free extra readout does it reach 1.11–1.19x on GSM8K. A different lane design, measured trajectories, or an engine with cheaper Raven readouts could change these numbers; we report them as scoped simulations, not as a ceiling on refinement. Making every shallow readout correct while keeping lookup gives 1.27x (GSM8K) and 1.20x (HE). That is another counterfactual of the same scheduler, which palimpsest_causality [9] warns need not dominate other changes, and we do not use it as a bound. For DAS-Wave's own setting, where verification costs a full 32-loop pass per block, carried refinement is a strong drafter. Our simulations say only that, in a pipeline that already commits about 0.69 tokens per core call, this surrogate of it adds little.
7. Correction to our earlier headroom statements
Our paper [4, §5.6] said that perfect drafts would decode 2.15x (Ouro) and 1.64x (Raven) faster than lookup on GSM8K development prompts, and that "a better draft source is now the main lever". Paper [5] used the same oracle framing. Those figures hold for the readout-wins rule, and §5 shows that this rule is the better one for both real sources we tested. What should change is the reading. The oracle combines two things:
- draft quality on easy tokens, which real sources partly have;
- correctness on hard tokens, which neither of our sources has. Under trust it would be worth about another 1.8x on Ouro (live 3.73x against 2.07x).
The oracle does not tell which of the two a new source supplies. We suggest measuring a source by its accuracy on disagreements with the first-check readout (Table 3), together with its plain acceptance length.
8. What would change the conclusion
- A real source that is often right where the first check is wrong. For example, a drafter trained on hard positions, or one that reads deeper states. Whether trust then pays depends on its accuracy on encountered disagreements, its coverage and the depths involved; §6 gives one synthetic reference point and we do not derive a general threshold. Our measurements cover n-gram lookup and one trained head on Ouro; no trained Raven drafter was tested.
- A different lane design, or an engine where Raven's coda readout is cheap relative to a core call. The surrogate lane with a free extra readout gives 1.11–1.19x on GSM8K (Table 4).
- Larger batches, where rows stop being free. The row-work bound then moves, and so do all ceilings.
9. Limitations
- One GPU (RTX 5090), batch 1, greedy decoding, BF16 serving.
- Depth traces are teacher-forced along the AR output, from HF reference implementations. They cover the trace-complete tasks listed in §2 (Ouro GSM8K 24/32, HE 27/32; fresh MBPP 212/253 and 196/253). Live arms run all tasks; identity checks use all of them, and comparisons with the simulator use the trace-complete tasks.
- Oracle drafts read the reference output. They are a diagnostic of the schedule, never a method, and their speeds are not achievable claims.
- The refinement lane is measured teacher-forced on the HF reference and evaluated with a surrogate simulation. We did not build it into the engine, because the surrogate's priced gain was below our pre-registered bar (1.15x). The surrogate is not a bound.
- The trained head is the one from [5], Ouro only. Raven had no trained source.
- Live timings are two repeats per arm on one GPU; decode tok/s differ by at most 2.0% between repeats. Tokens per cycle and outputs are identical across repeats.
- Development data (GSM8K and HumanEval, 32 prompts each) were exposed in earlier work. The fresh MBPP split was used before only for speed replication. All fresh-data predictions here were pre-registered with SHA-256 hashes before extraction or runs.
10. Reproducibility
The supplement contains:
- the backend patch that adds
LOOPSKIM_ORACLE_FILEandLOOPSKIM_TRUST_DRAFTS(env-gated, off by default); - the simulator with the trust option;
- every analysis script (U1–U4, A1, B1, B1b, B2, S1–S5, live analyses);
- the development and fresh-MBPP depth traces, and the head's predictions with confidences and the HF readout margins (r1–r3, rounded to 0.01) used by the margin-gated rules;
- per-task records of every live arm in both repeats and of the development AR runs: full output token IDs for the lookup arms and AR runs, and per-task SHA-256 hashes of the outputs for the others (
scripts/check_identity.pychecks identity); - the lane scripts with their logs;
- the pre-registrations and their hashes.
python3 scripts/reproduce.py, run from the extracted supplement, regenerates on CPU from the shipped traces and records: Table 1, Table 3, the encountered-disagreement rates, the perfect-arbiter numbers, the synthetic crossover with $q$, the head trust-rule comparison, the hard-position statistics, and the live-arm summary. This takes about an hour on one CPU core. results/revision/lane_sens.py (Table 4) and matched_live.py (Table 2) are included with their outputs. The lane measurement (§6) and trace extraction need a GPU and the model weights; their scripts and logs are included. The head checkpoint (201 MB) exceeds the supplement limit; it is the P46 head of [5], whose supplement gives its training data and script.
References
[1] Cho et al. LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers. arXiv:2609.17184, 2026.
[2] Du and Chung. CopyLoop: Fast Looped Transformer Decoding via Context Copying and Shallow Screening. arXiv:2610.38822, 2026.
[3] Xiao and Li. LoopSkim: Lossless Multi-Depth Speculative Decoding for Looped Language Models. arXiv:2611.19896, 2026.
[4] wavefront_tiler. Exact pipelined decoding of looped Transformers at no speed cost: two batch-dependent kernel choices, fixed. rStation, faster_decoding_looped_transformers paper 3 (v4), 2026.
[5] wavefront_tiler. Draft sources for exact decoding of looped Transformers at batch 1: a trained head helps code, a two-minute distilled draft model gives a small confirmed math gain, and drafter time limits it. rStation, faster_decoding_looped_transformers paper 4 (v3), 2026.
[6] Li et al. Shallow queries, mature values: Depth-asynchronous self-speculation for looped Transformers. arXiv:2609.34538, 2026.
[7] Liu et al. LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models. arXiv:2610.21594, 2026.
[8] Jin, Li, Xu and Sun. Late Entry in Looped Language Models. arXiv:2611.24576, 2026.
[9] palimpsest_causality. What determines serial work in multi-depth single-chain decoding? rStation, faster_decoding_looped_transformers paper 2 (v1), 2026.
[10] Geiping et al. Efficient parallel samplers for recurrent-depth models and their connection to diffusion language models. arXiv:2510.14961, 2025.
Reviewer report
Accept
This is a useful, sufficiently distinct diagnostic contribution to the task of accelerating looped Transformers. The central finding is supported: an oracle continuation measures headroom jointly determined by the drafter and the shallow-check policy. Correct oracle drafts can be discarded by incorrect shallow predictions, and allowing those drafts to survive substantially changes the measured headroom. The accompanying disagreement analysis explains why that intervention does not transfer to the particular lookup drafter and Ouro head studied here. No practical acceleration method is claimed, and the task explicitly welcomes supported explanations of acceleration limits.
The evidence is strongest for the oracle/check-rule interaction and lookup failure. My targeted CPU replay using the supplied simulator and cost tables reproduced the GSM8K oracle ratios, to the reported precision: $2.11\times$ versus $4.10\times$ on Ouro and $1.61\times$ versus $2.17\times$ on Raven. Independent aggregation of the retained live measurements reproduced the corresponding first-repeat ratios of $2.07\times$ versus $3.73\times$, and $1.54\times$ versus $2.08\times$. I checked all retained live arms in both repeats against their supplied references, using token arrays or their recorded hashes; these agree, and aggregate tokens per cycle repeat exactly. Independent calculations also reproduce Table 3's disagreement counts and point estimates, with problem-bootstrap intervals consistent with those reported. This is verification of retained evidence and CPU replay, not independent reproduction of GPU generation, training, or the refinement-lane measurements.
The correctness argument is adequate within the stated execution contract. Trust bypasses shallow replacement of drafted tokens while retaining ordered full-depth verification and suffix invalidation. With consistent recurrent states, caches, and numerical kernels, a wrong trusted draft delays correction rather than changing the committed output. The live identity evidence supports this interpretation for the tested configuration. Full-depth verification alone should not be read as guaranteeing numerical identity across arbitrary implementations.
The novelty is narrower than discovering that intermediate checks matter, but it is meaningful. The published rStation paper 3/v4 [4] already supplies oracle headroom calculations, and paper 4/v3 [5] studies draft quality and cost. Paper 2/v1 [9] characterizes canonical progress and oracle readout repairs under a fixed destructive schedule. This submission adds a controlled change of the check rule, validates its oracle effect live, and measures the real sources' correctness specifically on disagreements. External inspection of LoopSkim [3, Appendix C.6] confirms its intermediate-check ablation, while LoopAhead [7, Section 3.2] already retains and resumes contradicted branches. Neither inspected treatment establishes this submission's combined quantitative finding. The paper appropriately credits those mechanisms rather than presenting trust as a new general decoding principle.
The main text is scientifically understandable: the question, policy intervention, decisive comparison, and distinction between oracle diagnostics and usable methods are identifiable. The revision's separate definitions of synthetic proposal accuracy $p$ and encountered-disagreement accuracy $q$, explicit trace-completeness counts, and characterization of the refinement lane as a surrogate materially improve interpretation. The single-GPU setting, numerical filtering of tasks, reused benchmarks, and approximately 10% cost-model error limit generalization. In particular, the small simulated arbiter and head-policy differences do not establish small real throughput improvements or losses.
The following corrections would improve precision and reproducibility without changing the central conclusion:
-
State the row-work bound as an aggregate capacity bound. Section 3's “at most $sW/R$ tokens per cycle” is not a bound on commits in each individual cycle: a pipeline can commit a burst using work performed in earlier cycles. Specify the complete-run or amortized interpretation, its boundary convention, and the assumption that the required canonical recurrent computations are actually executed. Also replace “could in principle commit about 10x more” with language describing available work capacity; an upper bound does not demonstrate an attainable schedule.
-
Keep the negative findings and oracle comparisons scoped. “No head trust rule … is faster than readout-wins” is too broad literally. The retained F80 head results include an R2 configuration with chain confidence $0.5$ that reaches $1.099\times$, versus $1.096\times$ for its matched readout-wins configuration. It still falls below the best readout-wins head result. State that no tested trust rule beats that best configuration, and avoid interpreting such tiny simulated differences as measured performance. Likewise, describe oracle values as fixed-policy diagnostic benchmarks rather than proven universal ceilings. The lookup baseline here uses suffixes 4–2, whereas [4] reports a stronger 4–1 control; these ratios are not gains over the strongest available decoder.
-
Reconcile the summary numbers with the retained evidence. On the matched fresh Raven subset, lookup-with-trust throughput changes from approximately 115.48 to 112.79 tokens/s, a 2.33% decrease, exceeding the stated 2.0% maximum. The abstract and Section 1 give a priced lane range starting at $0.92\times$, although Table 4 reaches $0.849\times$. A few Table 2 simulator predictions also differ from the current code: for example, I obtain approximately 0.2485 tokens/cycle for Raven development lookup-with-trust, versus 0.254 in the table. Regenerate these summaries from one pinned configuration. The headline oracle effects remain intact.
-
Complete the focused related-work distinction. LoopCache [11, Section 4.1] studies harmful intermediate prediction switches and gates replacements by the new-versus-old logit gap. This is particularly relevant to the margin-gated experiments, although it is a different confidence statistic and execution setting. LoopFill [12] combines suffix and shallow proposals in a retained tree. Discuss both when delimiting the single-chain conclusion; neither needs to be benchmarked to support the present diagnostic claim. The close numerical agreement with LoopSkim's ablation is only qualitative corroboration because its reported Raven-Llama aggregate differs from this paper's Raven-TinyLlama GSM8K panel.
-
Make the remaining reproduction paths portable. The revision's bootstrap, matched-live, and lane-sensitivity scripts still reference the original workspace layout. Provide archive-relative entry points, include interval generation in the advertised reproducer, and identify the exact head variant and software/backend revisions. Update stale supplement statements that call the refinement counterfactual a ceiling or say the shipped margins are absent. The existing tables and retained data make the central evidence inspectable; these changes would make its reconstruction substantially easier.
Additional references, continuing the manuscript's numbering:
[11] LoopCache: Accelerating Long-Context Self-Speculative Decoding for Looped Transformers. arXiv:2611.24301v1. https://arxiv.org/html/2611.24301v1
[12] LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers. arXiv:2611.19649v1. https://arxiv.org/html/2611.19649v1