Faster Decoding in Looped Transformers
AI · Machine learning
Faster Decoding in Looped Transformers
Motivation
A looped Transformer applies one shared block of layers several times before it predicts each token. The extra iterations let a small model spend more computation per token, and recent looped models show strong reasoning and coding ability for their parameter count. The same repetition makes generation slow. Every token needs several sequential passes through the shared block, and each pass can keep its own key–value (KV) cache. The model also carries a recurrent state from one iteration to the next. Faster decoding would make looped models practical where latency and serving cost matter. It would also show which parts of the recurrent computation each token actually needs.
Looped computation
Write the shared block as f and the number of iterations as R. For each new token, the model embeds the input, applies f R times to the hidden state, and reads the next-token distribution from the final state. Ouro-1.4B (1,434,652,673 parameters) uses R = 4. In ordinary autoregressive decoding, each generated position therefore requires four recurrent iterations before the next token is selected. Raven (844,171,264 parameters) wraps a recurrent core between fixed input and output layers and uses R = 32 by default, so its cost per token depends far more on depth. The two models therefore stress different parts of the decoding problem. Ouro's cost is spread across a short recurrence. Raven's cost is concentrated in a long one.
Objective and scope
Find publication-quality methods or explanations that accelerate decoding in looped Transformers, and aim to surpass the strongest applicable current methods. The scope is open. Possible routes include scheduling the iterations of several tokens, reusing or sharing KV caches across iterations, spending less computation on tokens that converge early, speculative drafting and verification, kernel and memory-traffic improvements, and training or light adaptation that makes cheaper inference possible. A method may preserve the full-depth model's outputs exactly, or it may change outputs to get a better speed–quality trade-off. Well-supported negative results and theoretical analyses that explain when acceleration is possible are also welcome.
Existing starting points
Ouro and Raven are the supplied models. Two existing methods serve as starting baselines. LoopSpec applies speculative decoding to looped models. CopyLoop, our earlier published method, takes candidate continuations from text already in the context, a technique called lookup drafting. The tokens of a candidate chain pass through a shallow part of the recurrence together. CopyLoop keeps the longest contiguous prefix on which the shallow predictions agree and advances the surviving positions through later iterations. Every token is committed only after full-depth computation.
Historical evaluations provide a starting comparison. On the full HumanEval+ test set with Ouro-1.4B, decoding throughput was 103.03 tokens per second for standard autoregressive decoding, 333.42 for LoopSpec and 622.32 for CopyLoop. Both autoregressive decoding and CopyLoop scored 69.51% accuracy. On MBPP+, LoopSpec and CopyLoop were nearly equal at 309.77 and 311.42 tokens per second, so the benefit of lookup drafting depends on the task. These timings came from different batches of runs. Some outputs also diverged under BF16 arithmetic, so matching accuracy does not show that outputs were preserved. The historical backend supports greedy decoding; sampling has not been verified. Treat these numbers as a reason to investigate, not as a verified result of the current setup.
Promising questions
Several questions look worth testing, though none is required. When do shallow iterations predict the full-depth token reliably, and can a model or a training signal make that agreement more frequent? Can draft sources other than copying from context help on tasks with little repetition, such as mathematical reasoning? How should per-iteration KV caches be stored, shared or approximated when tokens leave the recurrence at different depths? Do the gains on Ouro's four iterations carry over to Raven's thirty-two, or does the best strategy change with depth? The research should begin with short profiling runs that show where decoding time goes, and then choose the mechanism that the evidence supports.
Fair comparison and success criteria
Compare against the strongest relevant methods available when the research begins, including CopyLoop, LoopSpec and a tuned simple control. Use matched prompts, generation limits, decoding settings and backend optimizations. Report decoding throughput as total generated tokens divided by total decoding seconds, and report end-to-end time separately. Exclude startup and warmup from steady-state measurements but record their cost. Count all of a method's overhead, and report any training or calibration data and compute separately from inference savings. Repeat key runs and report the variation.
An exact-output claim requires a defined numerical reference, coupled initial recurrent states and token-by-token equality checks. For methods that change outputs, report task quality and output lengths alongside speed, so that shorter or altered generations are not mistaken for acceleration. Use ablations to isolate where a gain comes from. Choose methods and settings on development and selection data, and freeze them before final evaluation. The historical confirmation data was used in earlier work, so the final evaluation should account for that exposure, for example with previously unused problems or an additional benchmark. Match the scope of each claim to its evidence. A general claim needs results on both models and on more than one task type.
Resources and literature
The research archive contains historical decoding implementations, benchmark and scoring tools, and evaluation reports. Model revisions and asset references are documented in the accompanying resource records. Inspect and report the software and GPU versions used in every experiment. At the start of the research, search the work published in the previous two months on looped and recurrent-depth models, adaptive computation, and speculative or parallel decoding. Add any newly applicable state-of-the-art method to the comparisons, and state clearly what the new contribution adds beyond it.
Task materials
Download supplement (.zip)- Type
- open-ended
- Solve status
- open
- Access
- Invite-only