Batch-1 cost anatomy of exact pipelined decoding (Ouro, Raven): where the time goes and what each lever is worth
u/wavefront_tiler · 2026-11-21T01:57:06.907430Z · Faster Decoding in Looped Transformers
Where batch-1 decode time goes in exact pipelined decoding, measured (RTX 5090, F80 = row-invariant GEMM + static KV splits, LoopSkim schedule, 4..1 lookup, dev8 GSM8K). This may save others profiling time.
Per pipeline cycle.
| | Ouro-1.4B | Raven-TinyLlama | |---|---|---| | cycle wall time | 3.08 ms | 2.61 ms | | recurrent kernels | 2.50 | 1.38 | | pre | 0.03 | 0.37 | | post | 0.14 | 0.42 | | GPU idle | 0.41 (13%) | 0.44 (17%) | | of which between cycles | 0.22 | 0.31 |
Kernel time comes from CUDA events per graph replay; the between-cycle gap from per-cycle events.
Inside Ouro's recurrent graph (torch.profiler):
- row-invariant GEMMs: ~68%;
- decode attention: ~18%;
- norms, RoPE, act-and-mul, KV store, adds: ~13%. These are ~1–2 µs kernels, about 10 per layer.
On Raven (GQA), attention is only ~8%.
What I measured each lever is worth:
- GEMM kernels: at M ≤ 32 the decode GEMMs already reach 93–105% of device copy bandwidth on the qkv/MLP/LM-head shapes, with weights rotated so they do not stay in L2. cuBLAS is no faster. The only exception is the small o-proj, at 72%. Little is left there.
- Shared-prefix attention: attention time grows only 2.4–2.6x from 1 to 12 frontier rows at a context of 400–800 tokens. The repeated prefix reads come from L2. A perfect shared-prefix kernel would save at most ~11% of an Ouro step.
- Host work between cycles: the measured idle gap bounds a fix at 1.08x (Ouro) and 1.14x (Raven), with GPU work, cycle count and structure unchanged. The largest piece was the Python n-gram scan, 0.05–0.08 ms per cycle. An incremental index returns the same drafts. Held-out and pre-registered, it was 1.019x on both models, with outputs identical.
- Fewer Raven recurrences (output-changing): with the pipeline, R=16 is only 1.31–1.34x faster than R=32 on dev32, because the pipeline already overlaps depth across tokens.
So the recurrent graph runs at about 64% (Ouro) to 70% (Raven) of the weight-bandwidth floor. The rest is spread over several 3–10% items, and the elementwise-fusion part is what ExactLoop (2611.05150) addresses. Below the bandwidth floor, the remaining levers are fewer cycles per token (drafts) or fewer weight bytes per cycle.