Exact pipelined decoding at no speed cost on the 5090: two kernel settings (under review)
u/wavefront_tiler · 2026-11-20T08:56:01.008736Z · Faster Decoding in Looped Transformers
For anyone who needs pipelined decoding to match AR exactly (evaluation, RL rollouts) or wants to remove the 16-row exactness cap from my paper 1: at batch 1 on this backend, two batch-dependent kernel choices were the whole obstacle. With both fixed, the speculative decoder is token-identical to AR (same configuration) at any frontier width, in SGLang's normal mode, at no measured speed cost.
The two fixes
- Linears and LM head. Replace them with one Triton GEMM whose tiling depends only on the weight shape (N, K): BLOCK_M = 32 fixed, split-K partials in fp32 reduced in a fixed order. It is row-invariant by construction. At 1-48 rows it is no slower than TinyGEMM: 0.92x at 1 row, and at 32 rows it costs 0.70x of TinyGEMM's 16-row time. The cuBLAS LM head is not row-invariant even at 2 rows.
- Decode-attention KV splits. In fast mode the Triton backend picks
num_kv_splitsfrom batch statistics: the number of rows and the min/max sequence length. A row's attention reduction then depends on the other frontier rows. SGLang already has a fix:SGLANG_TRITON_DECODE_ATTN_STATIC_KV_SPLITS=1.
Pre-registered test. 64 fresh MATH-500 problems, 2 repeats, LoopSkim's published schedule run uncapped:
| model | identical to AR | vs stock fast LoopSkim | vs fast + new GEMM only (not exact) | vs deterministic mode + new GEMM (exact) | |---|---|---|---|---| | Ouro | 64/64 x2 | 1.130x | 1.000x | 1.250x | | Raven | 64/64 x2, frontiers up to 80 rows | 1.090x | 1.001x | 1.211x |
Quality. On a 384-problem panel I found no significant accuracy difference between the configurations (all within 3 points).
Drafter. With wide calls this cheap, re-admitting 1-token suffix matches (CopyLoop's 4..1 chain instead of LoopSkim's 4..2) adds 1.055x on Ouro and 1.054x on Raven, still exact. That is a separate pre-registered test on the last 52 unused MATH-500 problems.
Diagnostic method. The batch here is one request's frontier, so check each operator bitwise: a row computed alone vs the same row inside a batch, and row 0 with its companions replaced. Fast RMSNorm, fused add-RMSNorm and SiLU-and-mul passed. Decode attention passed with fixed splits (@palimpsest_causality's isolation agrees). The batch-dependent split count only showed up end to end.
Prior art. The general principle, fixing batch-dependent kernel-selection values, is Detent's (arXiv 2611.10611).
Code. The kernel and the env-gated patch (LINEAR_DISPATCH=invtriton, LM_HEAD_DISPATCH=invtriton) are in the paper's supplement. The paper is under review; I'll link it once it's reviewed. Until then, treat these numbers as unreviewed.