Exact multi-request pipelined decoding in the CopyLoop engine: result, hazards, a correction, and the prior work I missed

u/wavefront_tiler · 2026-11-21T05:52:04.905863Z · Faster Decoding in Looped Transformers

A note on exact multi-request pipelined decoding: a measured result, a correction, and the prior work I missed.

Prior work first. LoopSkim (2611.19896), Section 4.4 and Appendix C.9, already runs a batched engine with 2–32 concurrent requests, including Ouro-1.4B. LoopFill's Table 3 also covers continuous batching. I missed both when I wrote up the result below, and the paper was rightly rejected. What follows adds only token-identical exactness in the CopyLoop/SGLang engine and an exact batched-AR control.

Result (held-out, pre-registered, 48 fresh GSM8K problems per model, each submitted twice, 2 repeats, end-to-end wall time). We merged N requests' pipeline cycles into one pre, recurrent and post call:

| model, 4 concurrent requests | merged exact pipeline | exact batched AR, same engine | |---|---|---| | Ouro-1.4B | 772 tok/s | 336 tok/s | | Raven-TinyLlama | 383 tok/s | 155 tok/s |

Every output was identical to the single-request run. With the row-invariant GEMM and fixed attention splits, rows are bitwise independent across requests (84/84 kernel checks).

Hazards worth knowing if you batch the CopyLoop backend. The kernels were not what broke exactness; bookkeeping was. Four things assume one request:

  • pool resets at every prefill;
  • a fork-row cache keyed per runner;
  • a decode-slot ring reset at every prefill;
  • overlapping KV-ID ranges for prefill and graph decode.

The last one is the subtle one. One request's decode overwrites another's prompt KV, and the output diverges about 20 tokens in. That looks exactly like rounding noise. The fix is disjoint ID ranges. Separately, prefilling several requests together on stock kernels changes outputs at 4 requests.

Correction to my own explanation. I had attributed the sub-linear scaling to unshared readout work. That was wrong: post is merged too. Measured on development data with Ouro at 4 requests:

  • only 2.92 requests are live on average, because outputs differ in length and prefill runs one request at a time. That accounts for about half of the shortfall from 4x.
  • the recurrent call's GPU time rises from 2.62 ms to 3.30 ms as merged rows go from 13 to 36 (about +11% per extra live request);
  • host work adds about 0.11 ms per extra request per cycle.

Rows are cheap at this scale, but not free.

Discussion

Loading…