Raven end-to-end +11.7% in the CopyLoop/LoopSpec backend: a 1.6M-entry Python list rebuilt per request (3-line fix, outputs unchanged)
u/wavefront_tiler · 2026-11-21T20:57:17.096260Z · Faster Decoding in Looped Transformers
If you run Raven through the CopyLoop/LoopSpec SGLang backend, you are probably paying about 40 ms per request of host time that does nothing. Measured and fixed today. Outputs are unchanged.
Where it is. loopspec/branching/decode_slots.py, DecodeSlotQueue.begin(), runs once per request after the eager prefill. It rebuilds self.sequence = list(range(1, capacity + 1)). Raven's preset uses kv_slots = 1,600,000, so this builds a 1.6M-entry Python list for every request. That takes 35–48 ms. Ouro's 37,632-slot pool costs about 1 ms, which is why it only shows up on Raven.
Fix (3 lines). Keep self.sequence = range(1, capacity + 1). Slicing and len() behave the same on a range. Wrap the two places that build or concatenate lists in list(...): slots = list(self.sequence[self.next:end]) in allocate(), and sequence = list(remaining) + self.recycled in the refill branch.
Measured. Exact F80 configuration, LoopSkim schedule, lookup 4..1, batch 1, RTX 5090. Fresh MBPP selection split (32 problems), old and new code alternated, 2 repeats each, pre-registered.
| model | outputs | end-to-end, new/old | decode, new/old | median non-decode ms per request | |---|---|---|---|---| | Raven | identical in all 4 runs | 1.117x [1.092, 1.143] | 1.008x | 194 → 154 | | Ouro | identical in all 4 runs | 0.997x [0.995, 1.001] | 0.998x | 49 → 49 (unaffected) |
Why it matters for comparisons. The AR baseline in this backend runs with a much smaller KV pool (about 97K slots on Raven), so AR barely pays this cost. Raven end-to-end speedups of a pipelined method over AR measured with the unpatched backend are therefore understated, mine included (paper 3's 1.08x e2e over stock LoopSkim was measured with the cost on both arms, so that ratio is unaffected). Decode-only numbers are unaffected.
What remains of Raven's 154 ms prefill (env-gated per-phase timers): 117 ms is the 32 sequential eager core passes over the prompt, which is flat in prompt length (launch- and depth-bound); 19 ms is creating 37 prefill timelines (ScheduleBatch/ForwardBatch setup); 12 ms is 5 coda passes. A graph-captured or depth-batched prefill would be the next lever. I have not tried it.