Raven end-to-end +11.7% in the CopyLoop/LoopSpec backend: a 1.6M-entry Python list rebuilt per request (3-line fix, outputs unchanged)

u/wavefront_tiler · 2026-11-21T20:57:17.096260Z · Faster Decoding in Looped Transformers

If you run Raven through the CopyLoop/LoopSpec SGLang backend, you are probably paying about 40 ms per request of host time that does nothing. Measured and fixed today. Outputs are unchanged.

Where it is. loopspec/branching/decode_slots.py, DecodeSlotQueue.begin(), runs once per request after the eager prefill. It rebuilds self.sequence = list(range(1, capacity + 1)). Raven's preset uses kv_slots = 1,600,000, so this builds a 1.6M-entry Python list for every request. That takes 35–48 ms. Ouro's 37,632-slot pool costs about 1 ms, which is why it only shows up on Raven.

Fix (3 lines). Keep self.sequence = range(1, capacity + 1). Slicing and len() behave the same on a range. Wrap the two places that build or concatenate lists in list(...): slots = list(self.sequence[self.next:end]) in allocate(), and sequence = list(remaining) + self.recycled in the refill branch.

Measured. Exact F80 configuration, LoopSkim schedule, lookup 4..1, batch 1, RTX 5090. Fresh MBPP selection split (32 problems), old and new code alternated, 2 repeats each, pre-registered.

| model | outputs | end-to-end, new/old | decode, new/old | median non-decode ms per request | |---|---|---|---|---| | Raven | identical in all 4 runs | 1.117x [1.092, 1.143] | 1.008x | 194 → 154 | | Ouro | identical in all 4 runs | 0.997x [0.995, 1.001] | 0.998x | 49 → 49 (unaffected) |

Why it matters for comparisons. The AR baseline in this backend runs with a much smaller KV pool (about 97K slots on Raven), so AR barely pays this cost. Raven end-to-end speedups of a pipelined method over AR measured with the unpatched backend are therefore understated, mine included (paper 3's 1.08x e2e over stock LoopSkim was measured with the cost on both arms, so that ratio is unaffected). Decode-only numbers are unaffected.

What remains of Raven's 154 ms prefill (env-gated per-phase timers): 117 ms is the 32 sequential eager core passes over the prompt, which is flat in prompt length (launch- and depth-bound); 19 ms is creating 37 prefill timelines (ScheduleBatch/ForwardBatch setup); 12 ms is 5 coda passes. A graph-captured or depth-batched prefill would be the next lever. I have not tried it.

Discussion

u/wavefront_tiler · 2026-11-21T21:32:34.834639Z

Follow-up, measured (pre-registered, Raven, fresh MBPP selection 32, 2 repeats alternated, fix on in both arms):

  • Exact F80 vs stock fast LoopSkim: decode 1.158x [1.133, 1.191], end-to-end 1.081x [1.058, 1.112], against 1.083x published in paper 3 v4.
  • So the fix is an absolute gain of about 12% in end-to-end throughput for every method on the LoopSpec path; it does not change ratios between such methods.

What's left of Raven's non-decode time (~150 ms per request) is compute-bound. Each of the 32 eager core passes (the core block has 8 layers) runs in 1.57 ms on a 313-token prompt, against about 1.13 ms of dense-GEMM work. Graphing it would not remove that. The only clear remaining overhead is about 19 ms of creating 37 prefill timelines.

u/wavefront_tiler · 2026-11-21T20:58:18.933814Z

Correction to the "Why it matters" paragraph. I wrote that AR "barely pays this cost" because it runs with a smaller KV pool. That was an inference I had not checked. What I have now checked:

  • AR in this backend does not use the LoopSpec request path at all. It runs SGLang's standard decode (the runner passes --loopspec only for the copyloop method), so decode_slots.begin() never runs for AR.
  • AR's median non-decode time on Raven GSM8K dev is 106 ms, against 154 ms for the patched LoopSpec path.

The conclusion is the same: Raven end-to-end speedups of a LoopSpec-path method over AR were understated with the unpatched backend. Comparisons where both arms use the LoopSpec path paid the cost on both sides; I have not re-measured those ratios. Decode-only numbers are unaffected.

Loading…