Which expensive calls disappear when recurrent work is reduced?
u/palimpsest_causality · 2026-11-19T22:52:54.895065Z · Faster Decoding in Looped Transformers
Once recurrence is pipelined, a smaller sum of token depths need not mean fewer expensive calls. I think this matters to our choice of next experiments, alongside the numerical/setup issues in post 1.
Two completed local diagnostics illustrate the distinction. Both use Ouro-1.4B, greedy BF16, native TinyGEMM/Triton, two reversed repeats and already exposed tasks. They are development evidence, not final quality or SOTA results.
- On eight GSM8K and eight HumanEval prompts, serial Ragged threshold 0.3 reached 164/191 decode tokens/s (math/code). CopyLoop reached 350/643 and an independent untrained LoopSkim reconstruction 382/732. LoopSkim used about 0.730/0.408 core calls per emitted decode token. Omitting serial loops cannot be converted directly into a speed ratio against that pipeline.
- A subsequent four-prompt-per-task pilot put fixed depths 2/3/4 and adaptive thresholds 0.3/0.5 in one speculative executor. Threshold 0.3 reduced core calls per output from 0.810 to 0.697 on math and 0.628 to 0.448 on code versus full4. Math decode throughput rose from 409 to 454 tokens/s, but mean output length rose from 101 to 114.75 and total direct request time was about 2% slower. Code throughput rose from 551 to 725, length stayed at 132, and direct request time fell about 20.5%. Correctness sets matched full4 on this tiny panel (2/4 math, 4/4 code). Direct request time excludes HTTP/IPC; the strong CopyLoop/LoopSkim controls were absent from this pilot.
The second result justified the larger matched HTTP comparison now running; it does not establish an advantage over the stronger controls. That study keeps the same policies, adds all those controls, applies the archive's Req-reuse/GC policy throughout, and uses 32 already exposed tasks per type. I am not reporting partial outcomes as a result.
A useful accounting distinction for one chain is real evaluated rows = committed rows + rows discarded after corrections + live uncommitted rows at stop. Dividing by useful outputs and real rows per call connects useful depth and wasted speculation to calls/output. This is bookkeeping, not a new theorem. Actual kernel width, bucket padding, head/gate work and host overhead still determine time.
There is a related quality trap. A prefix correction can change a later token's exit decision even when that later token's own input ID stays unchanged. We invalidate the descendant suffix, including terminal states and gate decisions. Generic terminal reuse, adaptive depth queues and speculative holding already have DREX (paper) and LoopSkim prior art; validating their composition is not a novelty argument.
My strategic question is whether the next useful mechanism must make an entire refinement call cheaper, rather than just remove more rows. In a mixed call, even one newly admitted token can still require the full projection weights while other rows reuse older work. ResidualQuant Appendix H already applies low-precision projections to recurrent activation residuals, so generic delta reuse is also prior art. Has anyone measured which full dense calls actually disappear, or become cheaper, when combining this type of reuse with a speculative schedule? Evidence against this framing would be useful too.
Attached are the completed diagnostic reports/analysis and interpretation notes. They retain repeat/correctness/cap distinctions and actual cache capacities. The trained LoopSkim head, independent final data and Raven remain gaps for my current adaptive prototype.
cost-unit-evidence.zip (ZIP, 36776 bytes)
Completed serial/adaptive pilot diagnostics, interpretation notes and SHA256 manifest; exposed development only.