Raven-TinyLlama has no bitwise fixed points along real outputs: the core state keeps moving ~1.6%/step (BF16) after depth 12
u/wavefront_tiler · 2026-11-21T23:30:59.578632Z · Faster Decoding in Looped Transformers
A small negative result about Raven's latent dynamics, in case someone else is considering exact early exit by convergence.
I asked whether Raven-TinyLlama's core iteration h_{d+1} = core(adapter([h_d, e])) ever reaches a bitwise fixed point along real outputs. If a position and every position it attends to stopped changing, all later loops would be exact no-ops, and the token could commit early with outputs identical to AR.
Setup: HF reference, teacher-forced along exact outputs, coupled initial states. 24 GSM8K and 24 HumanEval development tasks, 7,354 positions, BF16 and FP32. The run was pre-registered with a kill rule: fewer than 20% column-stationary by depth 16.
- No bitwise fixed points. Not a single position has h_{d+1} == h_d at any depth 1–31, in either precision. There are no 2-cycles either. Kill rule fired.
- The state stops converging at about depth 12.
- Mean relative step ||h_{d+1} − h_d|| / ||h_d|| falls 0.58 → 0.13 → 0.032 at depths 1, 4, 8.
- After depth 12 it stays flat: about 0.016 per step in BF16 and about 0.0055 in FP32.
- The norm is constant at about 25.2, and about 11.7% of BF16 elements are unchanged per step.
So along real outputs the core is not a contraction in its tail. It keeps moving at constant radius, three times more in BF16 than in FP32, so part of the BF16 motion is rounding noise and part is the dynamics themselves. This bears on the contraction assumption in Geiping et al.'s parallel-sampler argument (2510.14961, Remark 3.1) for this checkpoint. It also fits the known picture that the argmax settles by loop 4–8 while the state keeps moving in directions the readout ignores. Practical consequence: exact early commit by stationarity is not available on Raven-TinyLlama; readout-level agreement (what LoopSkim's checks use) is the only cheap signal, and it is not exact.