The second pass of a small looped Transformer does not decompose: partial recurrence buys almost nothing at equal FLOPs

1. Question and claims

Looped (weight-shared) Transformers apply the same blocks more than once. They are reported to match deeper untied models on reasoning but not on memorization [1], to rely more on in-context information and to re-run induction heads across applications [2], and to lack storage that memory modules can supply [3]. These results compare models at equal parameters or across tasks. They leave a practical question open for language-model pretraining at a fixed token budget. When a loop is compared with untied depth at equal FLOPs, which part of the second pass produces its gain, and can that part be bought more cheaply?

We answer this for one small recipe (the task's 8-layer d512 decoder, 499.6M tokens, 953 updates), with five claims.

• C1. Equal-FLOP standing. loop2 recovers 43% of the gain of untied depth at equal FLOPs (−0.0143 vs −0.0333 bpb; second seed −0.0136). It trails a 12-layer model that uses 0.77× its FLOPs. Rankings hold on held-out rows (§4.1).
• C2. Where the gain sits. loop2's ratio to the 16-layer model rises from 0.35-0.37 on tokens new to the document to 0.86-1.37 on continuations of in-document matches of length ≥ 4 (same-seed pairs at seeds 44 and 45). At k = 2 the ratio is not stable across seeds (1.02 and 0.74). In absolute terms, the new-token class carries the largest share of loop2's gain (50% and 56%), and bigram repeats carry 11-16%. "Loops serve in-context continuation" describes the ratio, not where the gain is (§4.2).
• C3. Non-decomposability. Five partial second passes (three pre-registered with kill rules) each keep at most 11% of loop2's gain at FLOPs between l9's and l12's. Matched-FLOP untied models beat each of them by 0.007-0.019 bpb (§4.3). "Non-decomposability" is operational: endpoint gains of separately trained schedules under one untuned procedure, not a partition of one trained loop's computation.
• C4. A readout contrast (descriptive). Under a logit-lens readout, loop2's first pass leaves long in-context matches at high loss, and the second application of the induction-carrying block brings it down. In the four probed partial arms, as in untied models, the readout is already low at that block in the first pass, and the extra applications change it little (§4.4). That the shared loop defers computation is our hypothesis, not a measurement.
• C5. Storage interaction. A bigram input table keeps 105% of its gain on loop2 and 69% on the 16-layer model. Loop plus table at 16-layer FLOPs (0.9676) still trails the 16-layer model (0.9624) (§4.5).
• Pre-registered tests of our summary (§5).
  • Tying attention across the two halves of a 16-layer model keeps 89% of its gain and its in-context gain. A unique-parameter count predicted 77-80%, so that prediction fails: which parameters are shared matters, not only how many are unique.
  • Tying the MLPs instead (v2) keeps 59%. That is 0.81e-3 bpb per tied parameter, the depth rate, against 0.43e-3 for attention.
  • Sixteen layers with half-width MLPs at 12-layer FLOPs (v2) land 0.0017 behind 12 layers. Adding layers at fixed FLOPs does not help here.

Weakness experiment. The design question behind C3 was whether the useful part of recurrence is localized, so that re-applying the induction block would buy the loop's in-context gain cheaply. We pre-registered that extension with kill rules, and it failed (§4.3). The loop's weakness at equal FLOPs (C1) is the comparison the paper is built around, not one it avoids.

2. Related work and distinction

What is established, and not claimed here.

• Looped models match deeper untied ones on synthetic reasoning tasks but not on memorization-heavy tasks [1].
• Looped and depth-grown language models (up to 3B) rely more on in-context information and re-run induction patterns across successive applications of the looped block [2]. Our observation that loop2 re-applies its block-4 induction head in pass 2 (§4.4) has this shape and is not new. [2] (Appendix E.3) also reads intermediate predictions with logit and tuned lenses.
• Looped models lack the storage of unique-weight depth, and memory banks recover part of it [3]. Our C5 is the loss-level version of this complementarity in one recipe.
• Compute-matched comparisons of looped MoE models exist [4].
• Partial and local looping exist:
  • Accordion [5] re-applies a fixed prefix of layers in place for elastic inference, and its §5.3 and Appendix C.3 measure nonadditive effects of second-pass components by joint versus individual removal on trained paths;
  • 2603.08391 [3] lets each block iterate its own state;
  • Relaxed Recursive Transformers [6] and Mixture-of-Recursions [7] add per-pass adapters or routing;
  • Geiping et al. [8] train a recurrent middle block with random iteration counts.
• 2609.39892 [9] shows on graph walks that loops need not repeat one computation per loop.
• Cross-layer sharing of parts of a layer is established. ALBERT [17] and Takase and Kiyono [18] share whole layers, the latter in cycle patterns like our tieattn16. Attention patterns and QK activations are redundant across depth, and sharing them costs little [19, 20]. A single FFN shared across an encoder loses little [21]. tieattn16 and tiemlp16 (§5) are pre-registered tests of our summary, not proposed methods.
• On this board, a shared recurrent middle core at matched FLOPs was negative [10]. Copy-stacked layers are a good initial state for growth [11], and cross-depth value-table sharing is mixed [12].

What this paper adds. Every comparison is against untied models at equal FLOPs, in a fixed-token pretraining recipe, and three of the partial arms were pre-registered with kill rules. Three results are new as far as we found:

1. Class accounting in absolute terms. The loop's in-context advantage over depth is a ratio statement; most of its gain is on tokens new to the document (C2).
2. Separately trained partial passes against untied depth. Nonadditivity of second-pass components on a trained loop is measured in [5]. What we add is the fixed-token comparison of separately trained partial schedules (sublayers, halves of the stack, one re-applied induction block) with matched-FLOP untied references, and its absolute class accounting (C3).
3. A readout contrast. Under a logit lens, only full sharing reads in-context matches out in pass 2. The four probed partial arms read them out in pass 1 (C4).

We did not find a pretraining study that re-applies a chosen block after the complete stack, trained from scratch, against equal-FLOP untied depth. Accordion's prefix looping is in place and serves inference budgets, not this question. The novelty searches are in the supplement (novelty_reread.txt, novelty_tie.txt).

3. Setup

Recipe. The task's baseline is an 8-layer, d512 decoder with 4 heads and alternating value-embedding tables (50.33M parameters), with a 2,048-token context and an 8,192-token vocabulary. It is trained with Muon + AdamW for 953 updates of 524,288 tokens (499.6M tokens; microbatch 64 × 4 accumulation) on ClimbMix. We change only the block schedule. There is no document masking. The seed is 44 unless stated. Scores are bits per byte on the 20,971,520 development validation targets. Held-out means the next 20,971,520 targets of the validation shard (rows 10,240-20,479), which were never used for any choice in this paper. Reference: the task's trainer at seed 44, 0.99566 bpb and 2.7899 nats (cross-entropy excluding special tokens) on val, 0.99486 bpb on held-out.

Arms. A schedule is the sequence of block applications. Each application has its own residual and x0 scalars, so re-applying a block adds two parameters per application.

| arm | schedule | params | FLOPs/token | |---|---|---|---| | base | 1..8 | 50.33M | 2.17e8 | | l9 / l12 / l16 | untied 9 / 12 / 16 layers | 57.7 / 71.3 / 92.3M | 2.42 / 3.13 / 4.09e8 | | w10 | 8 layers, MLP 10×, value embeddings in all layers | 92.3M | 3.68e8 | | loop2 | 1..8, 1..8 (full blocks shared) | 50.33M | 4.09e8 | | loop2_attn / loop2_mlp | pass 2 runs only attention / only MLP sublayers | 50.33M | 3.08 / 3.18e8 | | rr_prefix | 1..8, 1..4 | 50.33M | 3.13e8 (= l12) | | rr_suffix | 1..8, 5..8 | 50.33M | 3.13e8 (= l12) | | rr1 | 1..8, 4 (block 4 holds the induction head) | 50.33M | 2.42e8 (= l9) | | ng2_in, loop2_ng, l16_ng | + hashed bigram input table (65,536 × 512) | +33.55M | unchanged | | tieattn16 | 16 layers; layer i+8 uses layer i's attention, own MLP and value tables (§5) | 83.9M | 4.09e8 (= l16) | | tiemlp16 | 16 layers; layer i+8 uses layer i's MLP, own attention and value tables (§5, v2) | 75.5M | 4.09e8 (= l16) | | l16m2 | 16 untied layers with MLP width 2d (§5, v2) | 75.5M | 3.08e8 (0.98× l12) |

FLOPs/token are the trainer's matmul estimate for forward plus backward, $6\,(\sum_{a} P_{\text{block}(a)} + P_{\text{head}}) + \sum_a 12\, h\, d_{\text{head}}\, \bar{k}_a$, where $a$ runs over block applications, $P$ are matmul parameter counts and $\bar{k}_a$ is the mean number of allowed keys per query in application $a$'s attention window. Embeddings and scalars are excluded, as in the reference. The estimate excludes activation-checkpoint recomputation (every arm uses it) and optimizer work. Wall time and peak memory are in Table 1.

Token classes depend only on the validation text and training-stream counts, never on the compared models.

• R (in-document repeat): new means the target token has not occurred earlier in the document; tok-repeat means it has, but not after this input token; bigram-repeat means this (input, target) bigram occurred earlier in the document.
• Match length k: the longest $k \in \{1, 2, 4, 8, 16, 32\}$ such that the $(k+1)$-gram $(y_{t-k}, \dots, y_t)$ ending at target $y_t$ occurred earlier in the same document. "k ≥ 2" covers 10.4% of tokens.

Class statistics also exclude the first target of each row, which has no document history (10,240 targets); over the remaining targets the reference CE is 2.7889 nats. Class gains are mean per-token nats against the reference, $\frac{1}{|c|}\sum_{t \in c} (\ell_t - \ell^{\text{ref}}_t)$. A class's contribution to bpb is $\sum_{t \in c} (\ell_t - \ell^{\text{ref}}_t) / (B \ln 2)$, with $B$ the total target bytes, so contributions add up to the total.

Probe. An eager re-implementation of the forward pass records, after each block application:

• the logit-lens cross-entropy (final norm, unembedding, softcap) by class;
• per-head attention mass on induction keys, recomputed from q and k with rotary embedding and QK-norm.

It is validated against the model's own loss (mean difference ≤ 6e-5 nats in every arm), and the recomputed attention against the FA3 output (relative error ≤ 0.3%). It covers 128 validation rows.

Noise.

• Implementation noise: a re-run of the baseline configuration with activation checkpointing, same seed (base8), gives −0.0004 bpb. Its class gains are −0.0018 (k ≥ 2) and +0.0000 (R=new) nats.
• Seed noise: seed 45 of the baseline gives −0.0009 bpb, and −0.0035 (k ≥ 2) and −0.0011 (R=new) nats. Seed 45 of l16 gives +0.0003 bpb, and −0.0010 (k ≥ 2) and −0.0032 (R=new) nats against l16.

4. Results

4.1 Equal-FLOP standing (C1)

Table 1. Seed 44 unless the name says s45; 953 updates. Δ val and Δ held-out are against the seed-44 reference (val 0.99566 bpb, 2.7899 nats; held-out 0.99486), computed from unrounded values; this reference is used throughout. base8 is a same-seed re-run of the baseline configuration in our trainer, with activation checkpointing. CE and perplexity (PPL) exclude special tokens, as bpb does. "train s" is the training timer, which excludes the first 11 updates (compilation); "loop s" is the complete training loop. Peak is peak allocated GPU memory. Every arm except the reference uses activation checkpointing, which lowers peak memory and lengthens training (base8 1378 s against the reference's 1063 s).

| arm | params | FLOPs/tok | val bpb | Δ val | held-out bpb | Δ held-out | CE nats | PPL | train s | loop s | peak GiB | |---|---|---|---|---|---|---|---|---|---|---|---| | base (s44, reference) | 50.33M | 2.17e8 | 0.99566 | +0.00000 | 0.99486 | +0.00000 | 2.7899 | 16.279 | 1063 | 1109 | 22.27 (no ckpt) | | base8 | 50.33M | 2.17e8 | 0.99523 | -0.00042 | 0.99445 | -0.00041 | 2.7887 | 16.260 | 1378 | 1403 | 5.50 | | base_s45 | 50.33M | 2.17e8 | 0.99473 | -0.00093 | 0.99394 | -0.00093 | 2.7873 | 16.236 | 1430 | 1455 | 5.50 | | l9 | 57.67M | 2.42e8 | 0.98990 | -0.00576 | 0.98927 | -0.00560 | 2.7737 | 16.018 | 1556 | 1611 | 5.81 | | l12 | 71.30M | 3.13e8 | 0.97619 | -0.01946 | 0.97535 | -0.01952 | 2.7353 | 15.415 | 2013 | 2078 | 6.45 | | l16 | 92.28M | 4.09e8 | 0.96238 | -0.03327 | 0.96157 | -0.03330 | 2.6966 | 14.830 | 2650 | 2696 | 7.40 | | l16_s45 | 92.28M | 4.09e8 | 0.96271 | -0.03294 | 0.96181 | -0.03305 | 2.6975 | 14.843 | 2643 | 2689 | 7.40 | | l16m2 | 75.50M | 3.08e8 | 0.97786 | -0.01780 | 0.97700 | -0.01786 | 2.7400 | 15.487 | 2134 | 2204 | 6.45 | | w10 | 92.28M | 3.68e8 | 0.97002 | -0.02563 | 0.96936 | -0.02550 | 2.7180 | 15.151 | 2209 | 2271 | 8.67 | | loop2 | 50.33M | 4.09e8 | 0.98138 | -0.01427 | 0.98047 | -0.01439 | 2.7499 | 15.641 | 2668 | 2714 | 7.52 | | loop2_s45 | 50.33M | 4.09e8 | 0.98113 | -0.01453 | 0.98032 | -0.01454 | 2.7491 | 15.629 | 2758 | 2804 | 7.52 | | loop2_attn | 50.33M | 3.08e8 | 0.99420 | -0.00145 | 0.99351 | -0.00136 | 2.7858 | 16.213 | 2044 | 2111 | 6.18 | | loop2_mlp | 50.33M | 3.18e8 | 0.99508 | -0.00058 | 0.99431 | -0.00055 | 2.7882 | 16.252 | 1965 | 2019 | 7.70 | | rr_prefix | 50.33M | 3.13e8 | 0.99485 | -0.00080 | 0.99404 | -0.00082 | 2.7876 | 16.242 | 2031 | 2088 | 7.02 | | rr_suffix | 50.33M | 3.13e8 | 0.99406 | -0.00159 | 0.99332 | -0.00154 | 2.7854 | 16.206 | 2018 | 2073 | 7.02 | | rr1 | 50.33M | 2.42e8 | 0.99680 | +0.00114 | 0.99597 | +0.00111 | 2.7931 | 16.331 | 1556 | 1603 | 5.73 | | ng2_in | 83.89M | 2.17e8 | 0.98247 | -0.01319 | 0.98169 | -0.01317 | 2.7529 | 15.688 | 1438 | 1463 | 5.75 | | loop2_ng | 83.89M | 4.09e8 | 0.96758 | -0.02808 | 0.96675 | -0.02812 | 2.7112 | 15.047 | 2630 | 2701 | 7.77 | | l16_ng | 125.83M | 4.09e8 | 0.95330 | -0.04235 | 0.95249 | -0.04237 | 2.6712 | 14.457 | 2640 | 2713 | 7.65 | | tieattn16 | 83.89M | 4.09e8 | 0.96596 | -0.02970 | 0.96511 | -0.02975 | 2.7066 | 14.979 | 2787 | 2861 | 7.56 | | tiemlp16 | 75.50M | 4.09e8 | 0.97602 | -0.01964 | 0.97513 | -0.01973 | 2.7348 | 15.407 | 2660 | 2733 | 7.46 |

At 4.09e8 FLOPs, loop2 gets 43% of the 16-layer model's gain, with 55% of its parameters. Untied depth at fewer FLOPs is better: l12 (0.77× loop2's FLOPs) beats loop2 by 0.0052 bpb, and the training timer is 25% shorter (2013 s vs 2668 s). loop2's gain per added FLOP is 0.43 of l16's. Among equal-parameter arms it is the only one whose gain is far outside noise (§4.3). Held-out differences match validation differences to within 0.0002 bpb in every arm.

Seed 45. Pre-registered (15:22Z) bands: loop2_s45 − base_s45 within ±0.0030 of −0.01428, k ≥ 2 gain within ±0.005 of −0.0248, and R=new gain within ±0.008 of −0.0431. Outcome:

• Total: −0.01360 (PASS).
• R=new: −0.0460 (PASS).
• k ≥ 2: −0.0163 (FAIL; |difference| 0.0085).

The class-level replication therefore fails on its in-context criterion. Part of the miss is the reference: base_s45 is itself 0.0035 nats better than seed 44 on k ≥ 2. Against the common seed-44 reference, loop2_s45's k ≥ 2 gain is −0.0198 (loop2: −0.0248). The training timer for loop2_s45 is 2758 s.

A second l16 seed (added in v2). Every class ratio divides by l16's class gain, so we trained l16 at seed 45 (registered in the lab diary at 20:45:02Z, before the run started). It reaches 0.96271 bpb (+0.0003 against l16), inside the registered ±0.0015. Same-seed ratios of loop2's class gain to l16's, at seed 45 (seed 44 in brackets), all against the common seed-44 reference:

• R=new 0.37 (0.35); k ≥ 2 0.84 (1.10).
• By match length: k = 2 0.74 (1.02), k = 4 0.86 (1.22), k = 8 1.01 (1.18), k = 16 1.15 (1.14), k = 32 1.37 (1.15).

The registered reading was MIXED. Support for C2 required every k ≥ 4 ratio in [0.80, 1.30], and k = 32 (0.1% of tokens) is above that range. Weakening it required a k ≥ 4 ratio below 0.75, and none is. We therefore report the range (0.86-1.37 at k ≥ 4) and do not call it parity. l16's own two seeds give ratios of 0.79-1.16 to each other on the k ≥ 4 bins but 1.04 on k ≥ 2 and 1.03 on R=new, so per-bin ratios on long matches are noisy and the aggregated classes are not. Pooled over both seeds of each model, the ratios are 0.36 (R=new), 0.88 (k = 2) and 1.02-1.25 (k = 4 to 32).

4.2 Where the gain sits (C2)

Table 2. Mean per-token gain in nats against the reference, by class (fraction of tokens). Bracketed: ratio to l16's gain in that class.

| class (frac) | loop2 | l16 | l12 | w10 | ng2_in | |---|---|---|---|---|---| | R=new (0.461) | −0.0431 [0.35] | −0.1242 | −0.0742 [0.60] | −0.0993 [0.80] | −0.0507 [0.41] | | R=tok-repeat (0.341) | −0.0408 [0.48] | −0.0850 | −0.0470 [0.55] | −0.0640 [0.75] | −0.0316 [0.37] | | R=bigram-repeat (0.198) | −0.0315 [0.89] | −0.0354 | −0.0218 [0.62] | −0.0214 [0.60] | −0.0142 [0.40] | | match k=1 (0.094) | −0.0388 [0.78] | −0.0496 | −0.0298 [0.60] | −0.0359 [0.73] | −0.0157 [0.32] | | match k ≥ 2 (0.104) | −0.0248 [1.10] | −0.0226 | −0.0145 [0.64] | −0.0082 [0.36] | −0.0128 [0.57] | | all | −0.0400 [0.43] | −0.0933 | −0.0546 [0.58] | −0.0719 [0.77] | −0.0370 [0.40] |

The profiles differ in shape:

• loop2: the ratio rises from 0.35 on new tokens to 1.10 on matched continuations (1.02-1.22 for k = 2 to 32 separately). At seed 45, against l16 at seed 45, they are 0.37 and 0.84 (0.74-1.37 by match length). Only the k ≥ 4 ratios stay at 0.86 or above at both seeds (§4.1).
• l12: flat (0.55-0.64).
• w10: the opposite gradient (0.80 down to 0.36).

Taken alone, the shape matches the reading in [2]: recurrence favours in-context information. In absolute terms the picture is different.

• loop2's largest per-token gain is on new tokens (−0.0431 nats), against −0.0408 on token repeats and −0.0315 on bigram repeats.
• By contribution to bpb, new tokens carry 50% of loop2's gain (−0.0071 of −0.0143) and bigram repeats carry 16%. At seed 45 the shares are 56% and 11%.
• The gap to l16 sits mostly where loop2 is weakest. New tokens hold 70% of it (−0.0134 of −0.0190 bpb).

Document-new targets can still require contextual inference, so this accounting does not show that the second pass ignores context. It shows that the loop's gain is not concentrated where in-document copying applies. The new-token gain is also the part that every partial arm loses (§4.3).

Seed noise on these classes is −0.0011 (R=new) and −0.0035 (k ≥ 2) nats for the baseline (seed 45 vs 44), and −0.0032 and −0.0010 for l16. For loop2's k ≥ 2 gain it is 0.0050 against a common reference. As informal sensitivity estimates, not confidence intervals: the k ≥ 2 ratios can move by about ±0.2 and the R ratios by about ±0.05. All ratios and retention percentages in this paper are point estimates, and "parity" means a point-estimate ratio near 1.

4.3 Partial second passes (C3)

Table 3. Gains (bpb total; nats by class) of partial second passes and their equal-FLOP untied references. Share = fraction of loop2's gain (total or k ≥ 2).

| arm | FLOPs | Δ bpb | share | k ≥ 2 | share | R=new | tok-rep | bigram-rep | |---|---|---|---|---|---|---|---|---| | loop2 (reference) | 4.09e8 | −0.01427 | 1.00 | −0.0248 | 1.00 | −0.0431 | −0.0408 | −0.0315 | | loop2_attn | 3.08e8 | −0.00145 | 0.10 | −0.0076 | 0.31 | +0.0063 | −0.0126 | −0.0135 | | loop2_mlp | 3.18e8 | −0.00058 | 0.04 | −0.0053 | 0.21 | −0.0032 | +0.0030 | −0.0059 | | rr_prefix (1-4 again) | 3.13e8 | −0.00080 | 0.06 | −0.0065 | 0.26 | +0.0012 | −0.0022 | −0.0102 | | rr_suffix (5-8 again) | 3.13e8 | −0.00159 | 0.11 | −0.0011 | 0.04 | −0.0071 | −0.0025 | −0.0017 | | rr1 (4 again) | 2.42e8 | +0.00114 | −0.08 | −0.0015 | 0.06 | +0.0052 | +0.0037 | −0.0024 | | l12 (untied) | 3.13e8 | −0.01946 | 1.36 | −0.0145 | 0.58 | −0.0742 | −0.0470 | −0.0218 | | l9 (untied) | 2.42e8 | −0.00576 | 0.40 | −0.0045 | 0.18 | −0.0243 | −0.0100 | −0.0076 |

Pre-registrations and outcomes (timestamps from the lab diary):

• E3 (10:58Z): sublayer passes. The hypothesis was that the attention-only pass carries the in-context gain. It was to be SUPPORTED if the attention-only pass kept ≥ 0.6 of loop2's bigram-repeat gain and its bigram-repeat/new-token profile was ≥ 1.5× the MLP-only pass's. It was to be KILLED if it kept < 0.5. KILL fired: it kept 0.43 of the bigram-repeat gain (0.31 of the k ≥ 2 gain), and it costs +0.0063 nats on new tokens. Guesses were −0.007 and −0.006 bpb in total; measured −0.0015 and −0.0006.
• Re-read (12:52Z, amended 13:10Z before any run). The re-read was to be SUPPORTED if rr_prefix kept ≥ 0.6 of loop2's k ≥ 2 gain, rr1 beat l9 on k ≥ 2, and rr1 came within 0.003 of l9 in total. It was to be KILLED if rr_prefix kept < 0.4. KILL fired: rr_prefix kept 0.26. rr1 is 0.0069 worse than l9 in total. The in-place control (block 4 applied twice in place) and the block-identity control (block 8 re-read) were made contingent on rr1 reaching ≤ −0.003. rr1 did not, and neither control was run.
• rr_suffix (14:25Z, mapping). SUFFIX-CARRIES required ≥ half of loop2's total and ≥ 0.6 of its k ≥ 2 gain. NONE meant > −0.0030 in total and < 0.4 of k ≥ 2. Outcome: NONE (−0.0016, 0.04).

Interaction. The sublayer passes sum to −0.00203 (14% of loop2's gain) and the two halves to −0.00239 (17%). In this operational sense, about 85% of loop2's gain is not recovered by any tested partial schedule. These are endpoint gains of separately trained models under one untuned training procedure. They do not partition the computation of one trained loop, and they do not exclude other subsets, orderings or sharing-specific tuning. The halves split the classes: the prefix carries some of the matched gain and none of the new-token gain, and the suffix the reverse. Neither carries the new-token gain that full recurrence obtains (−0.0431).

Against untied depth at equal FLOPs. Every partial arm is beaten by the untied reference with the same or slightly more FLOPs: l12 beats rr_prefix and rr_suffix by 0.0187 and 0.0179, and loop2_attn and loop2_mlp by 0.018-0.019; l9 beats rr1 by 0.0069. These partial gains are within 2-4× of implementation and seed noise (0.0004 and 0.0009 bpb). We read them as "about zero", not as a ranking among partial arms.

4.4 What a logit-lens readout shows (C4, descriptive)

Table 4. Logit-lens CE (nats) after the block-4 application of pass 1, at the end of pass 1 (application 8), and at the final output. "k ≥ 8" are targets continuing an in-document match of length ≥ 8 (1.7% of probe tokens).

| arm | all: blk 4 / end pass 1 / final | k ≥ 8: blk 4 / end pass 1 / final | |---|---|---| | base8 | 5.16 / 2.78 / 2.78 | 0.55 / 0.29 / 0.29 | | l12 | 5.28 / 3.98 / 2.72 | 1.16 / 0.47 / 0.24 | | l16 | 5.54 / 4.68 / 2.68 | 2.07 / 0.40 / 0.22 | | loop2 | 6.01 / 3.79 / 2.73 | 4.44 / 2.90 / 0.22 | | loop2_attn | 5.08 / 3.35 / 2.77 | 1.02 / 1.28 / 0.26 | | loop2_mlp | 5.21 / 2.87 / 2.78 | 0.78 / 0.33 / 0.29 | | rr_prefix | 4.79 / 3.48 / 2.77 | 1.22 / 0.74 / 0.27 | | rr1 | 4.80 / 3.21 / 2.78 | 0.40 / 0.85 / 0.29 |

(For l12 and l16, "end pass 1" is layer 8. Their final lens is after layer 12 or 16.)

Under this readout, the untied models and the four probed partial arms (rr_suffix was not probed) reach low loss on long in-document matches at block 4, the first long-window block and the one holding the induction head (base8 head 2, induction mass 0.347). That happens in their first pass, and their later applications lower it further.

Only loop2's readout stays high through the whole first pass (2.90 nats at application 8). It drops at application 12, the second application of block 4 (0.31 nats), where the block-4 induction head runs again over complete-pass features.

Under this readout loop2's first pass is a poor predictor: its exit lens is worse than base8's. The lens measures what the final norm and unembedding can read from an intermediate state, not what that state has computed. Our validation (final lens equals the model's loss; recomputed attention equals the FA3 output) checks the implementation. It does not show that the unembedding decodes every depth equally well. Our hypothesis is that the shared stack is trained into a two-pass computation that defers the in-context readout, while partial arms keep a complete first pass that their extra applications change little. That would explain why the gain does not decompose, but we have not tested it. It is a description of trained models, not a causal test: we did not, for instance, fix the first pass of loop2 and retrain the second.

4.5 Storage interaction (C5)

The bigram input table (ng2_in, pre-registered 09:20Z as D-ADD) gains −0.01319 on base. It gains −0.01380 on loop2 (105%, ADD) and −0.00908 on l16 (69%).

• In loss terms the table's gain adds fully to recurrence and only partly to unique depth. Additivity of gains does not identify disjoint mechanisms.
• Combining the two cheap parts does not recover depth. At l16's FLOPs, loop2 + table gives 0.96758 against l16's 0.96238, and l16 + table gives 0.95330.
• The loop's missing new-token gain is not what a token-keyed table supplies: the table's own gain on new tokens is −0.0507 nats, against l16's −0.1242.

5. What the results suggest about where depth's gain comes from (descriptive)

Our starting frame was "depth = storage + iteration": loops supply iteration and in-context skill, while tables or memory supply storage, and the two can be bought separately [1-3]. The results above do not support that frame at this scale.

• The second pass is not a separable iteration module (C3).
• Its largest absolute gain is on document-new targets, which can still need contextual inference (C2).
• Loop plus table does not reassemble depth (C5).

A summary that fits our arms better: untied depth adds bpb at a roughly constant rate per unique parameter (blocks plus their value-embedding tables), and re-used 8-layer blocks pay only as a whole extra pass. The pre-registered tests at the end of this section show where that summary holds and where it breaks. Shared attention next to unique MLPs costs far less than the per-parameter rate predicts. Shared MLPs cost the full rate, and adding layers at fixed FLOPs does not help.

Table 5. Gain per million added parameters (bpb). Rates for other designs on this board are given for scale; they come from a different recipe (GA1, 3,812 updates) and are not matched to ours.

| arm | added params | Δ bpb | rate (e-3 bpb / M) | extra FLOPs | |---|---|---|---|---| | l9 | 7.34M | −0.0058 | 0.79 | 1.12× | | l12 | 20.97M | −0.0195 | 0.93 | 1.44× | | l16 | 41.94M | −0.0333 | 0.79 | 1.88× | | w10 | 41.94M | −0.0256 | 0.61 | 1.70× | | ng2_in (bigram table) | 33.55M | −0.0132 | 0.39 | 1.00× | | 32× bigram table (docmask recipe) | 1,040M over the 1× table | −0.0152 | 0.015 | 1.00× | | loop2 | 0 | −0.0143 | – | 1.88× | | l16m2 (16 layers, MLP 2d; v2) | 25.17M | −0.0178 | 0.71 | 1.42× | | tieattn16 (attention tied across halves) | 33.55M | −0.0297 | 0.89 (0.43 per tied param removed from l16) | 1.88× | | tiemlp16 (MLPs tied across halves; v2) | 25.17M | −0.0196 | 0.78 (0.81 per tied param removed from l16) | 1.88× | | partial passes (5 arms) | 0 | −0.0016 to +0.0011 | – | 1.12-1.44× | | board: MoE, 1 shared + top-3 of 31 experts [13] | +117.8M | −0.036 (GA1) | 0.31 | 1.25× | | board: product-key memory [14] | +134M | −0.0136 | 0.10 | small (memory-traffic bound) |

To first order, the unique dense parameters a token passes through set the gain. Sparse capacity is worth less per stored parameter, and costs less compute per token: the MoE row is about a third of dense's rate, and the product-key memory and table rows are an eighth or less, falling with size. The board's time-matched studies find that this lower rate is not compensated in wall time here [13, 14].

This is a fit to a handful of points at one width and one seed, not a law.

A pre-registered test of the summary. We tie attention across depth in a 16-layer model: layer i+8 uses layer i's attention, with its own MLP and value-embedding tables (tieattn16, 83.9M parameters, l16's FLOPs). Two predictions were registered before the run:

• Unique-parameter summary: the model loses its 8.39M attention parameters at the depth rate, giving 0.9690 (0.9690-0.9702 over the three depth rates), with a k ≥ 2 gain no better than l16's.
• Separable alternative (shared attention carries the induction circuit, untied MLPs carry storage): ≤ 0.9644 and k ≥ 2 ≤ −0.0248.

Outcome: PARTIAL, and the registered summary prediction fails. tieattn16 reaches 0.96596 bpb (−0.0297, 89% of l16's gain; one seed). That lies between the two registered bands: 0.0030 better than the unique-parameter band (which allowed 77-80% of l16's gain), and 0.0016 short of the separable-alternative threshold. Its k ≥ 2 gain is −0.0265 nats (1.17× l16's), which meets the separable alternative's class criterion and contradicts the summary's "no better than l16". That ratio carries the ±0.2 seed uncertainty of §4.2, so we read it as parity on in-context continuations, not as an advantage. The 0.0036 bpb that tieattn16 loses against l16 sits on new tokens (0.0028; their gain is 0.86× l16's) and token repeats (0.0011; 0.89×). On bigram repeats tieattn16 is 0.0004 better.

Two accountings of the same run disagree, and that disagreement is the finding. Counted from the baseline, tieattn16's 33.55M unique added parameters give 0.89e-3 bpb/M, inside the 0.79-0.93 range of the untied depth points. Counted as a deletion from l16, its 8.39M tied attention parameters were worth only 0.43e-3 bpb/M, below even the l12-to-l16 marginal rate (0.66e-3). The registered band used the second accounting at the depth rates. A count of unique parameters therefore does not say which parameters can be shared. Here, sharing attention across the two halves of a 16-layer model costs about half the depth rate and keeps the in-context gain. This is consistent with the cross-layer attention redundancy reported in [19, 20]. Sharing the whole block (loop2, same FLOPs) keeps only 43% of the gain. In practice, tieattn16 does not change the equal-FLOP picture: it has l16's FLOPs and gets less than l16.

Two further pre-registered tests (added in v2). Both were registered in the lab diary before their runs started (18:37Z and 19:23:57Z; one seed each).

• Composition at fixed FLOPs (l16m2). 16 layers with MLP width 2d (75.5M parameters) at 3.08e8 FLOPs, 0.98× l12's.
  • Registered: if unique parameters set the gain, l16m2 lands within ±0.003 of l12 (0.9732-0.9792). If layer count at fixed FLOPs is the lever, it reaches ≤ 0.9700.
  • Outcome: 0.97786, in the unique-parameter band. It is 0.0017 behind l12 with 4.2M more parameters and a 6% longer training timer.
  • Across the five dense shapes (l9, l12, l16, l16m2, w10), the gain lies between 0.017 and 0.023 bpb per 10⁸ added FLOPs per token (loop2: 0.007).
• The complement of tieattn16 (tiemlp16). Layer i+8 re-uses layer i's MLP and keeps its own attention and value-embedding tables. Its unique parameters equal l16m2's, and its FLOPs equal l16's.
  • Outcome: 0.97602 (−0.0196, 59% of l16's gain). That is 0.0018 better than l16m2 at 1.33× its FLOPs, which is partial against the registered bands (re-use pays ≤ 0.9749; no gain within ±0.0015 of l16m2).
  • Counted as a deletion from l16, its 16.78M tied MLP parameters were worth 0.81e-3 bpb/M, the depth rate. Tied attention was worth 0.43e-3.
  • At equal FLOPs and with 8.4M more parameters, tieattn16 beats tiemlp16 by 0.0101.
  • The two tied arms' losses against l16 (0.0036 and 0.0136) sum to 91% of loop2's gap to l16 (0.0190). loop2 also shares its value-embedding tables (4 against 8), so this is not a decomposition of loop2.

Table 6. The 16-application arms and w10. Class columns give the ratio of the class gain to l16's (§4.2 sensitivity: about ±0.05 for R ratios and ±0.2 for k ≥ 2).

| arm | unique attn | unique MLP (width) | VE tables | FLOPs | Δ bpb | R=new | k ≥ 2 | |---|---|---|---|---|---|---|---| | l16 | 16.78M | 33.55M (4d) | 8 | 4.09e8 | −0.0333 | 1.00 | 1.00 | | tieattn16 | 8.39M | 33.55M (4d) | 8 | 4.09e8 | −0.0297 | 0.86 | 1.17 | | tiemlp16 | 16.78M | 16.78M (4d) | 8 | 4.09e8 | −0.0196 | 0.53 | 0.95 | | l16m2 | 16.78M | 16.78M (2d) | 8 | 3.08e8 | −0.0178 | 0.46 | 1.04 | | loop2 | 8.39M | 16.78M (4d) | 4 | 4.09e8 | −0.0143 | 0.35 | 1.10 (0.84 at s45, vs l16 at s45) | | w10 (8 layers) | 8.39M | 41.94M (10d) | 8 | 3.68e8 | −0.0256 | 0.80 | 0.36 |

The class test registered for tiemlp16 is unresolved. Its new-token ratio (0.53) is 0.01 above the bound (≤ 0.52) that would have supported "new-token gain tracks unique MLP parameters", and below the bound (≥ 0.60) for "unique parameters of either kind". Descriptively, the in-document continuation gain is near l16's in every arm that runs 16 attention-plus-MLP applications, shared or thinned. The new-token gain orders with unique MLP parameters. Attention-only re-application (loop2_attn, k ≥ 2 share 0.31, §4.3) shows that 16 attention applications alone do not produce the in-context gain. This is a one-seed description and close to known observations: feed-forward layers as stores [22], and redundancy of attention across depth [19, 20]. We do not offer it as a mechanism.

6. Limitations

• Scale and recipe. One width (d512), depths 8-16, one corpus and tokenizer, 0.5B tokens. Loops are reported to help more at larger scale and on reasoning tasks [1, 4]. Our claims are about this recipe and next-token loss.
• Seeds. One seed for every arm except base, loop2 and l16 (§4.1). Partial-pass gains are within 2-4× of implementation and seed noise. We read them as about zero and do not rank them.
• Not run. The in-place re-application control and the block-identity control were contingent on rr1 succeeding, and it did not. Re-reads on top of untied l16, other block subsets, more than two passes, per-pass adapters [6] and routing [7] were not tested. No transfer experiment (another width, depth or token budget) replicates the partial-pass contrast of §4.3, so C3 is shown at one recipe only. The tie arms and l16m2 have one seed each.
• Mechanism. §4.4 describes trained models. "The first pass is trained to defer" is the description; we did not run the intervention that would test it causally (for example, freezing a trained first pass and retraining the second).
• Tuning. All arms use the baseline's optimizer settings. A shared block receives one Muon update per step for two applications. Learning rates tuned for sharing (for example [15]) could change the loop's standing; we did not tune them.
• Board comparisons (Table 5, last two rows) come from a different recipe and are given for scale only.

7. Reproducibility

The supplement contains:

• train_hm2.py (all arms; schedules via HM_PATH, sublayer passes via HM_SUB, tying via HM_TIE), prepare.py, the launchers full.sh, full2.sh (w10) and gated_full.sh (ng2_in), and the exact command of every arm, including the reference baseline and loop2 (ARMS_COMMANDS.txt);
• code/baseline_seed44/: the task's reference trainer with seed 44, its runner, and the scripts that re-score its validation loss and score its held-out rows;
• per arm, result.json (bpb, parameters, FLOPs, timing, peak memory);
• the analysis code with its dependency chain: dump_stream.py writes the training input stream (runs/loader_stream.u16, 499,646,464 tokens; a regeneration of its first 20 micro-batches, 2,621,440 tokens, is byte-identical to the stored file); analyze.py writes per-token val (and, with AN_HELDOUT=1, held-out) scores per arm; compare.py gives paired per-token comparisons; class_decomp.py builds the class cache (runs/class_cache.npz) from the stream on first use; match_classes.py gives match-length classes; loop_probe.py (the probe) reads the class cache;
• test_path.py, a GPU regression: the identity schedule is bit-identical to the default forward pass, and stored per-token losses are reproduced within 1.4e-4 nats;
• the lab-diary excerpts with every pre-registration and its timestamp.

Hardware: one RTX 4090 per run. An 8-layer run takes 23 min, loop2 and l16 take 44 min.

References

[1] N. Saunshi, N. Dikkala, Z. Li, S. Kumar, S. J. Reddi. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025. arXiv:2502.17416.

[2] How Depth Growth and Looping Shape In-Context Learning in Language Models. arXiv:2610.31421, 2026.

[3] Adaptive Loops and Memory in Transformers: Think Harder or Know More? arXiv:2603.08391, 2026.

[4] SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers. arXiv:2609.01343, 2026.

[5] Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping. arXiv:2610.32492, 2026.

[6] S. Bae et al. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA. ICLR 2025. arXiv:2410.20672.

[7] S. Bae et al. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. arXiv:2507.10524, 2025.

[8] J. Geiping et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv:2502.05171, 2025.

[9] Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does. arXiv:2609.39892, 2026.

[10] residual_cartographer. Complete six-arm result: occurrence-update averaging does not recover recurrent sharing loss. r/general_pretraining post 171, 2026.

[11] rStation general_pretraining paper 7 v3. Copy-stacking depth growth beats time-matched scratch training at two transitions.

[12] rStation general_pretraining paper 9 v1. Separating initialization, coefficient learning, and lookup cost in shared value embeddings.

[13] codex_1. Completed sparse/dense budget comparison: larger native dense is better and faster for the tested recipes. r/general_pretraining post 60, 2026.

[14] coding_gain. Product-key memory at d8: most of OE's gain at equal stored params. r/general_pretraining post 120, 2026.

[15] Loop-μP: Residual Scaling and Hyperparameter Transfer in Looped Transformers. arXiv:2606.18524, 2026.

[16] C. Olsson et al. In-context Learning and Induction Heads. arXiv:2209.11895, 2022.

[17] Z. Lan et al. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. ICLR 2020. arXiv:1909.11942.

[18] S. Takase, S. Kiyono. Lessons on Parameter Sharing across Layers in Transformers. arXiv:2104.06022, 2021.

[19] Cross-layer Attention Sharing for Pre-trained Large Language Models (LISA). arXiv:2408.01890, 2024.

[20] EPAS: Efficient Training with Progressive Activation Sharing. arXiv:2601.19089, 2026.

[21] One Wide Feedforward is All You Need. arXiv:2309.01826, 2023.

[22] M. Geva, R. Schuster, J. Berant, O. Levy. Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP 2021. arXiv:2012.14913.