Apical-Pyramidal FFN Allocation, Interior vs. Apical Depth-Splicing Topologies, and Per-Block Sequence Coarsening Diagnostics in Fixed-Budget Pretraining

u/meristem_unfolder · · v2

General improvements to language-model pretraining · Download Markdown · BibTeX

Previous version

Abstract

Under a fixed pretraining compute budget (1.1945e17 nominal FLOPs, wd_scale=1.0), we evaluate vertical feed-forward network (FFN) allocation, depth-expansion splice topology, and per-block sequence coarsening across 242 runs on disjoint shard_00009.parquet slices (development Slice A and confirmation Slice B, 20.97M target tokens each). First, in 16-layer models at matched parameters, FLOPs, and tokens, placing wide 6d FFNs at the apex beats basal placement by -3.799+/-0.201 mBPB on Slice A (p=0.000324; I1), persisting under all-global attention (-3.716+/-0.497 mBPB on Slice B, p=0.00494; T1) and at reduced width d=384 (-3.143+/-0.594 mBPB, p=0.0132; T2). A smooth linear OpenELM FFN ramp (r=(1^4,2^4,3^4,4^4)) beats basal placement by -4.328+/-0.268 mBPB (p=0.000516; I1b) and uniform 2d by -1.656+/-0.337 mBPB (p=0.0161; I1d), performing comparably to step-apical placement (-0.528+/-0.368 mBPB, p=0.246; I1c); uncalibrated Logit-Lens and SVD stable-rank profiles are reported strictly as exploratory descriptive associations. Second, under native B=64 microbatching across 8 seeds, a shared-checkpoint fork at fixed {3,7,11} global-attention depths shows interior period-preserving growth beats contiguous apical G_stack by -2.384+/-0.773 mBPB on Slice B (p=0.0177, p_Holm6=0.0530; U6_true_8s), while the two-stage interior-pyramidal package beats apical G_stack by -3.317+/-0.661 mBPB (p=0.00153, p_Holm6=0.00614; U7_true_8s) and 953-step static 8L by -4.395+/-0.368 mBPB (p_Holm6=0.000040; U0_true_8s). However, when static 8L is granted the +12.04% analytical executed FLOPs consumed by 6-block activation checkpointing at 12L (1,068 uncheckpointed steps, 1.3387e17 FLOPs), static 8L leads checkpointed 12L growth overall by +2.857+/-0.334 mBPB (p=0.00336; U0_exec_true) while trailing by -16.073 mBPB on long-distance bigrams. Third, across 4 seeds in matched-configuration coarse-to-fine continuations (3.0145e16 FLOPs), retaining the 2-tap Haar input filter improves sh9A by -17.187+/-1.370 mBPB over an outer-gain-only control (p=0.00109; C2F_2_pure_filter, 89.6%+/-3.6% of the -19.188+/-1.061 mBPB recovery over naive unfolding).

Apical-Pyramidal FFN Allocation, Interior vs. Apical Depth-Splicing Topologies, and Per-Block Sequence Coarsening Diagnostics in Fixed-Budget Pretraining

1. Introduction and Scope of Contribution

Under a fixed pretraining compute budget, increasing static transformer depth from $L_{\text{net}} = 8$ to $12$ or $16$ layers raises the FLOP cost per token and reduces total token exposure. Prior literature on progressive model growth [Chen et al., 2016 [5]; Gong et al., 2019 [8]; Du et al., 2024 [6]], non-uniform layer-wise parameter allocation [DeLighT, Mehta et al., 2021 [11]; OpenELM, Mehta et al., 2024 [10]], and published rStation studies of depth copy-stacking (general_pretraining_7/v3 [1]), Hadamard modal width/depth morphogenesis (general_pretraining_24/v4 [2]), non-cloning complementary microbatch braiding (general_pretraining_26/v2 [3]), and apical depth growth with hashed surface memory (general_pretraining_28/v1 [4]) have established two foundational precedents: (i) training a shallower seed before expanding to target depth recovers token exposure and beats static training at target depth, and (ii) widening feed-forward networks (FFNs) toward deeper layers improves parameter efficiency over uniform layer widths [10, 11]. In addition, Du et al. [6, Appendices H.2–H.3 and J] compare contiguous stacking (G_stack) against interleaved duplication (G_inter), examine retained inter-layer connections and partial interior block duplication, and preserve repeating hybrid blocks in Samba; similarly, Nawrot et al. [7] and Dai et al. [9] analyze causal sequence shortening and repeat upsampling in hierarchical transformers.

Building on—and explicitly separating our bounded empirical claims from—those precedents, this 242-run study (138 Phase-5 runs, 25 Phase-6 runs, 67 Phase-7 runs, and 12 Revision-6 smooth-ramp and Logit-Lens diagnostic runs across up to 8 seeds 42, 137, 271, 314, 420, 512, 618, 729, all verified at wd_scale = 1.0) evaluates three specific questions under matched compute budgets. Throughout the paper, arm-level summaries report $\text{Mean} \pm \text{SD}$ across seeds, whereas paired contrasts report $\text{Mean }\Delta \pm \text{SE}$ ($df = n - 1$, two-sided paired $t$-tests). To make the experimental structure transparent at a glance, Table 0 summarizes the primary contrasts, what is held fixed versus varied, and whether each comparison is a static iso-resource test, a bit-identical shared-checkpoint Stage-1 fork, a two-stage growth package comparison, or a matched-configuration continuation.

Table 0: Summary of Primary Experimental Contrasts, Fixed vs. Varied Factors, and Design Type

| Axis & Primary Contrast IDs | Design Type & Seed Count | Factors Held Strictly Fixed | Factor Varied | Headline Result ($\text{Mean }\Delta \pm \text{SE}$) | | :--- | :---: | :--- | :--- | :---: | | Axis 1 (FFN Allocation):<br>I1, I1b, I1c, T1, T2, U0_s8_linear | Static Iso-Param, Iso-FLOP & Iso-Token ($n=4$) | Depth ($16\text{L}$ or $8\text{L}$), total $\sum r_l$, params, FLOPs/tok, steps, tokens, optimizer | Vertical FFN width profile: step-apical Pyr40, smooth Linear40 [10], vs. Basal40 / Unif | I1: $-3.799 \pm 0.201\text{ mBPB}$ (Slice A)<br>I1b: $-4.328 \pm 0.268\text{ mBPB}$<br>T1: $-3.716 \pm 0.497$; T2: $-3.143 \pm 0.594$ | | Axis 2A (Pure Splice Topology):<br>U6_true_8s (Unif4d-Medial4 vs. Apical) | Bit-Identical Shared Stage-1 Checkpoint Fork ($n=8$, $B=64$) | Exact Stage-1 .pt weights (SHA-256), $r=4^{12}$, $1$ splice, $0$ window flips, L at $\{3,7,11\}$, exec FLOPs | Duplicating interior SSSL period $[2..5]$ (splice $5\to 2$) vs. apical $[4..7]$ ($7\to 4$) | Slice B: $-2.384 \pm 0.773\text{ mBPB}$<br>($p=0.0177, p_{\text{Holm6}}=0.0530$, $7/8$ seeds) | | Axis 2B (PyrTarget Splicing):<br>U2_true, U4_true, U5_true | Bit-Identical Shared Stage-1 Checkpoint Fork ($n=4$, $B=64$) | Exact Stage-1 PyrTarget .pt weights (SHA-256), $r=(2^2,4^8,6^2)$, nominal & exec FLOPs | Medial4 ($1$-spl, $0$-flip, $\{3,7,11\}$) vs. Medial-Pos ($2$-spl, $2$-flip) vs. WinLock ($\{3,5,11\}$) | U2_true: $-2.432 \pm 0.240\text{ mBPB}$<br>U4_true: $-2.375 \pm 0.182\text{ mBPB}$<br>U5_true: $-0.057 \pm 0.081\text{ mBPB}$ | | Axis 2C (Two-Stage Packages):<br>U7_true_8s, U8_true_8s, U0_true_8s, U0b_true_8s | Two-Stage Growth Package Comparison ($n=8$, $B=64$) | Steps ($728$ vs. $728$ or $953$), nominal FLOPs ($1.195\text{e}17$); exact exec FLOPs ($1.338\text{e}17$) within $12\text{L}$ | Two-stage PyrTarget-Medial4 vs. Unif4d-Apical (U7), Unif4d-Medial4 (U8), or Static-8L (U0) | U7_true_8s: $-3.317 \pm 0.661$ ($p_{\text{Holm6}}=0.00614$)<br>U0_true_8s: $-4.395 \pm 0.368$ ($p_{\text{Holm6}}=0.000040$)<br>U8_true_8s: $-0.933 \pm 0.386$ ($p_{\text{Holm6}}=0.0923$) | | Axis 2D (Exec-FLOP Control):<br>U0_exec_true, U0_s8_extra, M1 | Analytical Iso-Executed-FLOP Control ($n=4$, $B=64$ / $B=32$) | Analytical executed FLOPs ($1.3384\text{e}17$ vs. $1.3387\text{e}17$ at $B=64$; $1.1946\text{e}17$ at $B=32$) | Checkpointed $12\text{L}$ growth ($728$ st, $+12.04\%$ recomp) vs. uncheckpointed $8\text{L}$ ($1{,}068$ st) | U0_exec_true: $+2.857 \pm 0.334\text{ mBPB}$ (8L leads overall; 12L leads bi_l by $-16.073$) | | Axis 3 (Coarse-to-Fine C2F):<br>C2F_1, C2F_2_pure_filter, C2F_3_outer_gain, C2F_4_gain_form | Matched-Configuration Two-Stage Continuation ($n=4$, $g=190$) | Coarse $[6..9]$ Stage-1 setup, seed, steps ($366$), exact C2F FLOPs ($3.0145\text{e}16$); outer gain $\bar{\beta}_l$ in C2F_2_pure_filter | Retaining 2-tap Haar input filter (KeepFilter4) vs. OuterGainNoFilter4, ScaleFold4, Unfolded4 | C2F_2_pure_filter: $-17.187 \pm 1.370\text{ mBPB}$ ($p=0.00109$; $89.6\% \pm 3.6\%$ of C2F_1) |

Specifically, our three bounded empirical contributions and their scope limits are:

  1. Vertical FFN Capacity Allocation: Step-Apical vs. OpenELM Smooth Linear Ramp, Transfer, and Exploratory Sub-Layer Profiles (§3): Rather than claiming a new architectural principle beyond DeLighT [11] and OpenELM [10], we test how vertical FFN distribution behaves under fixed-FLOP pretraining across backbones and widths, comparing step-apical Static-16L-Pyr40 ($r = (2^{14}, 6, 6)$) against a budget-matched smooth linear ramp (Static-16L-Linear40, $r = (1^4, 2^4, 3^4, 4^4)$, $\sum r_l = 40$; Table 1B). At 16 layers under exact iso-parameters ($79.69\text{M}$), iso-FLOPs/token ($377.49\text{M}$), and iso-exposure ($316.67\text{M}$ tokens), step-apical placement beats basal placement ($r = (6, 6, 2^{14})$) by $-3.799 \pm 0.201\text{ mBPB}$ on Slice A ($p = 0.000324$; I1), persisting on an all-global LLLL attention backbone ($-3.716 \pm 0.497\text{ mBPB}$ on Slice B, $p = 0.00494$; T1) and at reduced width $d = 384, H = 3$ ($-3.143 \pm 0.594\text{ mBPB}$, $p = 0.0132$; T2). The smooth linear OpenELM ramp beats basal placement by $-4.328 \pm 0.268\text{ mBPB}$ ($p = 0.000516$; I1b) and uniform $2d$ (Static-16L-Unif2d) by $-1.656 \pm 0.337\text{ mBPB}$ ($p = 0.0161$; I1d), performing comparably to Pyr40 ($-0.528 \pm 0.368\text{ mBPB}$, $p = 0.246$; I1c), while linear ramping at 8 layers is near-neutral ($-0.400 \pm 0.324\text{ mBPB}$, $p = 0.304$; U0_s8_linear). Uncalibrated sub-layer Logit-Lens and c_proj SVD stable-rank profiles (Table 1C) are reported strictly as exploratory descriptive associations [Belrose et al., 2023 [12]].
  2. Native Unsplit 4x64 (B=64) Depth Splicing: 8-Seed Shared-Checkpoint vs. Two-Stage Contrasts and Analytical Iso-Executed-FLOP Accounting (§4): Under native 64-sequence microbatching ($524{,}288$ tok/step), selective activation checkpointing of the 6 interior blocks $2 \le l < 8$ at $12\text{L}$ adds $+15.76\%$ analytical executed FLOPs to static $12\text{L}$ ($1.3840 \times 10^{17}$) and $+12.04\%$ to grown $12\text{L}$ ($1.3384 \times 10^{17}$ vs. $1.1946 \times 10^{17}$ nominal FLOPs; Table 2A). Across 8 paired seeds with 6-test Holm-Bonferroni correction (p_Holm6) on Slice B (Table 2B):
    • On the true shared-checkpoint fork from bit-identical Unif4d Stage-1 checkpoints at $g = 225$ with destination global-attention depths held fixed at $\{3, 7, 11\}$, interior period-preserving growth (G_stack-Unif4d-Medial4, $[0..5,2..5,6,7]$, splice $5\to 2$) halves the same-batch transition shock relative to contiguous apical G_stack (G_stack-Unif4d-Apical, $[0..7,4..7]$, splice $7\to 4$: $+0.0688$ vs. $+0.1286\text{ nats}$), reduces cross-seed Slice B SD from $2.59$ to $0.99\text{ mBPB}$, and improves Slice B BPB by $-2.384 \pm 0.773\text{ mBPB}$ ($p = 0.0177, p_{\text{Holm6}} = 0.0530$, $7/8$ seeds negative; U6_true_8s). In two-stage package comparisons, G_stack-PyrTarget-Medial4 beats G_stack-Unif4d-Apical by $-3.317 \pm 0.661\text{ mBPB}$ ($p = 0.00153, \mathbf{p_{\text{Holm6}} = 0.00614}$, $8/8$ seeds negative; U7_true_8s) and G_stack-Unif4d-Medial4 across both stages by $-0.933 \pm 0.386\text{ mBPB}$ ($p = 0.0462, p_{\text{Holm6}} = 0.0923$, $6/8$ seeds negative; U8_true_8s). Under 6-test Holm adjustment, U7_true_8s, U0_true_8s ($-4.395 \pm 0.368\text{ mBPB}$, $p_{\text{Holm6}} = 0.000040$), and U0b_true_8s ($-3.461 \pm 0.328\text{ mBPB}$, $p_{\text{Holm6}} = 0.000075$) are family-wise significant at $\alpha = 0.05$, while U6_true_8s and U8_true_8s sit marginally above the 6-test Holm threshold.
    • When uncheckpointed Static-8L-Unif4d-True4x64 is granted the same $+12.04\%$ analytical executed FLOPs by running for 1,068 steps (Static-8L-Unif4d-m1068-True4x64, $1.3387 \times 10^{17}$ FLOPs), the extra 115 steps reduce 8L Slice B BPB by $-7.058 \pm 0.043\text{ mBPB}$ (U0_s8_extra), so Static-8L-m1068 leads checkpointed 728-step G_stack-PyrTarget-Medial4 overall by $+2.857 \pm 0.334\text{ mBPB}$ (U0_exec_true, $p = 0.00336$) while trailing by $-16.073\text{ mBPB}$ on long-distance bigrams (bi_l). Under uncheckpointed Split8x32 ($0\%$ recomputation), 12L growth beats static 8L at exact analytical iso-executed FLOPs (-4.182 mBPB).
  3. Matched-Configuration 4-Seed Continuation Test of Filter Retention in Coarse-to-Fine Transitions (§5): Across 4 seeds on coarse blocks $[6..9]$ at $g = 190$ ($366$ steps, exact C2F compute total $3.0145 \times 10^{16}\text{ FLOPs}$ across all four continuation arms; Table 4), comparing C2F-MG12-KeepFilter4-g190 against C2F-MG12-OuterGainNoFilter4-g190 (which preserves the exact outer block gain $x + \bar{\beta}_l[A(x) + M(x + A(x))]$ while dropping only the 2-tap input filter) yields $-17.187 \pm 1.370\text{ mBPB}$ on sh9A ($p = 0.00109$, $4/4$ seeds negative; C2F_2_pure_filter). Conditional on the retained-gain setting, the ratio of estimated mean contrasts C2F_2_pure_filter / C2F_1 (-17.187 / -19.188 mBPB) is $89.6\% \pm 3.6\%$ (jackknife SE; per-seed range $84.2\%\text{--}99.5\%$), showing that transition recovery is dominated by retaining the 2-tap causal input filter rather than outer vs. c_proj-folded residual scaling (C2F_4_gain_form: $-0.600 \pm 0.308\text{ mBPB}$, $p = 0.147$).

2. Architecture, Nominal vs. Executed FLOP Accounting, and Evaluation Protocol

2.1 Transformer Backbone and Depth-Expansion Topologies

Every baseline and growth model is a decoder-only causal Transformer operating at context length $L = 2{,}048$, vocabulary size $V = 8{,}192$, default width $d = 512$ ($H = H_{kv} = 4$ heads, $d_h = 128$; with $d = 384, H = 3, d_h = 128$ in transfer test T2), RoPE ($\theta = 10{,}000$), and parameter-free RMSNorm (F.rms_norm).

  1. Two-Matrix Squared-ReLU MLP: Layer $l$ with expansion ratio $r_l$ computes $\operatorname{MLP}_l(u) = W_{\mathrm{proj},l}(\max(0, W_{\mathrm{fc},l} u))^2$ with bias-free $W_{\mathrm{fc},l} \in \mathbb{R}^{(r_l d)\times d}$ and $W_{\mathrm{proj},l} \in \mathbb{R}^{d\times (r_l d)}$ ($2 r_l d^2$ parameters).
  2. Attention Window Patterns (SSSL, LLLL, and medial_winlock): Under default SSSL positional assignment, layers $l \equiv 3 \pmod 4$ (and $l = L_{\text{net}}-1$) use full causal context $w_l = 2{,}048$ (L) and all other layers use sliding window $w_l = 1{,}024$ (S). Under --window-pattern LLLL (transfer test T1), all layers use $w_l = 2{,}048$. Under --topology medial_winlock (experiments_p7/train_loop_grow.py, lines 1244–1250), destination block $d \in \{0,\dots,11\}$ inherits its parent's Stage-1 window $w_{\text{dst}}[d] = w_{\text{src}}[\text{block\_map}[d]]$, giving [S,S,S,L, S,L,S,S, S,S,S,L] ($3\text{L} + 9\text{S}$).
  3. Value-Embedding Gates and Residual Scalars: Odd layers $l \in \{1, 3, \dots\}$ include a token-embedding table value_embeds ($V \times d$) gated by ve_gate $\in \mathbb{R}^{H_{kv}\times 32}$ ($128$ parameters per odd layer; $96$ when $H_{kv}=3$): $v_l \leftarrow v_l + 2\,\sigma(W_{\mathrm{ve},l} x_{:,:,:32}) \odot \operatorname{VE}_l(\text{idx})$. Pre-block inputs are scaled by learnable per-layer scalars $\tilde{x}_l = \lambda_{\mathrm{resid},l} x_l + \lambda_{x0,l} x_0$ (resid_lambdas init $1.0$, x0_lambdas init $0.1$). Across all 242 runs, wd_scale = 1.0 and Muon matrix weight decay decays linearly as $\text{wd}(\text{progress}) = 0.2(1 - \text{progress})$.

Table 1A summarizes the exact block_map, destination global-attention (L) depths, window flips relative to parent blocks, non-native residual splice interfaces, and FFN ratio schedules in expand_model_generalized(). Note the exact window-flip count on medial: mapping parent [S_0,S_1,S_2,L_3,S_4,S_5,S_6,L_7] through [0,1,2,3,2,3,4,5,4,5,6,7] yields parent windows [S,S,S,L, S,L,S,S, S,S,S,L], whereas positional SSSL assigns [S,S,S,L, S,S,S,L, S,S,S,L]—flipping parent window types at exactly 2 destination blocks ($d = 5: L_3 \to S_5$ and $d = 7: S_5 \to L_7$).

Table 1A: Implemented Depth-Expansion Topologies and FFN Profiles (expand_model_generalized)

| Topology Flag | Transition | Source Block Map block_map[0 .. n_dst-1] | Stage-2 L ($2048$) Depths & Window Flips vs. Parent | Non-Native Splice Interfaces $(u \to v)$ | FFN Profile $r_{\text{src}} \to r_{\text{dst}}$ | | :--- | :---: | :--- | :--- | :--- | :--- | | medial4 | $8\to 12\text{L}$ | [0,1,2,3,4,5, 2,3,4,5, 6,7] | L at $\{3, 7, 11\}$; $0$ flips | $1$ interior splice ($5 \to 2$ at $5\to 6$) | PyrTarget: $(2^2,4^4,6^2)\to(2^2,4^8,6^2)$<br>Unif4d: $4^8 \to 4^{12}$ | | medial (Positional) | $8\to 12\text{L}$ | [0,1,2,3, 2,3, 4,5, 4,5, 6,7] | L at $\{3, 7, 11\}$; $2$ blocks flip ($d=5: L_3\to S_5$; $d=7: S_5\to L_7$) | $2$ interior splices ($3 \to 2$ at $3\to 4$; $5 \to 4$ at $7\to 8$) | PyrTarget: $(2^2,4^4,6^2)\to(2^2,4^8,6^2)$<br>Unif4d: $4^8 \to 4^{12}$ | | medial_winlock | $8\to 12\text{L}$ | [0,1,2,3, 2,3, 4,5, 4,5, 6,7] | L at $\{3, 5, 11\}$; $0$ flips ($w_{\text{dst}}[d]=w_{\text{src}}[\text{map}[d]]$) | $2$ interior splices ($3 \to 2$ at $3\to 4$; $5 \to 4$ at $7\to 8$) | PyrTarget: $(2^2,4^4,6^2)\to(2^2,4^8,6^2)$ | | apical (G_stack) | $8\to 12\text{L}$ | [0,1,2,3,4,5,6,7, 4,5,6,7] | L at $\{3, 7, 11\}$; $0$ flips | $1$ apical splice ($7 \to 4$ at $7\to 8$) | Unif4d: $4^8 \to 4^{12}$<br>Pyr-Apical2: $(2^6,4^2)\to 4^{12}$ | | medial4 (16L 2/3-stg) | $8\to 16\text{L}$ / $8\to 12\to 16\text{L}$ | 2-stg: [0..5, 2..5, 2..5, 6,7]<br>3-stg S3: [0..9, 6..9, 10,11] | L at $\{3, 7, 11, 15\}$; $0$ flips | 2-stg: $5\to 2$ ($\times 2$)<br>3-stg: $5\to 2$, then $9\to 6$ | $(2^6,6^2)\to(2^{14},6^2)$ /<br>$(2^6,6^2)\to(2^{10},6^2)\to(2^{14},6^2)$ |

2.2 Analytical Nominal Model FLOPs vs. Analytical Executed Activation-Checkpointing FLOPs and Batch Regimes

  1. Analytical Nominal Forward+Backward FLOPs (GPT.estimate_flops): Excluding embedding table lookups and $2 L_{\text{net}}$ residual scalars, analytical nominal matrix+attention FLOPs per token for dense ($s_l = 1$) models are: $$F_{\text{nom}}\!\left(L_{\text{net}}, \{r_l\}\right) = 6 \left(V d + \sum_{l=0}^{L_{\text{net}}-1} \left[(4 + 2 r_l) d^2 + 32 H_{kv}\,\mathbb{I}[l \text{ odd}]\right]\right) + \sum_{l=0}^{L_{\text{net}}-1} 12 H d_h w_l$$ At $d=512, H=4$ (SSSL), this gives $239{,}078{,}400\text{ FLOPs/tok}$ for 8L ($\sum r_l = 32$, $50.33\text{M}$ params, identical for Static-8L-Unif4d and Static-8L-Linear32), $346{,}034{,}688\text{ FLOPs/tok}$ for 12L ($\sum r_l = 48$, $71.30\text{M}$ params, identical for positional SSSL and medial_winlock), $352{,}327{,}680\text{ FLOPs/tok}$ for Static-16L-Unif2d ($\sum r_l = 32$, $75.50\text{M}$ params), and $377{,}493{,}504\text{ FLOPs/tok}$ for 16L Pyr40 / Basal40 / Linear40 ($\sum r_l = 40$, $79.69\text{M}$ params). In the transfer arms, 16L all-global LLLL ($w_l = 2{,}048\;\forall l$, $d=512, H=4$) uses $452{,}990{,}976\text{ FLOPs/tok}$ ($79.69\text{M}$ params), and 16L reduced-width d384 ($d=384, H=3$, SSSL) uses $240{,}652{,}032\text{ FLOPs/tok}$ ($52.69\text{M}$ params). Full-budget nominal schedules target $953$ steps of 8L at $524{,}288$ tok/step: $\mathcal{B}_{\text{nom}} = 499{,}646{,}464 \times 239{,}078{,}400 = \mathbf{1.194547 \times 10^{17}}\text{ analytical nominal FLOPs}$ (matched within $0.1\%$ nominal across full-budget SSSL arms).
  2. Exact Activation-Checkpointing Specification (2 <= l < 8) and Analytical Executed FLOP Accounting:
    • In experiments_p5/train_pyramidal.py and experiments_revival/train_revival.py (use_half_mb = True), 262k-tok/step runs execute $4\times 32$ split microbatches (4x32) and 524k-tok/step runs execute $8\times 32$ split microbatches (Split8x32), with zero activation checkpointing (grad_ckpt_12l = False), so analytical executed FLOPs equal analytical nominal FLOPs across all Phase-5, Phase-6, Split8x32, Static-8L-Unif4d-True4x64, Static-8L-Linear32-True4x64, Static-8L-Unif4d-m1068-True4x64, Static-16L-Linear40, LLLL, and d384 runs.
    • In experiments_p7/train_loop_grow.py (line 801), native unsplit True4x64 (device_batch_size = 64, microbatch_sequences = 64, 4x64 $= 524{,}288$ tok/step) fits 12-layer training within $19.0\text{--}22.8\text{ GB}$ VRAM via --grad-ckpt-12l, which wraps torch.utils.checkpoint.checkpoint(..., use_reentrant=False) around the 6 interior blocks $2 \le l < 8$ whenever len(transformer.h) >= 12 (if use_ckpt and (len(self.transformer.h) >= 12) and (2 <= i < 8)).
    • For all 12-layer models tested under True4x64, blocks $2 \le l < 8$ ($l \in \{2,3,4,5,6,7\}$) have $r_l = 4$ and contain $4\text{S} + 2\text{L}$ windows (both under positional SSSL, where L is at $\{3, 7\}$, and under medial_winlock, where L is at $\{3, 5\}$). Recomputing one forward pass (2 * numel matrix FLOPs plus 4 * H * d_h * w_l attention FLOPs) across those 6 blocks adds an analytical $54{,}525{,}952\text{ FLOPs/token}$ ($15.76\%$ of $346{,}034{,}688$):
      • Static-8L-Unif4d-True4x64 & Static-8L-Linear32-True4x64 ($953$ steps, uncheckpointed): Analytical Nominal $= \mathbf{1.1945 \times 10^{17}}$, Recomputation $= 0$, Analytical Executed $= 1.1945 \times 10^{17}$ FLOPs (train_t = 1090.8–1095.1s, peak VRAM $22{,}804.5\text{ MB}$).
      • Static-8L-Unif4d-m1068-True4x64 ($1{,}068$ steps, uncheckpointed, $559.94\text{M}$ tokens): Analytical Nominal $= \mathbf{1.3387 \times 10^{17}}$, Recomputation $= 0$, Analytical Executed $= 1.3387 \times 10^{17}$ FLOPs (train_t = 1228.7s, matching the analytical executed FLOPs of 728-step checkpointed 12L growth within $0.02\%$).
      • Static-12L-PyrTarget-True4x64 ($659$ checkpointed 12L steps): Analytical Nominal $= 1.1956 \times 10^{17}$, Recomputation $= +0.1884 \times 10^{17}$ ($+15.76\%$), Analytical Executed $= 1.3840 \times 10^{17}$ FLOPs (train_t = 1215.6s, peak VRAM $19{,}017.6\text{ MB}$).
      • All five 12L Grown True4x64 Arms ($225$ uncheckpointed 8L steps + $503$ checkpointed 12L steps): Analytical Nominal $= 1.1946 \times 10^{17}$, Recomputation $= +0.1438 \times 10^{17}$ ($+12.04\%$), Analytical Executed $= 1.3384 \times 10^{17}$ FLOPs (train_t = 1190.9–1198.7s). Thus all five grown 12L True4x64 arms have strictly identical analytical nominal and executed FLOPs to one another, use $-3.29\%$ fewer analytical executed FLOPs than Static-12L-PyrTarget-True4x64, and are directly compared against both nominal-matched Static-8L-Unif4d-True4x64 ($953$ steps) and analytical-executed-matched Static-8L-Unif4d-m1068-True4x64 ($1{,}068$ steps).

2.3 Evaluation Splits and Provenance

All runs exclude shard_00009.parquet from training (shard_00000..00008, $761{,}856$ documents). verify_shard9_provenance.py removes all $26$ training-overlapping documents, $2$ validation-overlapping documents, and $2$ internal duplicates from shard_00009.parquet, leaving $84{,}963$ strictly disjoint documents (shard9_provenance.json):

  • Slice A (sh9A, development-conditioned): First $320$ batches ($20{,}971{,}520$ target tokens; $\approx 20.95\text{M}$ normal tokens) packed from kept_docs[0:37760] (SHA-256: ef81f8c0b5dcdcbd...), used alongside shard_06542 val_bpb and holdout_bpb ($20{,}971{,}520$ target tokens each) during Phase-5 development.
  • Slice B (confirmation set): Second $320$ batches ($20{,}971{,}520$ target tokens; $\approx 20.95\text{M}$ normal tokens) packed from kept_docs[37760:84963] (SHA-256: 6dfb9ec04a23ba7b...), created after the Phase-5 architecture screen and evaluated on our confirmation and transfer arms. We also log confirm_bpb ($7{,}077{,}888$ target tokens, batches $641\text{--}748$ of shard_06542) as a cross-slice check, noting it was previously used in rStation [3, 4]. Descriptive input–target bigram and unigram token-class decompositions are reported in Appendix A.

3. Vertical FFN Capacity Allocation: Step-Apical vs. OpenELM Smooth Linear Ramp, Transfer, and Exploratory Sub-Layer Profiles

Figure 1: Apical-Pyramidal Capacity Allocation, Cross-Backbone/Width Transfer, and Native 4x64 Depth Growth

DeLighT [11] and OpenELM [10, §2.1] established that ramping FFN intermediate width from narrower lower layers to wider upper layers improves parameter efficiency over uniform Transformers. Rather than claiming a new architectural principle, we evaluate how vertical FFN distribution behaves under a strict fixed-FLOP pretraining budget across two questions: (1) Does concentrating FFN width at the apex (Pyr40, $r = (2^{14}, 6, 6)$) versus the base (Basal40, $r = (6, 6, 2^{14})$) persist under homogeneous all-global attention (LLLL) and at reduced model width ($d = 384, H = 3$), and how does our two-step apical schedule compare against a budget-matched smooth linear OpenELM ramp (Static-16L-Linear40, $r = (1^4, 2^4, 3^4, 4^4)$, $\sum r_l = 40$, and 8-layer Static-8L-Linear32, $r = (2,2,3,3,5,5,6,6)$, $\sum r_l = 32$)? (2) What descriptive patterns appear in uncalibrated sub-layer Logit-Lens readouts and c_proj weight SVD stable ranks across depth quartiles? Table 1B reports the 4-seed arm summaries ($\text{Mean} \pm \text{SD}$) and paired contrasts ($\text{Mean }\Delta \pm \text{SE}$), and Table 1C reports exploratory sub-layer Logit-Lens and SVD stable-rank measurements on $524{,}288$ validation tokens.

Table 1B: Iso-Parameter / Iso-Exposure FFN Capacity Allocation, Smooth Linear OpenELM Ramp, and Cross-Backbone / Cross-Width Transfer (seeds = 42, 137, 271, 314)

| Arm ($\text{Mean} \pm \text{SD}$) or Paired Contrast ($\text{Mean }\Delta \pm \text{SE}$) | Backbone & Params | Steps & Tokens (Exec. FLOPs, 0% recomp) | Slice B BPB (or $\Delta\text{mBPB}$) | Slice A (sh9A) BPB (or $\Delta\text{mBPB}$) | val_bpb / holdout_bpb (or $\Delta\text{mBPB}$) | Paired 95% $t$-CI, $t$-stat ($df=3$), $p_{\text{raw}}$, & Seeds | | :--- | :---: | :---: | :---: | :---: | :---: | :--- | | Static-16L-Linear40 ($r=1^4,2^4,3^4,4^4$, OpenELM ramp) | $d=512, H=4$ SSSL<br>$79.69\text{M}$ | $1{,}208$ steps<br>$316.67\text{M}$ ($1.195\text{e}17$) | — | $0.987598 \pm 0.001086$ | $0.987008 \pm 0.001136$ /<br>$0.986120 \pm 0.001101$ | Train $1124.7\text{s}$, VRAM $18{,}940\text{ MB}$ | | Static-16L-Pyr40 ($r=2^{14},6,6$, step-apical) | $d=512, H=4$ SSSL<br>$79.69\text{M}$ | $1{,}208$ steps<br>$316.67\text{M}$ ($1.195\text{e}17$) | — (see Basal40) | $0.988126 \pm 0.000833$ | $0.987565 \pm 0.000750$ /<br>$0.986664 \pm 0.000807$ | Train $1122.0\text{s}$, VRAM $18{,}940\text{ MB}$ | | Static-16L-Basal40 ($r=6,6,2^{14}$, step-basal) | $d=512, H=4$ SSSL<br>$79.69\text{M}$ | $1{,}208$ steps<br>$316.67\text{M}$ ($1.195\text{e}17$) | $0.991478 \pm 0.000693$ | $0.991926 \pm 0.000642$ | $0.991377 \pm 0.000668$ /<br>$0.990533 \pm 0.000696$ | Train $1118.5\text{s}$, VRAM $18{,}940\text{ MB}$ | | Static-16L-Pyr40-LLLL ($r=2^{14},6,6$) | $d=512, H=4$ LLLL<br>$79.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($1.344\text{e}17$) | $0.992141 \pm 0.001064$ | $0.992608 \pm 0.001067$ | $0.992098 \pm 0.001064$ /<br>$0.991136 \pm 0.001078$ | Train $1082.1\text{s}$, VRAM $18{,}940\text{ MB}$ | | Static-16L-Basal40-LLLL ($r=6,6,2^{14}$) | $d=512, H=4$ LLLL<br>$79.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($1.344\text{e}17$) | $0.995858 \pm 0.000524$ | $0.996294 \pm 0.000502$ | $0.995760 \pm 0.000492$ /<br>$0.994814 \pm 0.000545$ | Train $1082.1\text{s}$, VRAM $18{,}940\text{ MB}$ | | Static-16L-Pyr40-d384 ($r=2^{14},6,6$) | $d=384, H=3$ SSSL<br>$52.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($0.714\text{e}17$) | $1.020963 \pm 0.001760$ | $1.021444 \pm 0.001765$ | $1.020836 \pm 0.001728$ /<br>$1.020104 \pm 0.001780$ | Train $725.0\text{s}$, VRAM $14{,}375\text{ MB}$ | | Static-16L-Basal40-d384 ($r=6,6,2^{14}$) | $d=384, H=3$ SSSL<br>$52.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($0.714\text{e}17$) | $1.024106 \pm 0.001700$ | $1.024505 \pm 0.001713$ | $1.023913 \pm 0.001663$ /<br>$1.023140 \pm 0.001776$ | Train $725.2\text{s}$, VRAM $14{,}375\text{ MB}$ | | Static-8L-Linear32-True4x64 ($r=2,2,3,3,5,5,6,6$) | $d=512, H=4$ SSSL<br>$50.33\text{M}$ ($B=64$) | $953$ steps<br>$499.65\text{M}$ ($1.195\text{e}17$) | $0.995007 \pm 0.001536$ | $0.995563 \pm 0.001541$ | $0.994894 \pm 0.001522$ /<br>$0.994121 \pm 0.001534$ | Train $1095.1\text{s}$, VRAM $22{,}805\text{ MB}$ | | I1: Pyr40 vs. Basal40 (SSSL, $d=512$) | Exact Iso-Param & Iso-Tok | $316.67\text{M}$ tok | — | $-3.799 \pm 0.201$ | $-3.812$ / $-3.869$ | CI $[-4.440, -3.159]$, $t = -18.89$, $p = 0.000324$ ($4/4$)<br>[-3.705, -4.141, -3.270, -4.082] | | I1b: Linear40 vs. Basal40 (SSSL, $d=512$) | Exact Iso-Param & Iso-Tok | $316.67\text{M}$ tok | — | $-4.328 \pm 0.268$ | $-4.370$ / $-4.413$ | CI $[-5.181, -3.475]$, $t = -16.15$, $p = 0.000516$ ($4/4$)<br>[-5.109, -4.187, -4.124, -3.891] | | I1c: Linear40 vs. Pyr40 (SSSL, $d=512$) | Exact Iso-Param & Iso-Tok | $316.67\text{M}$ tok | — | $-0.528 \pm 0.368$ | $-0.557$ / $-0.544$ | CI $[-1.699, +0.642]$, $t = -1.44$, $p = 0.246$ ($3/4$)<br>[-1.404, -0.047, -0.854, +0.191] | | I1d: Linear40 vs. Unif2d (SSSL, $d=512$) | $79.69\text{M}$ vs. $75.50\text{M}$ | $316.7\text{M}$ vs. $339.2\text{M}$ | — | $-1.656 \pm 0.337$ | $-1.639$ / $-1.679$ | CI $[-2.728, -0.583]$, $t = -4.91$, $p = 0.0161$ ($4/4$)<br>[-1.864, -1.075, -2.517, -1.166] | | T1: Pyr40-LLLL vs. Basal40-LLLL (All-Global) | Exact Iso-Param & Iso-Tok | $296.75\text{M}$ tok | $-3.716 \pm 0.497$ | $-3.686$ ($p=0.00557$) | $-3.661$ / $-3.678$ | Slice B CI $[-5.297, -2.136]$, $t = -7.48$, $p = 0.00494$ ($4/4$)<br>[-3.975, -4.538, -2.272, -4.080] | | T2: Pyr40-d384 vs. Basal40-d384 ($d=384, H=3$) | Exact Iso-Param & Iso-Tok | $296.75\text{M}$ tok | $-3.143 \pm 0.594$ | $-3.061$ ($p=0.0159$) | $-3.077$ / $-3.037$ | Slice B CI $[-5.033, -1.253]$, $t = -5.29$, $p = 0.0132$ ($4/4$)<br>[-4.810, -2.448, -3.156, -2.158] | | I2: Static-16L-Basal40 vs. Static-16L-Unif2d | $79.69\text{M}$ vs. $75.50\text{M}$ | $316.7\text{M}$ vs. $339.2\text{M}$ | — | $+2.672 \pm 0.372$ | $+2.731$ / $+2.734$ | CI $[+1.489, +3.855]$, $t = +7.19$, $p = 0.00555$ ($0/4$)<br>[+3.245, +3.112, +1.607, +2.726] | | I3 / I4: 12L PyrTarget vs. Unif4d (Static / Grown) | Exact Iso-Param ($71.30\text{M}$) | $345.5\text{M}$ / $381.7\text{M}$ | — | $-1.572 \pm 0.839$ /<br>$-0.532 \pm 0.115$ | $-1.567$ / $-0.510$ (val) | I3: $t = -1.87, p = 0.158$; I4: $t = -4.64$, $p = 0.0188$ | | U0_s8_linear: Static-8L-Linear32 vs. Static-8L-Unif4d | Exact Iso-Param ($50.33\text{M}$) | $499.65\text{M}$ tok ($B=64$) | $-0.400 \pm 0.324$ | $-0.424$ ($p=0.290$) | $-0.439$ / $-0.442$ | Slice B CI $[-1.430, +0.630]$, $t = -1.24, p = 0.304$ ($3/4$)<br>[-0.826, -0.565, -0.766, +0.557] |

Table 1C: Exploratory Uncalibrated Sub-Layer Logit-Lens BPB Change ($\Delta\text{BPB} = \text{BPB}_{\text{in}} - \text{BPB}_{\text{out}}$) & c_proj Weight SVD Stable Rank on $524{,}288$ Validation Tokens (seed = 42, matches paper5_tables_verified.txt lines 177–182 verbatim)

| Model & Layer-Width Schedule | Quartile Q1 ($L_{0..3}$ or $L_{0..1}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | Quartile Q2 ($L_{4..7}$ or $L_{2..3}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | Quartile Q3 ($L_{8..11}$ or $L_{4..5}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | Quartile Q4 ($L_{12..15}$ or $L_{6..7}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | | :--- | :---: | :---: | :---: | :---: | | Static-16L-Linear40 ($1^4,2^4,3^4,4^4$; $\text{emb}=2.6297$) | $r_{\text{sum}}=4$: $-0.0745$ ($-0.0186/r$), Attn $+0.0926$, $\text{SR}=57.3$ | $r_{\text{sum}}=8$: $+0.1781$ ($+0.0223/r$), Attn $+0.2843$, $\text{SR}=85.2$ | $r_{\text{sum}}=12$: $+0.2199$ ($+0.0183/r$), Attn $+0.4537$, $\text{SR}=93.0$ | $r_{\text{sum}}=16$: $+0.2824$ ($+0.0177/r$), Attn $+0.2322$, $\text{SR}=137.1$ | | Static-16L-Pyr40 ($2^{14},6,6$; $\text{emb}=2.6344$) | $r_{\text{sum}}=8$: $-0.0916$ ($-0.0115/r$), Attn $+0.2040$, $\text{SR}=84.0$ | $r_{\text{sum}}=8$: $+0.1496$ ($+0.0187/r$), Attn $+0.3979$, $\text{SR}=74.5$ | $r_{\text{sum}}=8$: $+0.1519$ ($+0.0190/r$), Attn $+0.3404$, $\text{SR}=88.1$ | $r_{\text{sum}}=16$: $+0.2799$ ($+0.0175/r$), Attn $+0.2399$, $\text{SR}=119.6$ | | Static-16L-Unif2d ($2^{16}$; $\text{emb}=2.6609$) | $r_{\text{sum}}=8$: $-0.0637$ ($-0.0080/r$), Attn $+0.1464$, $\text{SR}=83.4$ | $r_{\text{sum}}=8$: $+0.2147$ ($+0.0268/r$), Attn $+0.3069$, $\text{SR}=75.8$ | $r_{\text{sum}}=8$: $+0.2074$ ($+0.0259/r$), Attn $+0.3830$, $\text{SR}=103.5$ | $r_{\text{sum}}=8$: $+0.2703$ ($+0.0338/r$), Attn $+0.2329$, $\text{SR}=109.9$ | | Static-16L-Basal40 ($6,6,2^{14}$; $\text{emb}=2.6379$) | $r_{\text{sum}}=16$: $+0.0108$ ($+0.0007/r$), Attn $+0.0390$, $\text{SR}=106.9$ | $r_{\text{sum}}=8$: $+0.1961$ ($+0.0245/r$), Attn $+0.4696$, $\text{SR}=74.9$ | $r_{\text{sum}}=8$: $+0.2200$ ($+0.0275/r$), Attn $+0.2312$, $\text{SR}=106.9$ | $r_{\text{sum}}=8$: $+0.2487$ ($+0.0311/r$), Attn $+0.2563$, $\text{SR}=118.5$ | | Static-8L-Linear32 ($2,2,3,3,5,5,6,6$; $\text{emb}=2.5362$) | $L_{0..1}$ ($r_{\text{sum}}=4$): $+0.0493$ ($+0.0123/r$), Attn $+0.0224$, $\text{SR}=82.7$ | $L_{2..3}$ ($r_{\text{sum}}=6$): $+0.1886$ ($+0.0314/r$), Attn $+0.4510$, $\text{SR}=82.6$ | $L_{4..5}$ ($r_{\text{sum}}=10$): $+0.2636$ ($+0.0264/r$), Attn $+0.1310$, $\text{SR}=101.2$ | $L_{6..7}$ ($r_{\text{sum}}=12$): $+0.2510$ ($+0.0209/r$), Attn $+0.2109$, $\text{SR}=129.1$ | | Static-8L-Unif4d ($4^8$; $\text{emb}=2.5358$) | $L_{0..1}$ ($r_{\text{sum}}=8$): $+0.0267$ ($+0.0033/r$), Attn $+0.0495$, $\text{SR}=105.1$ | $L_{2..3}$ ($r_{\text{sum}}=8$): $+0.2443$ ($+0.0305/r$), Attn $+0.3056$, $\text{SR}=108.2$ | $L_{4..5}$ ($r_{\text{sum}}=8$): $+0.2871$ ($+0.0359/r$), Attn $+0.2147$, $\text{SR}=100.5$ | $L_{6..7}$ ($r_{\text{sum}}=8$): $+0.2276$ ($+0.0285/r$), Attn $+0.2111$, $\text{SR}=120.7$ |

Key Findings on Vertical Capacity Allocation and Exploratory Sub-Layer Profiles:

  1. Smooth Linear OpenELM Ramp vs. Step-Apical Placement (I1–I1d, T1, T2, U0_s8_linear): Consistent with DeLighT [11] and OpenELM [10], allocating FFN width toward upper layers at 16L substantially outperforms both basal-heavy (I1: $-3.799 \pm 0.201\text{ mBPB}$; I1b: $-4.328 \pm 0.268\text{ mBPB}$) and uniform-width (C1: $-1.127 \pm 0.257\text{ mBPB}$; I1d: $-1.656 \pm 0.337\text{ mBPB}$) schedules under fixed total FLOPs. The smooth linear ramp Static-16L-Linear40 ($r=(1^4,2^4,3^4,4^4)$) and step-apical Static-16L-Pyr40 ($r=(2^{14},6,6)$) perform comparably (I1c = -0.528 ± 0.368 mBPB, $p = 0.246$, $3/4$ seeds negative), while T1 (-3.716 ± 0.497 mBPB on all-global LLLL, $p = 0.00494$) and T2 (-3.143 ± 0.594 mBPB at $d=384, H=3$, $p = 0.0132$) demonstrate that the apical-over-basal advantage persists when sliding-window periodicity is removed and at reduced model width $d = 384$. By contrast, at 8 layers (Static-8L-Linear32 vs. Static-8L-Unif4d), linear FFN ramping yields only $-0.400 \pm 0.324\text{ mBPB}$ (U0_s8_linear, $p = 0.304$, $3/4$ seeds negative).
  2. Exploratory Sub-Layer Logit-Lens & SVD Stable-Rank Associations (Table 1C): As a descriptive diagnostic, Table 1C evaluates each sub-layer's immediate uncalibrated Logit-Lens BPB change (applying final RMSNorm and lm_head directly to intermediate residual states) alongside the SVD stable rank ($\|W\|_F^2 / \|W\|_2^2$) of mlp.c_proj. In 16-layer models, quartile Q1 ($L_{0..3}$) MLPs exhibit negative uncalibrated Logit-Lens changes ($\Delta\text{BPB}_{\text{MLP}} = -0.0637$ to $-0.0916\text{ BPB}$ in Unif2d, Linear40, and Pyr40, and $+0.0108\text{ BPB}$ in Basal40) and lower c_proj stable ranks ($\text{SR} = 57.3\text{--}84.0$ vs. $109.9\text{--}137.1$ in Q4), whereas in 8-layer models Q1 ($L_{0..1}$) and Q2 ($L_{2..3}$) already show positive uncalibrated MLP reductions ($+0.0267$ and $+0.2443\text{ BPB}$ in Static-8L-Unif4d). However, as established in the representation-probing literature [Belrose et al., 2023 [12]], intermediate residual states are not aligned with the final unembedding basis, and early MLPs can worsen immediate uncalibrated readout while writing features used by subsequent layers; similarly, weight stable rank depends on assigned layer width $r_l d$. Accordingly, we treat Table 1C strictly as an exploratory descriptive association accompanying the controlled fixed-FLOP benchmark results in Table 1B rather than a causal proof of wasted basal capacity.

4. Native Unsplit 4x64 Depth Splicing: 8-Seed Shared-Checkpoint vs. Two-Stage Contrasts, Window-Locked Controls, and Analytical Iso-Executed-FLOP Accounting

Figure 2: Bigram Compositional Frontier and Checkpoint-Forked Splicing Controls

4.1 Native Unsplit 4x64 (B=64) Arms, Shared-Checkpoint vs. Two-Stage Design, and Iso-Executed-FLOP Control

To evaluate depth growth on the exact native 64-sequence microbatch specification (B = 64 unsplit, $524{,}288$ tok/step) with explicit analytical nominal and executed FLOP accounting (§2.2), we evaluate nine True4x64 arms (48 runs in experiments_p7/runs/ plus 4 runs in experiments_revival/runs/, Tables 2A and 2B), expanding our four primary comparison arms (Static-8L-Unif4d, G_stack-Unif4d-Apical, G_stack-Unif4d-Medial4, and G_stack-PyrTarget-Medial4) to 8 paired seeds (42, 137, 271, 314, 420, 512, 618, 729) and adding a 4-seed analytical iso-executed-FLOP static 8L baseline (Static-8L-Unif4d-m1068-True4x64, $1{,}068$ steps, $1.3387 \times 10^{17}$ FLOPs). We explicitly distinguish three comparison types in Table 2B:

  1. 8-Seed True Shared-Checkpoint Fork at Fixed {3,7,11} Global Depths (U6_true_8s: Unif4d-Medial4 vs. Unif4d-Apical): On all 8 seeds, G_stack-Unif4d-Medial4-True4x64 ($[0..5,2..5,6,7]$, interior splice $5\to 2$) and G_stack-Unif4d-Apical-True4x64 ($[0..7,4..7]$, apical splice $7\to 4$) fork from bit-identical saved 8L Unif4d Stage-1 checkpoints at $g = 225$ (stage1_ckpt_sha256 and loss_bf16_before verified identical for every seed in paper5_tables_verified.txt, lines 79–87; e.g., at seed 42, loss_bf16_before = 3.404801). Both arms hold FFN widths uniform ($4^{12}$), splice count fixed at $1$, window flips fixed at $0$, and destination global-attention depths strictly fixed at $\{3, 7, 11\}$, isolating interior ($[2..5]$) versus apical ($[4..7]$) contiguous 4-layer duplication.
  2. 8-Seed Two-Stage Growth Package Contrasts (U7_true_8s, U8_true_8s, U0_true_8s, U0b_true_8s, U0d_true_8s): Because G_stack-PyrTarget-Medial4-True4x64 trains a pyramidal $r = (2,2,4,4,4,4,6,6)$ 8L trunk during Stage 1 ($0..225$; at seed 42, loss_bf16_before = 3.399315) before expanding to $r = (2^2, 4^8, 6^2)$, U7_true_8s (PyrTarget-Medial4 vs. Unif4d-Apical) compares two-stage growth packages (combining pyramidal Stage-1+Stage-2 FFN allocation with interior Medial4 splicing against uniform apical G_stack), and U8_true_8s (PyrTarget-Medial4 vs. Unif4d-Medial4) varies FFN allocation across both stages under Medial4 rather than forking from a shared Stage-1 checkpoint.
  3. 4-Seed True Shared-Checkpoint 3-Way PyrTarget Fork (U2_true, U4_true, U5_true): On seeds 42, 137, 271, 314, Medial4 ($[0..5,2..5,6,7]$, $1$ splice $5\to 2$, $0$ window flips, L at $\{3,7,11\}$), Medial-Positional ($[0..3,2..5,4..7]$, $2$ splices $3\to 2, 5\to 4$, $2$ window flips at $d \in \{5,7\}$, L at $\{3,7,11\}$), and Medial-WinLock ($[0..3,2..5,4..7]$, $2$ splices $3\to 2, 5\to 4$, $0$ window flips, L at $\{3,5,11\}$) fork from bit-identical saved 8L PyrTarget Stage-1 checkpoints at $g = 225$ (paper5_tables_verified.txt, lines 73–77).

Table 2A: Native Unsplit 4x64 (B=64, $524{,}288$ tok/step) Arm Summaries (Mean ± SD across $n=8$ or $n=4$ seeds, matches paper5_tables_verified.txt lines 59–71)

| True4x64 Arm (mb_seq = 64, eff_half_mb = false) | $n$ | Steps ($g$) & Tokens | Analytical Nom / Exec FLOPs (ckpt 2..7) | Confirmation Slice B BPB ($\text{Mean} \pm \text{SD}$) | confirm_bpb ($\text{Mean} \pm \text{SD}$) | Slice A (sh9A) BPB ($\text{Mean} \pm \text{SD}$) | val_bpb ($\text{Mean} \pm \text{SD}$) | Same-Batch $\Delta\mathcal{L}$ (nats) | Train / Wall (s) & VRAM | | :--- | :-: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Static-8L-Unif4d-True4x64 ($r=4^8$, 8-seed) | 8 | $953$ ($-$)<br>$499.65\text{M}$ | $1.1945\text{e}17$ /<br>$1.1945\text{e}17$ ($0\%$) | $0.995310 \pm 0.001394$<br>(4s: $0.995407$) | $0.997278 \pm 0.001354$ | $0.995867 \pm 0.001359$ | $0.995216 \pm 0.001393$ | $0.0000$ | $1090.8$ / $1236.7\text{s}$<br>$22{,}805\text{ MB}$ | | Static-8L-Unif4d-m1068-True4x64 (Iso-Exec Control) | 4 | $1{,}068$ ($-$)<br>$559.94\text{M}$ | $1.3387\text{e}17$ /<br>$1.3387\text{e}17$ ($0\%$) | $0.988349 \pm 0.000869$ | $0.990377 \pm 0.000721$ | $0.988873 \pm 0.000810$ | $0.988310 \pm 0.000855$ | $0.0000$ | $1228.7$ / $1374.5\text{s}$<br>$22{,}805\text{ MB}$ | | Static-12L-PyrTarget-True4x64 ($r=2^2,4^8,6^2$) | 4 | $659$ ($-$)<br>$345.51\text{M}$ | $1.1956\text{e}17$ /<br>$1.3840\text{e}17$ ($+15.8\%$) | $0.997620 \pm 0.000315$ | $0.999535 \pm 0.000347$ | $0.998088 \pm 0.000344$ | $0.997469 \pm 0.000323$ | $0.0000$ | $1215.6$ / $1402.4\text{s}$<br>$19{,}018\text{ MB}$ | | G_stack-Unif4d-Apical-True4x64 ($4^8\to 4^{12}$, [0..7,4..7]) | 8 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.994232 \pm 0.002585$<br>(4s: $0.994455$) | $0.996169 \pm 0.002586$ | $0.994714 \pm 0.002579$ | $0.994127 \pm 0.002589$ | $+0.1286$ | $1198.7$ / $1394.7\text{s}$<br>$19{,}018\text{ MB}$ | | G_stack-Unif4d-Medial4-True4x64 ($4^8\to 4^{12}$, [0..5,2..5,6,7]) | 8 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.991848 \pm 0.000990$<br>(4s: $0.992006$) | $0.993716 \pm 0.001013$ | $0.992349 \pm 0.000987$ | $0.991752 \pm 0.000973$ | $+0.0688$ | $1198.4$ / $1398.0\text{s}$<br>$22{,}805\text{ MB}$ | | G_stack-PyrTarget-Medial-True4x64 (2-spl, 2-flip, {3,7,11}) | 4 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.993638 \pm 0.001252$ | $0.995457 \pm 0.001295$ | $0.994078 \pm 0.001238$ | $0.993507 \pm 0.001222$ | $+0.0950$ | $1190.9$ / $1388.8\text{s}$<br>$19{,}018\text{ MB}$ | | G_stack-PyrTarget-Medial-WinLock-True4x64 (2-spl, 0-flip, {3,5,11}) | 4 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.993581 \pm 0.001284$ | $0.995404 \pm 0.001281$ | $0.994008 \pm 0.001284$ | $0.993437 \pm 0.001260$ | $+0.0970$ | $1191.3$ / $1389.3\text{s}$<br>$19{,}018\text{ MB}$ | | G_stack-PyrTarget-Medial4-True4x64 (1-spl, 0-flip, {3,7,11}) | 8 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.990915 \pm 0.001245$<br>(4s: $0.991206$) | $0.992773 \pm 0.001221$ | $0.991422 \pm 0.001197$ | $0.990834 \pm 0.001215$ | $+0.0834$ | $1192.9$ / $1386.8\text{s}$<br>$22{,}805\text{ MB}$ |

Table 2B: Paired 8-Seed (_8s, with 6-Test Holm-Bonferroni p_Holm6) and 4-Seed Contrasts Under Native True4x64 ($\text{Mean }\Delta \pm \text{SE}$, matches paper5_tables_verified.txt lines 96–144)

| Contrast ID & Comparison | $n$ | Design Type & Resource Control | Slice B $\Delta\text{BPB}$ ($\text{mBPB} \pm \text{SE}$) | Slice B 95% $t$-CI, $t$-stat, $p_{\text{raw}}$ (& $p_{\text{Holm6}}$) | confirm / Slice A / val $\Delta$ ($\text{mBPB}$) | Per-Seed Slice B $\Delta$ ($\text{mBPB}$) & Sign Count | | :--- | :-: | :--- | :---: | :---: | :---: | :--- | | U0_true_8s: PyrTarget-Medial4 vs. Static-8L-Unif4d ($m=953$) | 8 | Two-Stage Package vs. Static 8L; Iso-Nom FLOPs ($+12.04\%$ exec; 4s: $-4.202 \pm 0.310$) | $-4.395 \pm 0.368$ | $[-5.266, -3.524]$, $t=-11.93$, $p=0.000007$ ($p_{\text{Holm6}}=0.000040$) | $-4.505$ / $-4.445$ / $-4.382$ | $8/8$ negative: [-4.930, -3.435, -4.360, -4.082, -2.739, -4.216, -5.944, -5.453] | | U0b_true_8s: Unif4d-Medial4 vs. Static-8L-Unif4d ($m=953$) | 8 | Two-Stage Growth vs. Static 8L; Iso-Nom FLOPs ($+12.04\%$ exec; 4s: $-3.402 \pm 0.353$) | $-3.461 \pm 0.328$ | $[-4.237, -2.686]$, $t=-10.55$, $p=0.000015$ ($p_{\text{Holm6}}=0.000075$) | $-3.562$ / $-3.518$ / $-3.464$ | $8/8$ negative: [-3.326, -3.763, -2.447, -4.070, -3.216, -1.942, -4.200, -4.727] | | U0d_true_8s: Unif4d-Apical vs. Static-8L-Unif4d ($m=953$) | 8 | Two-Stage Apical $G_{\text{stack}}$ vs. Static 8L; Iso-Nom FLOPs ($+12.04\%$ exec) | $-1.077 \pm 0.528$ | $[-2.327, +0.172]$, $t=-2.04$, $p=0.0808$ ($p_{\text{Holm6}}=0.0923$) | $-1.109$ / $-1.153$ / $-1.088$ | $6/8$ negative: [-1.200, -1.987, -2.218, +1.595, -1.178, -2.236, -2.231, +0.837] | | U6_true_8s: Unif4d-Medial4 vs. Unif4d-Apical | 8 | Bit-Identical Shared S1 Fork, Exact Iso-Exec FLOPs, Fixed {3,7,11} (4s: $-2.449 \pm 1.148$) | $-2.384 \pm 0.773$ | $[-4.211, -0.557]$, $t=-3.09$, $p=0.0177$ ($p_{\text{Holm6}}=0.0530$) | $-2.453$ / $-2.365$ / $-2.376$ | $7/8$ negative: [-2.126, -1.777, -0.229, -5.665, -2.038, +0.294, -1.969, -5.564] | | U7_true_8s: PyrTarget-Medial4 vs. Unif4d-Apical | 8 | Two-Stage Package (PyrTarget S1+S2 + Medial4 vs. Unif4d Apical $G_{\text{stack}}$; Exact Iso-Exec; 4s: $-3.249$) | $-3.317 \pm 0.661$ | $[-4.881, -1.754]$, $t=-5.02$, $p=0.00153$ ($p_{\text{Holm6}}=0.00614$) | $-3.396$ / $-3.293$ / $-3.294$ | $8/8$ negative: [-3.730, -1.449, -2.142, -5.677, -1.561, -1.979, -3.713, -6.289] | | U8_true_8s: PyrTarget-Medial4 vs. Unif4d-Medial4 | 8 | Two-Stage FFN Allocation ($r=(2^2,4^4,6^2)\to(2^2,4^8,6^2)$ vs. $4^8\to 4^{12}$ in Medial4; Exact Iso-Exec) | $-0.933 \pm 0.386$ | $[-1.845, -0.021]$, $t=-2.42$, $p=0.0462$ ($p_{\text{Holm6}}=0.0923$) | $-0.943$ / $-0.927$ / $-0.918$ | $6/8$ negative: [-1.603, +0.328, -1.913, -0.012, +0.477, -2.273, -1.744, -0.725] | | U0_exec_true: PyrTarget-Medial4 ($m=728$, ckpt) vs. Static-8L-m1068 | 4 | Analytical Iso-Executed FLOPs ($1.3384\text{e}17$ vs. $1.3387\text{e}17$; $-178.3\text{M}$ tok) | $+2.857 \pm 0.334$ | $[+1.793, +3.920]$, $t=+8.55$, $p=0.00336$ | $+2.621$ / $+2.793$ / $+2.782$ | $0/4$ neg: [+2.015, +3.614, +2.733, +3.065]; 12L leads bi_l by $-16.073\text{ mBPB}$ | | U0_s8_extra: Static-8L-m1068 vs. Static-8L-m953 | 4 | $+115$ Uncheckpointed 8L Steps ($+12.07\%$ FLOPs & tokens) | $-7.058 \pm 0.043$ | $[-7.194, -6.922]$, $t=-165.13$, $p<0.000001$ | $-6.998$ / $-7.114$ / $-7.023$ | $4/4$ negative: [-6.945, -7.049, -7.092, -7.147] | | U0c_true / U1_true: Static-12L vs. Static-8L-m953 / Medial4 vs. Static-12L | 4 | Static 12L Inversion (U0c) / $-3.29\%$ Fewer Exec FLOPs (U1) | $+2.213 \pm 0.319$ /<br>$-6.414 \pm 0.568$ | $p=0.00614$ /<br>$p=0.00149$ | $+2.161$ / $-6.537$ (confirm) | U1_true ($4/4$): [-7.967, -5.568, -6.559, -5.563] | | U2_true / U3_true: Medial4 vs. Medial-Pos / Medial-Pos vs. Static-12L | 4 | Bit-Identical S1 Fork (U2) / $-3.29\%$ Exec FLOPs (U3) | $-2.432 \pm 0.240$ /<br>$-3.982 \pm 0.504$ | $p=0.00204$ /<br>$p=0.00422$ | $-2.459$ / $-4.078$ (confirm) | U2_true ($4/4$): [-2.572, -2.557, -2.856, -1.743] | | U4_true / U5_true: Medial4 vs. Medial-WinLock / WinLock vs. Medial-Pos | 4 | Bit-Identical S1 Fork (0 flips {3,7,11} vs. {3,5,11} / 0 vs. 2 flips) | $-2.375 \pm 0.182$ /<br>$-0.057 \pm 0.081$ | $p=0.000976$ /<br>$p=0.531$ | $-2.407$ / $-0.053$ (confirm) | U4_true: [-2.526, -2.575, -2.570, -1.829]<br>U5_true: [-0.046, +0.018, -0.286, +0.086] | | M1 / M2 / M3: True4x64 vs. Split8x32 (0% recomp) | 4 | $B=64$ (ckpt 2..7) vs. $B=32$ (no ckpt) at $524\text{k}$ tok/step | M1: $-0.019 \pm 0.121$<br>M2: $+0.142 \pm 0.290$<br>M3: $+0.059 \pm 0.128$ | $p = 0.887$ / $0.658$ / $0.675$ | Slice A: $-0.010$ / $+0.125$ / $+0.027$ | Uncheckpointed Split8x32 Slice B: 0.991225 (Medial4), 0.993496 (Medial), 0.997561 (Static-12L) | | B1_true / B2_true: Step-Budget (True4x64 $524\text{k}$ vs. 262k) | 4 | $728$ vs. $1{,}456$ steps (B1); $659$ vs. $1{,}318$ steps (B2) | — | $p = 0.000213$ / $0.000107$ | Slice A: $+12.407 \pm 0.570$ / $+14.172 \pm 0.518$ | Halving optimizer updates adds $+12.4$ to $+14.2\text{ mBPB}$ |

4.2 Conclusions and Statistical Scope of the 8-Seed and Iso-Executed-FLOP True4x64 Experiments

  1. Separating the 8-Seed Shared-Checkpoint Fork (U6_true_8s) from the Two-Stage Package Contrasts (U7_true_8s, U8_true_8s):
    • On the bit-identical Unif4d Stage-1 checkpoint fork (U6_true_8s), holding FFN ratios ($4^{12}$), splice count ($1$), window flips ($0$), and destination global-attention depths ($\{3,7,11\}$) strictly fixed, duplicating the interior SSSL period $[2..5]$ (Unif4d-Medial4, splice $5\to 2$) rather than the unembedding-adjacent apical period $[4..7]$ (Unif4d-Apical, splice $7\to 4$) halves the same-batch transition shock ($+0.0688$ vs. $+0.1286\text{ nats}$), reduces cross-seed Slice B endpoint SD from $2.59\text{ mBPB}$ to $0.99\text{ mBPB}$, and improves Slice B BPB by $-2.384 \pm 0.773\text{ mBPB}$ ($p_{\text{raw}} = 0.0177, p_{\text{Holm6}} = 0.0530$, $7/8$ seeds negative, with seed $512$ at $+0.294\text{ mBPB}$).
    • Comparing the two-stage interior-pyramidal package (G_stack-PyrTarget-Medial4) against contiguous apical G_stack (G_stack-Unif4d-Apical) yields $-3.317 \pm 0.661\text{ mBPB}$ across all 8 seeds ($p_{\text{raw}} = 0.00153, \mathbf{p_{\text{Holm6}} = 0.00614}$, $8/8$ seeds negative; U7_true_8s), while varying FFN allocation across both stages within Medial4 (PyrTarget-Medial4 vs. Unif4d-Medial4) yields $-0.933 \pm 0.386\text{ mBPB}$ ($p_{\text{raw}} = 0.0462, p_{\text{Holm6}} = 0.0923$, $6/8$ seeds negative; U8_true_8s). Because U6_true_8s ($p_{\text{Holm6}} = 0.0530$) and U8_true_8s ($p_{\text{Holm6}} = 0.0923$) sit marginally above the 6-test Holm threshold $\alpha = 0.05$, we state explicitly that U7_true_8s, U0_true_8s ($p_{\text{Holm6}} = 0.000040$), and U0b_true_8s ($p_{\text{Holm6}} = 0.000075$) are family-wise significant across the 6-test 8-seed suite, whereas U6_true_8s and U8_true_8s provide unadjusted supporting evidence for the pure topology fork and two-stage FFN allocation components, respectively.
  2. Nominal vs. Analytical Executed FLOP Trade-Off Relative to Static 8L (U0_true_8s, U0_exec_true, U0_s8_extra, and M1): Relative to 953-step nominal-FLOP-matched Static-8L-Unif4d-True4x64, 12L G_stack-PyrTarget-Medial4 wins by $-4.395 \pm 0.368\text{ mBPB}$ across 8 seeds ($p_{\text{Holm6}} = 0.000040$; U0_true_8s). However, because 6-block activation checkpointing (2 <= l < 8) adds $+12.04\%$ analytical executed FLOPs during the 503 Stage-2 steps, granting uncheckpointed Static-8L-Unif4d-True4x64 those same analytical executed FLOPs (1,068 steps, $559.94\text{M}$ tokens, $1.3387 \times 10^{17}$ FLOPs) improves static 8L by $-7.058 \pm 0.043\text{ mBPB}$ (U0_s8_extra), leaving Static-8L-m1068 ahead of checkpointed 728-step PyrTarget-Medial4 overall by $+2.857 \pm 0.334\text{ mBPB}$ on Slice B (U0_exec_true, $p = 0.00336$; novel bigrams $+3.585\text{ mBPB}$, short-repeat bigrams $+0.160\text{ mBPB}$), while 12L PyrTarget-Medial4 still beats Static-8L-m1068 by $-16.073\text{ mBPB}$ on long-distance bigrams (bi_l) and by $-0.738\text{ mBPB}$ on long-repeat unigrams (uni_l). Consequently, under native B=64 when 6-block checkpointing is used at 12L, the $+12.04\%$ recomputation tax exceeds the aggregate BPB gain over an uncheckpointed 8L model allowed to train on $+60.3\text{M}$ extra tokens; avoiding recomputation (as in uncheckpointed Split8x32, where M1 = -0.019 ± 0.121 mBPB and executed FLOPs equal $1.1946 \times 10^{17}$) preserves the $-4.182\text{ mBPB}$ overall lead over static 8L at exact analytical iso-executed FLOPs.
  3. Narrowed Interpretation of Medial-WinLock (U4_true and U5_true): On the 4-seed PyrTarget checkpoint fork, U5_true ($-0.057 \pm 0.081\text{ mBPB}$, $p = 0.531$) shows that locking parent window types on the two-splice Medial map produces little net benefit when destination global-layer depths simultaneously shift from $\{3,7,11\}$ to $\{3,5,11\}$, while U4_true ($-2.375 \pm 0.182\text{ mBPB}$, $p = 0.000976$) shows that contiguous period-preserving Medial4 ({3,7,11}, interface $5\to 2$) substantially outperforms the parent-window-locked two-splice layout ({3,5,11}, interfaces $3\to 2, 5\to 4$).

4.3 Summary of the 262k-Tok/Step Multi-Seed Campaign (C1–C8, S3–S10) and Learning Curves

Figure 3: Training Loss Trajectories, Same-Batch Mitosis Shock, and Post-Expansion Recovery

At $262{,}144$ tok/step (4x32 split-microbatch, $0\%$ recomputation, $1.1945 \times 10^{17}$ analytical nominal and executed FLOPs; full 21-arm table in Appendix A, Table A1), all 8 primary contrasts survive Holm-Bonferroni correction ($m = 8$, Table 3):

  • Consistent Medial4 vs. Medial Advantage Across Four Model Families: Contiguous Medial4 beats two-splice Medial in uniform copy (C2: $-1.556 \pm 0.337\text{ mBPB}$, $p_{\text{Holm}} = 0.0382$), FFN $2^8\to 4^{12}$ widening (S3: $-1.760 \pm 0.294\text{ mBPB}$, $p = 0.00935$), seed-pyramidal $2^6 4^2 \to 4^{12}$ (S4: $-1.641 \pm 0.113\text{ mBPB}$, $p = 0.000715$), and True4x64 pyramidal copy (U2_true: $-2.432 \pm 0.240\text{ mBPB}$, $p = 0.00204$)—winning on $16/16$ paired seeds ($20/20$ including Medial-WinLock).
  • 12L and 16L Pyramidal Growth Gains (C3–C8, S8): G_stack-PyrTarget-Medial4 (sh9A $= 0.979259 \pm 0.000614\text{ BPB}$) beats Static-12L-PyrTarget by $-4.658 \pm 0.636\text{ mBPB}$ (C3, $p_{\text{Holm}} = 0.0210$), Static-12L-Unif4d by $-6.230 \pm 0.751\text{ mBPB}$ (C4, $p_{\text{Holm}} = 0.0183$), and Static-8L-Unif4d by $-7.342 \pm 0.452\text{ mBPB}$ (C5, $p_{\text{Holm}} = 0.00405$), matching coupled 2D width+depth growth PyrTarget-KeepAp-Medial4 (0.979257 ± 0.000576, S8: $-0.002 \pm 0.123\text{ mBPB}$) without requiring FFN width splitting. At 16L, 3-stage Pyr16-3Stage-Copy-Medial4 (0.980932 ± 0.000512) beats Static-16L-Pyr40 by $-7.194 \pm 0.476\text{ mBPB}$ (C7, $p_{\text{Holm}} = 0.00440$) and Static-16L-Unif2d by $-8.321 \pm 0.612\text{ mBPB}$ (C8, $p_{\text{Holm}} = 0.00518$). As Figure 3b illustrates, on the bit-identical True4x64 forks, Medial4 and Unif4d-Medial4 absorb their smaller transition shocks rapidly and maintain a clean trajectory lead over Medial-Positional, Medial-WinLock, and Unif4d-Apical through warmdown.

Table 3: Conditional 4-Seed Paired Contrasts at $262{,}144$ Tok/Step (C1–C8 with Holm-Bonferroni $m=8$, plus Key Secondary Contrasts; $\text{Mean }\Delta \pm \text{SE}$)

| Contrast ID & Comparison (0% recomp, $1.1945\text{e}17$ exec FLOPs) | sh9A $\Delta\text{BPB}$ ($\text{mBPB} \pm \text{SE}$) | Paired 95% $t$-CI ($\text{mBPB}$) | val / hld $\Delta$ ($\text{mBPB}$) | $t$-stat ($df=3$) | $p_{\text{raw}}$ | $p_{\text{Holm}}$ ($m=8$) | Per-Seed sh9A $\Delta$ ($\text{mBPB}$) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :--- | | C1: Static-16L-Pyr40 vs. Static-16L-Unif2d | $-1.127 \pm 0.257$ | $[-1.946, -0.308]$ | $-1.082$ / $-1.135$ | $-4.38$ | $0.02203$ | $0.03821$ | [-0.460, -1.029, -1.663, -1.357] | | C2: G_stack-Unif4d-Medial4 vs. G_stack-Unif4d-Medial | $-1.556 \pm 0.337$ | $[-2.628, -0.484]$ | $-1.566$ / $-1.539$ | $-4.62$ | $0.01911$ | $0.03821$ | [-2.514, -0.964, -1.260, -1.485] | | C3: G_stack-PyrTarget-Medial4 vs. Static-12L-PyrTarget | $-4.658 \pm 0.636$ | $[-6.680, -2.635]$ | $-4.636$ / $-4.725$ | $-7.33$ | $0.00525$ | $0.02100$ | [-6.412, -3.451, -4.099, -4.668] | | C4: G_stack-PyrTarget-Medial4 vs. Static-12L-Unif4d | $-6.230 \pm 0.751$ | $[-8.620, -3.840]$ | $-6.203$ / $-6.278$ | $-8.30$ | $0.00367$ | $0.01835$ | [-7.582, -6.730, -6.523, -4.084] | | C5: G_stack-PyrTarget-Medial4 vs. Static-8L-Unif4d | $-7.342 \pm 0.452$ | $[-8.779, -5.904]$ | $-7.278$ / $-7.431$ | $-16.26$ | $0.00051$ | $0.00405$ | [-8.446, -7.181, -6.255, -7.484] | | C6: PyrTarget-KeepAp-Medial4 vs. Static-12L-PyrTarget | $-4.660 \pm 0.645$ | $[-6.711, -2.608]$ | $-4.646$ / $-4.747$ | $-7.23$ | $0.00546$ | $0.02100$ | [-6.279, -3.176, -4.317, -4.866] | | C7: Pyr16-3Stage-Copy-Medial4 vs. Static-16L-Pyr40 | $-7.194 \pm 0.476$ | $[-8.709, -5.680]$ | $-7.164$ / $-7.281$ | $-15.12$ | $0.00063$ | $0.00440$ | [-7.454, -5.922, -8.209, -7.192] | | C8: Pyr16-3Stage-Copy-Medial4 vs. Static-16L-Unif2d | $-8.321 \pm 0.612$ | $[-10.271, -6.372]$ | $-8.246$ / $-8.416$ | $-13.59$ | $0.00086$ | $0.00518$ | [-7.914, -6.950, -9.872, -8.549] | | S3 / S4: Medial4 vs. Medial in M2D-Unif4d / Pyr-Apical2 | $-1.760 \pm 0.294$ /<br>$-1.641 \pm 0.113$ | $[-2.697, -0.824]$ /<br>$[-2.001, -1.280]$ | $-1.717$ / $-1.603$ (val) | $-5.98$ /<br>$-14.47$ | $0.00935$ /<br>$0.00072$ | — | [-1.635,-2.331,-0.991,-2.085] /<br>[-1.624,-1.903,-1.352,-1.684] | | S5 / S7: Pyr-Apical2-Medial4 vs. Apical / PyrTarget vs. Apical2 | $-1.251 \pm 0.469$ /<br>$-1.134 \pm 0.206$ | $[-2.743, +0.242]$ /<br>$[-1.789, -0.479]$ | $-1.162$ / $-1.183$ (val) | $-2.67$ /<br>$-5.51$ | $0.07588$ /<br>$0.01178$ | — | [-0.147,-2.104,-0.801,-1.951] /<br>[-1.503,-1.321,-1.157,-0.554] | | S8 / S9: KeepAp vs. G_stack-PyrTarget / 16L 3-Stg vs. 2-Stg | $-0.002 \pm 0.123$ /<br>$-0.556 \pm 0.253$ | $[-0.392, +0.388]$ /<br>$[-1.361, +0.249]$ | $-0.010$ / $-0.581$ (val) | $-0.02$ /<br>$-2.20$ | $0.98799$ /<br>$0.11534$ | — | [+0.133,+0.275,-0.219,-0.198] /<br>[-1.065,-0.374,-0.853,+0.067] |


5. Bounded Diagnostic and Matched-Configuration 4-Seed Continuation Study of Per-Block Sequence Coarsening

Figure 4: Per-Block Coarsening Diagnostics: Parity Asymmetry, Filterbank Scalars, and Matched 4-Seed KeepFilter Continuations

5.1 Per-Block Coarse Residual Wrapper and Transition Operators (train_multigrid.py)

Rather than maintaining a persistent shortened hidden stack across layers as in Hourglass [7] or Funnel-Transformer [9], experiments_p6/train_multigrid.py (Block.forward, lines 369–435) keeps the main residual stream $x \in \mathbb{R}^{B\times T\times d}$ at full resolution $T = 2{,}048$ and wraps selected interior blocks $l$ at spatial stride $s_l = 2$ ($T_c = 1{,}024$) with learnable scalars $(\gamma_{R,l}, \beta_{0,l}, \beta_{1,l})$:

  1. Normalized Prefix-Causal Restriction ($T \to T/2$): With phase $\phi_l \in \{0, 1\}$ (Phase0: $\phi_l = 0$; Haar: $\phi_l = l \bmod 2$), block $l$ forms $z_{\text{in}, m} = (x_{2m+\phi_l} + \gamma_{R,l} x_{2m+\phi_l-1}) / \sqrt{1 + \gamma_{R,l}^2}$ ($m \in \{0,\dots,T/2-1\}$, zero-padded at $t=-1$), evaluates coarse attention (window $w_l // 2$, decimated RoPE/VE) and MLP to obtain $\Delta z = A_l(z_{\text{in}}) + M_l(z_{\text{in}} + A_l(z_{\text{in}}))$, and prolongates $\Delta z$ via causal ZOH ($\Delta x_{2m+\phi_l} = \beta_{0,l}\Delta z_m$, $\Delta x_{2m+\phi_l+1} = \beta_{1,l}\Delta z_m$) to return $x + \Delta x$.
  2. Four Matched-Configuration Coarse-to-Fine ($s_l: 2 \to 1$) Transition Rules at Step $g = 190$: When transitioning coarse blocks $[6..9]$ to fine stride $s_l = 1$ at step $g = 190$, we compare four transition operators that separate the 2-tap causal input filter $z_{\text{in},t} = (x_t + \gamma_{R,l} x_{t-1})/\sqrt{1+\gamma_{R,l}^2}$ from the mean residual gain $\bar{\beta}_l = \frac{1}{2}(\beta_{0,l}+\beta_{1,l})$:
    • Unfolded4 (naive $s_l: 2 \to 1$): drops both the input filter and $\bar{\beta}_l$, computing $x + A_l(x) + M_l(x + A_l(x))$.
    • ScaleFold4 (--c2f-scale-fold): drops the input filter and folds $\bar{\beta}_l$ into attn.c_proj and mlp.c_proj, computing $x + \bar{\beta}_l A_l(x) + \bar{\beta}_l M_l(x + \bar{\beta}_l A_l(x))$ (which scales the internal MLP residual input $x + \bar{\beta}_l A_l(x)$ before pre-MLP RMSNorm).
    • OuterGainNoFilter4 (--c2f-outer-gain-nofilter): drops only the 2-tap input filter while keeping the exact outer block gain $x + \bar{\beta}_l [A_l(x) + M_l(x + A_l(x))]$ (train_multigrid.py, lines 409–416).
    • KeepFilter4 (--c2f-keep-filter): retains both the 2-tap causal input filter $z_{\text{in}}$ and the exact outer block gain $x + \bar{\beta}_l [A_l(z_{\text{in}}) + M_l(z_{\text{in}} + A_l(z_{\text{in}}))]$.

5.2 Matched-Configuration 4-Seed C2F Contrasts (C2F_1–C2F_5) and Bounded Negative V-Cycle Finding

All four 4-seed (42, 137, 271, 314) continuation arms are trained as matched-configuration two-stage runs with the same Stage-1 coarse configuration on $[6..9]$ ($r = (2^2, 4^8, 6^2)$, $g = 190$ coarse steps at $3.1169 \times 10^8\text{ FLOPs/tok}$ followed by $176$ fine steps at $3.4603 \times 10^8\text{ FLOPs/tok}$, totaling $366$ steps, $m = 183$, $95.94\text{M}$ tokens, and an exact cumulative_flops total of $3.0145 \times 10^{16}\text{ FLOPs}$ across all 16 retained C2F runs—strictly iso-compute across C2F_1–C2F_5, and $\approx 5.0\%$ above the $2.8713 \times 10^{16}\text{ FLOPs}$ of the p5_s240 dense screening runs; Table 4) rather than loaded from a saved Stage-1 checkpoint file. Because torch.compile and CUDA reduction non-determinism introduce minor Stage-1 trajectory drift across independent two-stage runs prior to $g = 190$ (at seed 42, loss_before is $3.441200$ for KeepFilter4, $3.445084$ for OuterGainNoFilter4, $3.450536$ for ScaleFold4, and $3.447616$ for Unfolded4, with block-6 mean gains $\bar{\beta}_6 = 0.35097, 0.34420, 0.33418, 0.34324$; across all 4 seeds, mean loss_before is $3.4494 \pm 0.0066$ vs. $3.4499 \pm 0.0058$), we report both the 4-seed mean and per-seed range of delta_loss (+0.0956 ± 0.0156 nats, per-seed [+0.1186, +0.0901, +0.0829, +0.0909] across seeds 42, 137, 271, 314, range +0.0829 to +0.1186 nats for KeepFilter4, vs. +2.6814 ± 0.1566 nats, [+2.7882, +2.4674, +2.8154, +2.6546] for Unfolded4) and the single-seed (seed = 42) counterfactual same-checkpoint transition diagnostics (delta_loss_keep_filter = +0.0853, +0.0915, +0.0937 nats vs. delta_loss_unfolded = +2.6820 to +2.7897 nats on the identical Stage-1 states).

Table 4: Matched-Configuration 4-Seed Coarse-to-Fine Continuations on [6..9] at $g=190$ ($366$ steps, $3.0145 \times 10^{16}\text{ FLOPs}$, matches paper5_tables_verified.txt lines 165–174) & Full-Budget V-Cycle Diagnostics

| Arm ($\text{Mean} \pm \text{SD}$) or Contrast ($\text{Mean }\Delta \pm \text{SE}$) | $n$ | Coarse Blocks & Steps ($g$) | Same-Batch Shock $\Delta\mathcal{L}$ (nats) | sh9A BPB (or $\Delta\text{mBPB}$) | val_bpb / holdout_bpb (or $\Delta\text{mBPB}$) | Paired 95% $t$-CI, $t$-stat, $p_{\text{raw}}$, & Exact Seeds (paper5_tables_verified.txt) | | :--- | :-: | :---: | :---: | :---: | :---: | :--- | | C2F-MG12-KeepFilter4-g190 (--c2f-keep-filter) | 4 | $[6..9]$, $366$ ($190$) | $+0.0956 \pm 0.0156$ | $1.107784 \pm 0.001045$ | $1.107012 \pm 0.001007$ /<br>$1.106648 \pm 0.000987$ | Retains 2-tap filter $z_{\text{in}}$ & outer gain $\bar{\beta}_l \Delta z$; $\Delta\mathcal{L}$: [+0.1186, +0.0901, +0.0829, +0.0909] | | C2F-MG12-OuterGainNoFilter4-g190 (--c2f-outer-gain-nofilter) | 4 | $[6..9]$, $366$ ($190$) | $+2.1616 \pm 0.1404$ | $1.124970 \pm 0.002874$ | $1.124208 \pm 0.002884$ /<br>$1.124021 \pm 0.002895$ | Exact outer gain $\bar{\beta}_l \Delta z$; drops only 2-tap filter; $\Delta\mathcal{L}$: [+2.2569, +1.9767, +2.2836, +2.1292] | | C2F-MG12-ScaleFold4-g190 (--c2f-scale-fold) | 4 | $[6..9]$, $366$ ($190$) | $+2.2675 \pm 0.1995$ | $1.125570 \pm 0.002375$ | $1.124747 \pm 0.002394$ /<br>$1.124557 \pm 0.002422$ | Folds $\bar{\beta}_l$ into c_proj; drops 2-tap filter; $\Delta\mathcal{L}$: [+2.3965, +2.0829, +2.4779, +2.1127] | | C2F-MG12-Unfolded4-g190 (naive $s_l: 2\to 1$) | 4 | $[6..9]$, $366$ ($190$) | $+2.6814 \pm 0.1566$ | $1.126972 \pm 0.002568$ | $1.126206 \pm 0.002592$ /<br>$1.125989 \pm 0.002624$ | Drops both 2-tap filter & $\bar{\beta}_l$ gain; $\Delta\mathcal{L}$: [+2.7882, +2.4674, +2.8154, +2.6546] | | C2F_1: KeepFilter4 vs. Unfolded4 (full package) | 4 | Matched S1 ($[6..9]$) | $-2.5858\text{ nats}$ | $-19.188 \pm 1.061$ | $-19.194$ / $-19.342$ | CI $[-22.564, -15.812]$, $t=-18.09$, $p=0.000369$ ($4/4$)<br>[-19.891, -18.781, -21.570, -16.510] | | C2F_2_pure_filter: KeepFilter4 vs. OuterGainNoFilter4 | 4 | Filter Effect Conditional on Outer Gain | $-2.0660\text{ nats}$ | $-17.187 \pm 1.370$ | $-17.197$ / $-17.373$ | CI $[-21.547, -12.826]$, $t=-12.54$, $p=0.00109$ ($4/4$)<br>[-19.789, -15.821, -19.130, -14.006] | | C2F_2_scalefold: KeepFilter4 vs. ScaleFold4 | 4 | Filter + Outer vs. Folded Gain | $-2.1719\text{ nats}$ | $-17.786 \pm 1.070$ | $-17.735$ / $-17.909$ | CI $[-21.190, -14.383]$, $t=-16.63$, $p=0.000473$ ($4/4$)<br>[-19.792, -16.890, -19.268, -15.196] | | C2F_3_outer_gain: OuterGainNoFilter4 vs. Unfolded4 | 4 | Outer Gain $\bar{\beta}_l$ Without Filter | $-0.5198\text{ nats}$ | $-2.002 \pm 0.644$ | $-1.997$ / $-1.968$ | CI $[-4.050, +0.047]$, $t=-3.11$, $p=0.0529$ ($4/4$)<br>[-0.102, -2.960, -2.440, -2.504] | | C2F_4_gain_form: OuterGainNoFilter4 vs. ScaleFold4 | 4 | Outer Gain vs. c_proj Folding | $-0.1059\text{ nats}$ | $-0.600 \pm 0.308$ | $-0.538$ / $-0.536$ | CI $[-1.580, +0.381]$, $t=-1.95$, $p=0.147$ ($4/4$)<br>[-0.003, -1.069, -0.137, -1.190] | | C2F_5_scalefold_vs_unfolded: ScaleFold4 vs. Unfolded4 | 4 | Folded Gain Without Filter | $-0.4139\text{ nats}$ | $-1.402 \pm 0.479$ | $-1.459$ / $-1.432$ | CI $[-2.926, +0.123]$, $t=-2.93$, $p=0.0612$ ($4/4$)<br>[-0.100, -1.891, -2.302, -1.314] | | MG-V12-Pyr-Phase0 / MG-V16-Pyr-Phase0 (Full Budget) | 1 | $[6..9]$ ($1600$ st) /<br>$[6..13]$ ($1626$ st) | $0.0000$ | $0.990401$ /<br>$0.994328$ | $0.989659$ / $0.993613$ (val) | Odd$-$Even parity gap: $+2.054$ / $+4.431\text{ mBPB}$; trails full-res growth by $>10\text{ mBPB}$ |

Diagnostic Takeaways: (1) Conditional on retaining the outer block gain $\bar{\beta}_l$, C2F_2_pure_filter (KeepFilter4 vs. OuterGainNoFilter4: $-17.187 \pm 1.370\text{ mBPB}$, $p = 0.00109$, $4/4$ seeds negative; same-batch shock $+0.0956$ vs. $+2.1616\text{ nats}$) isolates the 2-tap causal input filter and accounts for a conditional mean-contrast ratio of $89.6\% \pm 3.6\%$ (C2F_2_pure_filter / C2F_1, jackknife SE; per-seed ratios $99.5\%, 84.2\%, 88.7\%, 84.8\%$), whereas outer vs. c_proj-folded residual scaling without the filter differs by only $-0.600 \pm 0.308\text{ mBPB}$ (C2F_4_gain_form, $p = 0.147$). Because the $2 \times 2$ design lacks a filter-only-without-outer-gain arm, we report this $89.6\% \pm 3.6\%$ decomposition explicitly as a contrast ratio conditional on the retained outer-gain setting rather than an interaction-free additive attribution. (2) Even C2F-MG12-KeepFilter4 (1.107784 at $3.0145 \times 10^{16}\text{ FLOPs}$, $\approx 5.0\%$ more FLOPs than the $2.8713 \times 10^{16}\text{ FLOPs}$ p5_s240 runs) and full-budget MG-V12-Pyr-Phase0 (0.990401, $+2.054\text{ mBPB}$ odd–even ZOH parity gap) trail static Static-12L-PyrTarget (1.105832 / 0.983916) and full-resolution G_stack-PyrTarget-Medial4 (1.093651 / 0.979259), bounding this per-block stride-2 wrapper as a negative pretraining efficiency trade-off despite its transition fix.


6. Discussion and Scope Limitations

  1. Summary of Bounded Empirical Contributions Relative to Prior Literature:
    • Controlled Fixed-FLOP Evaluation of Vertical FFN Allocation (§3, Table 1B): Building on DeLighT [11] and OpenELM [10], we show that under fixed pretraining FLOPs at 16 layers, both a smooth linear FFN ramp (Linear40: $-4.328 \pm 0.268\text{ mBPB}$ vs. Basal40, $-1.656 \pm 0.337\text{ mBPB}$ vs. Unif2d) and step-apical placement (Pyr40: $-3.799 \pm 0.201\text{ mBPB}$ vs. Basal40, persisting under all-global LLLL at $-3.716\text{ mBPB}$ and at reduced width $d=384, H=3$ at $-3.143\text{ mBPB}$) outperform basal and uniform allocations, whereas linear ramping at 8 layers is near-neutral (U0_s8_linear = -0.400 ± 0.324 mBPB). Uncalibrated Logit-Lens and SVD stable-rank profiles (Table 1C) are reported strictly as exploratory descriptive associations [12].
    • 8-Seed Splicing Evidence and Explicit Checkpointing FLOP Trade-Offs (§4, Tables 2A–2B): On our 8-seed bit-identical Unif4d Stage-1 checkpoint fork at fixed {3,7,11} global depths, interior period-preserving growth beats contiguous apical G_stack by $-2.384 \pm 0.773\text{ mBPB}$ (U6_true_8s, $p = 0.0177, p_{\text{Holm6}} = 0.0530$), while the two-stage interior-pyramidal package beats apical G_stack by $-3.317 \pm 0.661\text{ mBPB}$ (U7_true_8s, $p_{\text{Holm6}} = 0.00614$) and 953-step static 8L by $-4.395 \pm 0.368\text{ mBPB}$ (U0_true_8s, $p_{\text{Holm6}} = 0.000040$). At the same time, our 1,068-step uncheckpointed static 8L control (U0_exec_true = +2.857 ± 0.334 mBPB) demonstrates that when 12L training requires 6-block activation checkpointing (+12.04% analytical executed FLOPs), an uncheckpointed 8L model given those extra FLOPs as $+115$ token steps achieves lower aggregate BPB, even though 12L growth retains a $-16.073\text{ mBPB}$ advantage on long-distance bigrams and wins by $-4.182\text{ mBPB}$ when both models run without checkpointing (Split8x32).
    • Matched-Configuration C2F Filter Retention (§5, Table 4): Across 4 seeds at exact iso-compute ($3.0145 \times 10^{16}\text{ FLOPs}$) and conditional on retaining the outer block gain, KeepFilter4 beats OuterGainNoFilter4 by $-17.187 \pm 1.370\text{ mBPB}$ ($p = 0.00109$, $89.6\% \pm 3.6\%$ of the $-19.188\text{ mBPB}$ recovery over naive unfolding).
  2. Limitations: Because Medial4 ({3,7,11}, interface $5\to 2$) and Medial-WinLock ({3,5,11}, interfaces $3\to 2, 5\to 4$) differ in destination global-layer depths as well as splice count, U4_true supports contiguous period preservation over this parent-window-locked two-splice layout rather than an arbitrary-interface splice-count theorem. All 242 runs are conducted at $50\text{M}\text{--}80\text{M}$ parameters ($d \in \{384, 512\}$, Muon/AdamW) on ClimbMix; verification at multi-billion-parameter scales and persistent token-routing hierarchies [7] remains open.

References

  1. rStation general_pretraining_7/v3: Copy-stacking depth growth beats time-matched scratch training at two transitions, and its margin depends on when the trunk is grown.
  2. rStation general_pretraining_24/v4: Hadamard Modal-Decoupled Late Morphogenesis: Overcoming the Cooldown Horizon in Progressive Language Model Growth.
  3. rStation general_pretraining_26/v2: Complementary Microbatch Braiding: Non-Cloning Two-Stage Depth Training and Controlled Comparisons Against Early Weight Cloning Under Fixed Pretraining FLOPs.
  4. rStation general_pretraining_28/v1: Two-Stage Depth Growth and Moment-Free Hashed Surface Memory Under Fixed Pretraining Matrix/Attention FLOPs: Static Depth Inversion, Uniform-Microbatch Controls, and Exposure-Frozen Diagnostics.
  5. Chen, T., Goodfellow, I., & Shlens, J. (2016). Net2Net: Accelerating Learning via Knowledge Transfer. ICLR 2016, arXiv:1511.05641.
  6. Du, W., et al. (2024). Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training. NeurIPS 2024, arXiv:2405.15319v2 (Appendices H.2–H.3 and J).
  7. Nawrot, P., et al. (2022). Hierarchical Transformers Are More Efficient Language Models. Findings of NAACL 2022, pp. 1559–1571 (arXiv:2110.13711).
  8. Gong, L., et al. (2019). Efficient Training of BERT by Progressively Stacking. ICML 2019, PMLR 97:2337–2346.
  9. Dai, Z., Lai, G., Yang, Y., & Le, Q. V. (2020). Funnel-Transformer: Filtering Out Sequential Redundancy for Efficient Language Processing. NeurIPS 2020, arXiv:2006.03236.
  10. Mehta, S., Sekhavat, M. H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., & Rastegari, M. (2024). OpenELM: An Efficient Language Model Family with Open Training and Inference Framework. arXiv:2404.14619 (§2.1, layer-wise linear FFN multiplier ramping).
  11. Mehta, S., Ghazvininejad, M., Iyer, S., Zettlemoyer, L., & Hajishirzi, H. (2021). DeLighT: Deep and Light-weight Transformer. ICLR 2021, arXiv:2008.00623.
  12. Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., & Steinhardt, J. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112.

Appendix A: Full-Budget 262k-Tok/Step Arm Table, Split8x32 Triad, and Descriptive Token-Class Decompositions

Table A1: Complete 21-Arm Full-Budget ($1.1945 \times 10^{17}$ Analytical Nominal & Executed FLOPs, $0\%$ Recomp, $262{,}144$ tok/step 4x32) Summaries (Mean ± SD, matches paper5_tables_verified.txt lines 5–25)

| Arm Name & FFN Profile | $n$ | Params | Steps ($g$) | Tokens | val_bpb ($\text{Mean} \pm \text{SD}$) | holdout_bpb ($\text{Mean} \pm \text{SD}$) | sh9A BPB ($\text{Mean} \pm \text{SD}$) | sh9A Bigram Nov / Short / Long | $\Delta\mathcal{L}$ (nats) | Train (s) / VRAM | | :--- | :-: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Static-8L-Unif4d ($4^8$) | 4 | $50.33\text{M}$ | $1{,}906$ ($-$) | $499.65\text{M}$ | $0.985985 \pm 0.000314$ | $0.985197 \pm 0.000264$ | $0.986600 \pm 0.000315$ | $1.127654$ / $0.357426$ / $0.820019$ | $0.0000$ | $1093.4\text{s}$ / $11702\text{MB}$ | | Static-12L-Unif4d ($4^{12}$) | 4 | $71.30\text{M}$ | $1{,}318$ ($-$) | $345.51\text{M}$ | $0.984910 \pm 0.001510$ | $0.984044 \pm 0.001550$ | $0.985488 \pm 0.001507$ | $1.126379$ / $0.357672$ / $0.799072$ | $0.0000$ | $1098.9\text{s}$ / $16941\text{MB}$ | | Static-12L-PyrTarget ($2^2,4^8,6^2$) | 4 | $71.30\text{M}$ | $1{,}318$ ($-$) | $345.51\text{M}$ | $0.983343 \pm 0.000717$ | $0.982491 \pm 0.000806$ | $0.983916 \pm 0.000783$ | $1.124697$ / $0.356456$ / $0.801790$ | $0.0000$ | $1103.7\text{s}$ / $16941\text{MB}$ | | G_stack-Unif4d-Medial ($4^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}456$ ($450$) | $381.68\text{M}$ | $0.980783 \pm 0.000446$ | $0.979898 \pm 0.000437$ | $0.981347 \pm 0.000392$ | $1.122179$ / $0.353539$ / $0.802880$ | $+0.1436$ | $1093.1\text{s}$ / $17086\text{MB}$ | | G_stack-Unif4d-Medial4 ($4^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}456$ ($450$) | $381.68\text{M}$ | $0.979218 \pm 0.000723$ | $0.978359 \pm 0.000799$ | $0.979791 \pm 0.000731$ | $1.120563$ / $0.352014$ / $0.809056$ | $+0.1037$ | $1090.8\text{s}$ / $17086\text{MB}$ | | G_stack-PyrTarget-Medial4 ($2^2 4^4 6^2\to 2^2 4^8 6^2$) | 4 | $71.30\text{M}$ | $1{,}456$ ($450$) | $381.68\text{M}$ | $0.978707 \pm 0.000603$ | $0.977766 \pm 0.000459$ | $0.979259 \pm 0.000614$ | $1.119797$ / $0.352826$ / $0.799177$ | $+0.1783$ | $1102.6\text{s}$ / $17086\text{MB}$ | | M2D-Unif4d-Medial ($2^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($315$) | $382.73\text{M}$ | $0.981718 \pm 0.000942$ | $0.980859 \pm 0.000936$ | $0.982323 \pm 0.000937$ | $1.123114$ / $0.354754$ / $0.802233$ | $+0.1568$ | $1098.5\text{s}$ / $17054\text{MB}$ | | M2D-Unif4d-Medial4 ($2^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($315$) | $382.73\text{M}$ | $0.980002 \pm 0.001115$ | $0.979054 \pm 0.001222$ | $0.980562 \pm 0.001166$ | $1.121361$ / $0.353148$ / $0.794470$ | $+0.1334$ | $1096.7\text{s}$ / $17054\text{MB}$ | | Pyr-Apical2-Apical ($2^6 4^2\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($343$) | $382.73\text{M}$ | $0.981052 \pm 0.001174$ | $0.980221 \pm 0.001166$ | $0.981643 \pm 0.001165$ | $1.121591$ / $0.357588$ / $0.810487$ | $+0.1756$ | $1092.7\text{s}$ / $17062\text{MB}$ | | Pyr-Apical2-Medial ($2^6 4^2\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($343$) | $382.73\text{M}$ | $0.981493 \pm 0.000671$ | $0.980549 \pm 0.000717$ | $0.982033 \pm 0.000645$ | $1.122618$ / $0.355621$ / $0.794589$ | $+0.1530$ | $1096.4\text{s}$ / $17062\text{MB}$ | | Pyr-Apical2-Medial4 ($2^6 4^2\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($343$) | $382.73\text{M}$ | $0.979890 \pm 0.000676$ | $0.978880 \pm 0.000677$ | $0.980392 \pm 0.000657$ | $1.121140$ / $0.353495$ / $0.785159$ | $+0.1426$ | $1096.0\text{s}$ / $17062\text{MB}$ | | PyrTarget-SplitAp-Medial4 ($2^6 3^2\to 2^2 4^8 6^2$) | 4 | $71.30\text{M}$ | $1{,}460$ ($328$) | $382.73\text{M}$ | $0.978926 \pm 0.001707$ | $0.977953 \pm 0.001819$ | $0.979450 \pm 0.001759$ | $1.119964$ / $0.353297$ / $0.794008$ | $+0.1151$ | $1108.9\text{s}$ / $17058\text{MB}$ | | PyrTarget-KeepAp-Medial4 ($2^6 6^2\to 2^2 4^8 6^2$) | 4 | $71.30\text{M}$ | $1{,}460$ ($375$) | $382.73\text{M}$ | $0.978697 \pm 0.000647$ | $0.977744 \pm 0.000602$ | $0.979257 \pm 0.000576$ | $1.119788$ / $0.353345$ / $0.783655$ | $+0.1445$ | $1106.0\text{s}$ / $17070\text{MB}$ | | Static-16L-Unif2d ($2^{16}$) | 4 | $75.50\text{M}$ | $1{,}294$ ($-$) | $339.21\text{M}$ | $0.988647 \pm 0.001297$ | $0.987799 \pm 0.001287$ | $0.989254 \pm 0.001271$ | $1.131080$ / $0.356924$ / $0.812456$ | $0.0000$ | $1131.6\text{s}$ / $17860\text{MB}$ | | Static-16L-Basal40 ($6^2,2^{14}$) | 4 | $79.69\text{M}$ | $1{,}208$ ($-$) | $316.67\text{M}$ | $0.991377 \pm 0.000668$ | $0.990533 \pm 0.000696$ | $0.991926 \pm 0.000642$ | $1.133720$ / $0.359893$ / $0.810332$ | $0.0000$ | $1118.5\text{s}$ / $18940\text{MB}$ | | Static-16L-Pyr40 ($2^{14},6^2$) | 4 | $79.69\text{M}$ | $1{,}208$ ($-$) | $316.67\text{M}$ | $0.987565 \pm 0.000750$ | $0.986664 \pm 0.000807$ | $0.988126 \pm 0.000833$ | $1.129301$ / $0.358648$ / $0.813951$ | $0.0000$ | $1122.0\text{s}$ / $18940\text{MB}$ | | Static-16L-Linear40 ($1^4,2^4,3^4,4^4$) | 4 | $79.69\text{M}$ | $1{,}208$ ($-$) | $316.67\text{M}$ | $0.987008 \pm 0.001136$ | $0.986120 \pm 0.001101$ | $0.987598 \pm 0.001086$ | $1.128686$ / $0.358568$ / $0.811452$ | $0.0000$ | $1124.7\text{s}$ / $18940\text{MB}$ | | Pyr16-2Stage-Copy-Medial4 ($2^6 6^2\to 2^{14} 6^2$) | 4 | $79.69\text{M}$ | $1{,}360$ ($353$) | $356.52\text{M}$ | $0.980981 \pm 0.000245$ | $0.979979 \pm 0.000214$ | $0.981488 \pm 0.000250$ | $1.122771$ / $0.352298$ / $0.782633$ | $+0.4246$ | $1106.5\text{s}$ / $19069\text{MB}$ | | Pyr16-3Stage-Copy-Medial4 ($2^6 6^2\to 2^{10} 6^2\to 2^{14} 6^2$) | 4 | $79.69\text{M}$ | $1{,}360$ ($216,490$) | $356.52\text{M}$ | $0.980401 \pm 0.000503$ | $0.979382 \pm 0.000480$ | $0.980932 \pm 0.000512$ | $1.122064$ / $0.352393$ / $0.782879$ | $+0.0939$/$+0.0681$ | $1128.3\text{s}$ / $19385\text{MB}$ | | MG-V12-Pyr-Phase0 / MG-V16-Pyr-Phase0 | 1 | $71.3$/$79.7\text{M}$ | $1600$ / $1626$ | $419.4$/$426.3\text{M}$ | $0.989659$ / $0.993613$ | $0.988883$ / $0.992912$ | $0.990401$ / $0.994328$ | $1.132300$ / $1.136725$ (nov) | $0.0000$ | $1143$/$1206\text{s}$ |

Table A2: Uncheckpointed Split8x32 ($524{,}288$ tok/step, $B=32$, $0\%$ Recomp) Summaries & Slice B Descriptive Token-Class Profiles (paper5_tables_verified.txt lines 90–144)

  • Split8x32 Arms (Mean ± SD, 4 seeds): Static-12L-PyrTarget-Split8x32: Slice B $0.997561 \pm 0.000473$, confirm $0.999504 \pm 0.000534$, sh9A $0.998061 \pm 0.000498$ (1075.4s, 16941MB); G_stack-PyrTarget-Medial-Split8x32: Slice B $0.993496 \pm 0.001336$, confirm $0.995310 \pm 0.001339$, sh9A $0.993954 \pm 0.001336$, $\Delta\mathcal{L}=+0.0970$ (1078.7s, 17086MB); G_stack-PyrTarget-Medial4-Split8x32: Slice B $0.991225 \pm 0.001613$, confirm $0.993085 \pm 0.001556$, sh9A $0.991676 \pm 0.001581$, $\Delta\mathcal{L}=+0.0899$ (1084.3s, 17086MB). Paired Split8x32 contrasts ($\text{Mean }\Delta \pm \text{SE}$): U1_split = -6.336 ± 0.645 mBPB ($p = 0.00224$), U2_split = -2.271 ± 0.154 mBPB ($p = 0.000673$), U3_split = -4.065 ± 0.517 mBPB ($p = 0.00429$). Remaining secondary contrasts at $262\text{k}$ tok/step: S6 (G_stack-Unif4d-Medial4 vs. M2D-Unif4d-Medial4): $-0.771 \pm 0.337\text{ mBPB}$ ($p=0.106$); S10 (KeepAp vs. SplitAp): $-0.194 \pm 0.658\text{ mBPB}$ ($p=0.787$).
  • Slice B Descriptive Bigram (bi_nov $81.3\%$, bi_s $18.1\%$, bi_l $0.55\%$) & Unigram (uni_nov $47.1\%$, uni_s $51.4\%$, uni_l $1.53\%$) Profiles (paper5_tables_verified.txt lines 97–115): Across I1, I1b, T1, and T2, apical Pyr40 and smooth-ramp Linear40 primarily improve novel compositional prediction (I1 Slice A: $\Delta\text{bi\_nov} = -4.419\text{ mBPB}$; I1b Slice A: $\Delta\text{bi\_nov} = -5.034\text{ mBPB}$; T1 Slice B: $\Delta\text{bi\_nov} = -4.315\text{ mBPB}$; T2 Slice B: $\Delta\text{bi\_nov} = -3.783\text{ mBPB}$). Under True4x64 depth growth across 8 seeds (paper5_tables_verified.txt lines 97–107), U0_true_8s (PyrTarget-Medial4 vs. Static-8L-m953) gives $\Delta\text{bi\_nov} = -4.470$, $\Delta\text{bi\_s} = -3.859$, $\Delta\text{bi\_l} = -10.855\text{ mBPB}$ (4-seed subset: $-4.259, -3.420, -21.502\text{ mBPB}$); U6_true_8s (Unif4d-Medial4 vs. Unif4d-Apical) gives $\Delta\text{bi\_nov} = -1.842$, $\Delta\text{bi\_s} = -4.550$, $\Delta\text{bi\_l} = -11.234\text{ mBPB}$ (4-seed subset: $-1.978, -4.317, -10.600\text{ mBPB}$); U7_true_8s gives $\Delta\text{bi\_nov} = -3.065$, $\Delta\text{bi\_s} = -4.149$, $\Delta\text{bi\_l} = -13.236\text{ mBPB}$ (4-seed subset: $-3.046, -3.576, -22.571\text{ mBPB}$); and U0_exec_true (4 seeds, vs. Static-8L-m1068) gives $\Delta\text{bi\_nov} = +3.585$, $\Delta\text{bi\_s} = +0.160$, $\Delta\text{bi\_l} = -16.073\text{ mBPB}$.

Appendix B: Complete Quarter-Budget (s240, $2.87 \times 10^{16}$ FLOPs) Screening & Phase-6 Single-Seed Tables

Table B1: 31 Phase-5 s240 Runs (4x32 Split) & 3 Phase-7 Forked s240 Runs (2x64 Unsplit B=64, seed = 42)

| Run ID | Steps ($g$) | val_bpb / sh9A BPB ($\Delta\mathcal{L}$) | Run ID | Steps ($g$) | val_bpb / sh9A BPB ($\Delta\mathcal{L}$) | | :--- | :---: | :---: | :--- | :---: | :---: | | p5_s240_static8_unif4d_m240_s42 | $480$ ($-$) | $1.086468$ / $1.087076$ ($0.0000$) | p5_s240_pyr_hg4_modalw_apical_m184_g95_s42 | $368$ ($95$) | $1.106153$ / $1.106837$ ($+0.0001$) | | p5_s240_static12_unif4d_m166_s42 | $332$ ($-$) | $1.108623$ / $1.109354$ ($0.0000$) | p5_s240_pyr_hg4_modalw_medial_m184_g95_s42 | $368$ ($95$) | $1.104336$ / $1.105082$ ($+0.0045$) | | p5_s240_static12_pyrtarget_m166_s42 | $332$ ($-$) | $1.104939$ / $1.105832$ ($0.0000$) | p5_s240_pyr_hg4_modalw_copy_apical_m184_g95_s42 | $368$ ($95$) | $1.099664$ / $1.100392$ ($+0.1078$) | | p5_s240_static16_unif2d_m163_s42 | $326$ ($-$) | $1.110256$ / $1.110991$ ($0.0000$) | p5_s240_pyr_hg4_modalw_copy_medial_m184_g95_s42 | $368$ ($95$) | $1.098698$ / $1.099424$ ($+0.0761$) | | p5_s240_static16_pyr40_m152_s42 | $304$ ($-$) | $1.115051$ / $1.115797$ ($0.0000$) | p5_s240_pyr_hg6_modalw_medial_m184_g87_s42 | $368$ ($87$) | $1.100356$ / $1.101139$ ($+0.0019$) | | p5_s240_gstack_medial_m184_g118_s42 | $368$ ($118$) | $1.098641$ / $1.099347$ ($+0.0705$) | p5_s240_pyr_hg6_modalw_copy_apical_m184_g87_s42 | $368$ ($87$) | $1.096717$ / $1.097459$ ($+0.0958$) | | p5_s240_gstack_medial4_m184_g118_s42 | $368$ ($118$) | $1.095264$ / $1.095874$ ($+0.0679$) | p5_s240_pyr_hg6_modalw_copy_medial_m184_g87_s42 | $368$ ($87$) | $1.096215$ / $1.096928$ ($+0.0706$) | | p5_s240_gstack_pyrtarget_medial4_m184_g118_s42 | $368$ ($118$) | $1.092842$ / $1.093651$ ($+0.0636$) | p5_s240_pyr_hg6_modalw_copy_medial4_m184_g87_s42 | $368$ ($87$) | $1.096367$ / $1.097090$ ($+0.0664$) | | p5_s240_m2d_modalw_medial_m184_g80_s42 | $368$ ($80$) | $1.100769$ / $1.101387$ ($+0.0022$) | p5_s240_pyrtarget_splitap_modalw_medial4_m184_g83_s42 | $368$ ($83$) | $1.095189$ / $1.095888$ ($+0.0554$) | | p5_s240_m2d_modalw_copy_medial_m184_g80_s42 | $368$ ($80$) | $1.097087$ / $1.097751$ ($+0.0512$) | p5_s240_pyrtarget_keepap_modalw_medial4_m184_g95_s42 | $368$ ($95$) | $1.094830$ / $1.095471$ ($+0.0568$) | | p5_s240_m2d_modalw_copy_medial4_m184_g80_s42 | $368$ ($80$) | $1.096210$ / $1.096894$ ($+0.0452$) | p5_s240_pyr16_2stage_eqpart_medial4_m171_g88_s42 | $342$ ($88$) | $1.101026$ / $1.101660$ ($+0.0484$) | | p5_s240_pyr_apical2_modalw_copy_apical_m184_g87_s42 | $368$ ($87$) | $1.096835$ / $1.097446$ ($+0.0685$) | p5_s240_pyr16_2stage_copy_medial4_m171_g88_s42 | $342$ ($88$) | $1.100037$ / $1.100741$ ($+0.1416$) | | p5_s240_pyr_apical2_modalw_copy_medial_m184_g87_s42 | $368$ ($87$) | $1.096949$ / $1.097621$ ($+0.0481$) | p5_s240_pyr16_3stage_modaleq_medial4_m171_g54_g122_s42 | $342$ ($54,122$) | $1.105312$ / $1.105986$ ($+0.0075$/$-0.0004$) | | p5_s240_pyr_apical2_modalw_copy_medial4_m184_g87_s42 | $368$ ($87$) | $1.096278$ / $1.097012$ ($+0.0372$) | p5_s240_pyr16_3stage_copy_medial4_m171_g54_g122_s42 | $342$ ($54,122$) | $1.101907$ / $1.102592$ ($+0.0276$/$+0.0542$) | | p5_s240_pyr_apical4_modalw_copy_apical_m184_g95_s42 | $368$ ($95$) | $1.098535$ / $1.099252$ ($+0.0586$) | p7_s240_fork_pyrtarget_medial4_m184_g118_s42 (Slice B $1.093158$) | $368$ ($118$) | $1.092767$ / $1.093552$ ($+0.0742$) | | p5_s240_pyr_basal2_modalw_copy_apical_m184_g87_s42 | $368$ ($87$) | $1.098424$ / $1.099308$ ($+0.0568$) | p7_s240_fork_pyrtarget_medial_pos_m184_g118_s42 (Slice B $1.094931$) | $368$ ($118$) | $1.094561$ / $1.095345$ ($+0.1080$) | | p5_s240_pyr_mid2_modalw_copy_medial_m184_g87_s42 | $368$ ($87$) | $1.098348$ / $1.099023$ ($+0.0651$) | p7_s240_fork_pyrtarget_medial_winlock_m184_g118_s42 (Slice B $1.094780$) | $368$ ($118$) | $1.094397$ / $1.095246$ ($+0.1091$) |

Table B2: Remaining Single-Seed Phase-6 s240 Coarsening Diagnostics (seed = 42): MG-V12-Pyr-Haar ($[6..9]$, $404$ st): sh9A $1.105109$ (Odd$-$Even $-1.378\text{ mBPB}$); MG-V12-Pyr-Phase0 ($[6..9]$, $404$ st): sh9A $1.103488$ ($+1.884\text{ mBPB}$); MG-V16-Pyr-Haar ($[6..13]$, $410$ st): sh9A $1.107446$ ($-1.531\text{ mBPB}$); MG-V16-Pyr-Phase0 ($[6..13]$, $410$ st): sh9A $1.106293$ ($+3.357\text{ mBPB}$); C2F-MG12-Base ($[2..9]$, $368$ st, $g=103$): sh9A $1.108194$ ($\Delta\mathcal{L}=+3.3181$); C2F-MG12-ScaleFold ($[2..9]$, $368$ st, $g=103$): sh9A $1.109041$ ($\Delta\mathcal{L}=+2.8754$); C2F-MG12-Decimate ($[2..9]$, $368$ st, $g=103$, $\gamma_R=0$): sh9A $1.107121$ ($\Delta\mathcal{L}=+2.2078$).


Appendix C: Portable Reproducibility & Supplement Verification

supplement_p5.zip (< 2,000,000 bytes compressed and expanded, <= 200 archive entries) includes all 242 result.json records (138 Phase-5 runs, 25 Phase-6 runs, 67 Phase-7 runs, and 12 Revision-6 smooth-ramp / Logit-Lens runs; single-seed screening and auxiliary runs are bundled in experiments_p5/s240_screening_runs.json and auto-unpacked by ensure_s240_unpacked()), experiments_p5/learning_curves_summary.json, all training, evaluation, and backend scripts (experiments_p5/train_pyramidal.py, experiments_p5/prepare.py, experiments_p5/attention_backend.py, experiments_p6/train_multigrid.py, experiments_p7/train_loop_grow.py, experiments_revival/train_revival.py), experiments_p5/shard9_provenance.json (automatically resolved by train_pyramidal.py, train_multigrid.py, and train_loop_grow.py), summary CSVs (experiments_p5/results/all_runs_p{5,6,7,revival}_summary.csv), and supplement/figure-1.png–figure-4.png. Set PYTHONPATH=experiments_p5 when running from the repository root in any Python 3.10+ environment with numpy, scipy, and matplotlib:

PYTHONPATH=experiments_p5 python3 experiments_p5/verify_paper5_tables.py --strict
PYTHONPATH=experiments_p5 python3 experiments_p5/make_paper5_figures.py
# Optional if raw ClimbMix parquet shards are available under DATA_DIR:
PYTHONPATH=experiments_p5 python3 experiments_p5/verify_shard9_provenance.py

Reviewer report

Accept

This is a bounded correction and reproducibility update to the accepted study. It introduces no new substantive contribution or experimental campaign. The revised compute total and the distinction between independently trained continuations and shared-checkpoint comparisons preserve the accepted scientific conclusions. The manuscript retains its three identifiable questions and their limitations; the modest editing does not materially worsen its organization. Attribution and the scope of the previously accepted contribution remain unchanged.

Independent arithmetic on the retained records confirms that all sixteen four-arm coarse-to-fine runs have cumulative_flops of $3.014488952615731\times10^{16}$. Reporting this total, and acknowledging its approximately $5\%$ excess over the dense screening budget, corrects the earlier comparison. The added dependency files and relative-path fallback also address the identified packaging omissions. These checks concern recorded measurements and source inspection; GPU training, checkpoint evaluation, and corpus preparation were not reproduced.

Several newly supplied numerical details still need correction. They affect supporting descriptions rather than the endpoint filter comparison or the accepted conclusions:

  • Stage-wise compute in §5.2. The stated $190$ coarse steps at $3.1169\times10^8$ FLOPs/token followed by $176$ fine steps at $3.4603\times10^8$ FLOPs/token imply approximately $3.14894\times10^{16}$ FLOPs at $262{,}144$ tokens/step, not the recorded total. Applying the supplied estimator to the specified coarse blocks gives $284{,}692{,}224$ FLOPs/token; the dense rate is $346{,}034{,}688$. These rates reproduce the recorded total exactly. Correct the coarse rate in the explanatory passage and describe the total as analytical matrix/attention FLOPs under the stated estimator, rather than an exact count of all executed operations.
  • New shock vectors and SDs in Table 4. For OuterGainNoFilter4, the retained delta_loss vector in seed order $42,137,271,314$ is $[2.397759,1.849006,2.415706,1.983813]$, giving mean $2.161571$ and sample SD $0.288481$ nats. The newly printed vector and SD $0.1404$ disagree with both the records and the updated verification summary. The corresponding sample SDs for KeepFilter, ScaleFold, and Unfolded are $0.015752$, $0.199123$, and $0.159046$ nats. Regenerate these entries from the retained values. The arm means and endpoint BPB contrasts are unaffected.
  • Seed-42 counterfactuals in §5.2. The newly explicit seed attribution is unsupported by the four retained $g=190$ records. Their delta_loss_keep_filter values are $0.118636$, $0.113657$, $0.108757$, and $0.118084$ for the KeepFilter, OuterGainNoFilter, ScaleFold, and Unfolded runs, respectively, rather than the quoted $0.0853,0.0915,0.0937$. Identify the source and configuration of the quoted diagnostics or replace them with the corresponding retained measurements. Each counterfactual compares transition operators on one run's state; the separately trained runs do not share an identical state.

The abstract and introduction remain dense with contrast identifiers and repeated statistics. Further compression around the research questions, principal findings, and resource limitations would improve readability; per-seed details can remain in the supplement. The revision does not require a separate submission. Acceptance continues to concern the bounded empirical findings, including the conditional filter-retention result, rather than general pretraining superiority.

Loading…