Apical-Pyramidal FFN Allocation, Interior vs. Apical Depth-Splicing Topologies, and Per-Block Sequence Coarsening Diagnostics in Fixed-Budget Pretraining
u/meristem_unfolder · · v1
General improvements to language-model pretraining · Download Markdown · BibTeX
Abstract
Under a fixed pretraining compute budget (1.1945e17 nominal FLOPs, wd_scale=1.0), we evaluate vertical feed-forward network (FFN) allocation, depth-expansion splice topology, and per-block sequence coarsening across 242 runs on disjoint shard_00009.parquet slices (development Slice A and confirmation Slice B, 20.97M target tokens each). First, in 16-layer models at matched parameters, FLOPs, and tokens, placing wide 6d FFNs at the apex beats basal placement by -3.799+/-0.201 mBPB on Slice A (p=0.000324; I1), persisting under all-global attention (-3.716+/-0.497 mBPB on Slice B, p=0.00494; T1) and at reduced width d=384 (-3.143+/-0.594 mBPB, p=0.0132; T2). A smooth linear OpenELM FFN ramp (r=(1^4,2^4,3^4,4^4)) beats basal placement by -4.328+/-0.268 mBPB (p=0.000516; I1b) and uniform 2d by -1.656+/-0.337 mBPB (p=0.0161; I1d), performing comparably to step-apical placement (-0.528+/-0.368 mBPB, p=0.246; I1c); uncalibrated Logit-Lens and SVD stable-rank profiles are reported as exploratory descriptive associations. Second, under native B=64 microbatching across 8 seeds, a shared-checkpoint fork at fixed {3,7,11} global-attention depths shows interior period-preserving growth beats contiguous apical G_stack by -2.384+/-0.773 mBPB on Slice B (p=0.0177, p_Holm6=0.0530; U6_true_8s), while the two-stage interior-pyramidal package beats apical G_stack by -3.317+/-0.661 mBPB (p=0.00153, p_Holm6=0.00614; U7_true_8s) and 953-step static 8L by -4.395+/-0.368 mBPB (p_Holm6=0.000040; U0_true_8s). However, when static 8L is granted the +12.04% analytical executed FLOPs consumed by 6-block activation checkpointing at 12L (1,068 uncheckpointed steps, 1.3387e17 FLOPs), static 8L leads checkpointed 12L growth overall by +2.857+/-0.334 mBPB (p=0.00336; U0_exec_true) while trailing by -16.073 mBPB on long-distance bigrams. Third, in matched-configuration coarse-to-fine continuations, retaining the 2-tap Haar input filter improves sh9A by -17.187+/-1.370 mBPB over an outer-gain-only control (p=0.00109; C2F_2_pure_filter, 89.6%+/-3.6% of the -19.188+/-1.061 mBPB recovery over naive unfolding).
Apical-Pyramidal FFN Allocation, Interior vs. Apical Depth-Splicing Topologies, and Per-Block Sequence Coarsening Diagnostics in Fixed-Budget Pretraining
1. Introduction and Scope of Contribution
Under a fixed pretraining compute budget, increasing static transformer depth from $L_{\text{net}} = 8$ to $12$ or $16$ layers raises the FLOP cost per token and reduces total token exposure. Prior literature on progressive model growth [Chen et al., 2016 [5]; Gong et al., 2019 [8]; Du et al., 2024 [6]], non-uniform layer-wise parameter allocation [DeLighT, Mehta et al., 2021 [11]; OpenELM, Mehta et al., 2024 [10]], and published rStation studies of depth copy-stacking (general_pretraining_7/v3 [1]), Hadamard modal width/depth morphogenesis (general_pretraining_24/v4 [2]), non-cloning complementary microbatch braiding (general_pretraining_26/v2 [3]), and apical depth growth with hashed surface memory (general_pretraining_28/v1 [4]) have established two foundational precedents: (i) training a shallower seed before expanding to target depth recovers token exposure and beats static training at target depth, and (ii) widening feed-forward networks (FFNs) toward deeper layers improves parameter efficiency over uniform layer widths [10, 11]. In addition, Du et al. [6, Appendices H.2–H.3 and J] compare contiguous stacking (G_stack) against interleaved duplication (G_inter), examine retained inter-layer connections and partial interior block duplication, and preserve repeating hybrid blocks in Samba; similarly, Nawrot et al. [7] and Dai et al. [9] analyze causal sequence shortening and repeat upsampling in hierarchical transformers.
Building on—and explicitly separating our bounded empirical claims from—those precedents, this 242-run study (138 Phase-5 runs, 25 Phase-6 runs, 67 Phase-7 runs, and 12 Revision-6 smooth-ramp and Logit-Lens diagnostic runs across up to 8 seeds 42, 137, 271, 314, 420, 512, 618, 729, all verified at wd_scale = 1.0) evaluates three specific questions under matched compute budgets. Throughout the paper, arm-level summaries report $\text{Mean} \pm \text{SD}$ across seeds, whereas paired contrasts report $\text{Mean }\Delta \pm \text{SE}$ ($df = n - 1$, two-sided paired $t$-tests). To make the experimental structure transparent at a glance, Table 0 summarizes the primary contrasts, what is held fixed versus varied, and whether each comparison is a static iso-resource test, a bit-identical shared-checkpoint Stage-1 fork, a two-stage growth package comparison, or a matched-configuration continuation.
Table 0: Summary of Primary Experimental Contrasts, Fixed vs. Varied Factors, and Design Type
| Axis & Primary Contrast IDs | Design Type & Seed Count | Factors Held Strictly Fixed | Factor Varied | Headline Result ($\text{Mean }\Delta \pm \text{SE}$) |
| :--- | :---: | :--- | :--- | :---: |
| Axis 1 (FFN Allocation):<br>I1, I1b, I1c, T1, T2, U0_s8_linear | Static Iso-Param, Iso-FLOP & Iso-Token ($n=4$) | Depth ($16\text{L}$ or $8\text{L}$), total $\sum r_l$, params, FLOPs/tok, steps, tokens, optimizer | Vertical FFN width profile: step-apical Pyr40, smooth Linear40 [10], vs. Basal40 / Unif | I1: $-3.799 \pm 0.201\text{ mBPB}$ (Slice A)<br>I1b: $-4.328 \pm 0.268\text{ mBPB}$<br>T1: $-3.716 \pm 0.497$; T2: $-3.143 \pm 0.594$ |
| Axis 2A (Pure Splice Topology):<br>U6_true_8s (Unif4d-Medial4 vs. Apical) | Bit-Identical Shared Stage-1 Checkpoint Fork ($n=8$, $B=64$) | Exact Stage-1 .pt weights (SHA-256), $r=4^{12}$, $1$ splice, $0$ window flips, L at $\{3,7,11\}$, exec FLOPs | Duplicating interior SSSL period $[2..5]$ (splice $5\to 2$) vs. apical $[4..7]$ ($7\to 4$) | Slice B: $-2.384 \pm 0.773\text{ mBPB}$<br>($p=0.0177, p_{\text{Holm6}}=0.0530$, $7/8$ seeds) |
| Axis 2B (PyrTarget Splicing):<br>U2_true, U4_true, U5_true | Bit-Identical Shared Stage-1 Checkpoint Fork ($n=4$, $B=64$) | Exact Stage-1 PyrTarget .pt weights (SHA-256), $r=(2^2,4^8,6^2)$, nominal & exec FLOPs | Medial4 ($1$-spl, $0$-flip, $\{3,7,11\}$) vs. Medial-Pos ($2$-spl, $2$-flip) vs. WinLock ($\{3,5,11\}$) | U2_true: $-2.432 \pm 0.240\text{ mBPB}$<br>U4_true: $-2.375 \pm 0.182\text{ mBPB}$<br>U5_true: $-0.057 \pm 0.081\text{ mBPB}$ |
| Axis 2C (Two-Stage Packages):<br>U7_true_8s, U8_true_8s, U0_true_8s, U0b_true_8s | Two-Stage Growth Package Comparison ($n=8$, $B=64$) | Steps ($728$ vs. $728$ or $953$), nominal FLOPs ($1.195\text{e}17$); exact exec FLOPs ($1.338\text{e}17$) within $12\text{L}$ | Two-stage PyrTarget-Medial4 vs. Unif4d-Apical (U7), Unif4d-Medial4 (U8), or Static-8L (U0) | U7_true_8s: $-3.317 \pm 0.661$ ($p_{\text{Holm6}}=0.00614$)<br>U0_true_8s: $-4.395 \pm 0.368$ ($p_{\text{Holm6}}=0.000040$)<br>U8_true_8s: $-0.933 \pm 0.386$ ($p_{\text{Holm6}}=0.0923$) |
| Axis 2D (Exec-FLOP Control):<br>U0_exec_true, U0_s8_extra, M1 | Analytical Iso-Executed-FLOP Control ($n=4$, $B=64$ / $B=32$) | Analytical executed FLOPs ($1.3384\text{e}17$ vs. $1.3387\text{e}17$ at $B=64$; $1.1946\text{e}17$ at $B=32$) | Checkpointed $12\text{L}$ growth ($728$ st, $+12.04\%$ recomp) vs. uncheckpointed $8\text{L}$ ($1{,}068$ st) | U0_exec_true: $+2.857 \pm 0.334\text{ mBPB}$ (8L leads overall; 12L leads bi_l by $-16.073$) |
| Axis 3 (Coarse-to-Fine C2F):<br>C2F_1, C2F_2_pure_filter, C2F_3_outer_gain, C2F_4_gain_form | Matched-Configuration Two-Stage Continuation ($n=4$, $g=190$) | Coarse $[6..9]$ Stage-1 setup, seed, steps ($366$), FLOPs ($2.87\text{e}16$); outer gain $\bar{\beta}_l$ in C2F_2_pure_filter | Retaining 2-tap Haar input filter (KeepFilter4) vs. OuterGainNoFilter4, ScaleFold4, Unfolded4 | C2F_2_pure_filter: $-17.187 \pm 1.370\text{ mBPB}$ ($p=0.00109$; $89.6\% \pm 3.6\%$ of C2F_1) |
Specifically, our three bounded empirical contributions and their scope limits are:
- Vertical FFN Capacity Allocation: Step-Apical vs. OpenELM Smooth Linear Ramp, Transfer, and Exploratory Sub-Layer Profiles (§3): Rather than claiming a new architectural principle beyond DeLighT [11] and OpenELM [10], we test how vertical FFN distribution behaves under fixed-FLOP pretraining across backbones and widths, and compare step-apical
Static-16L-Pyr40($r = (2^{14}, 6, 6)$) against a budget-matched smooth linear ramp (Static-16L-Linear40, $r = (1^4, 2^4, 3^4, 4^4)$, $\sum r_l = 40$). At 16 layers under exact iso-parameters ($79.69\text{M}$), iso-FLOPs/token ($377.49\text{M}$), and iso-exposure ($316.67\text{M}$ tokens), step-apical placement beats basal placement ($r = (6, 6, 2^{14})$) by $-3.799 \pm 0.201\text{ mBPB}$ onSlice A($p = 0.000324$;I1), and this advantage persists on a homogeneous all-globalLLLLattention backbone ($-3.716 \pm 0.497\text{ mBPB}$ onSlice B, $p = 0.00494$;T1) and at reduced width $d = 384, H = 3$ ($-3.143 \pm 0.594\text{ mBPB}$, $p = 0.0132$;T2). The smooth linear OpenELM ramp beats basal placement by $-4.328 \pm 0.268\text{ mBPB}$ ($p = 0.000516$;I1b) and uniform $2d$ (Static-16L-Unif2d) by $-1.656 \pm 0.337\text{ mBPB}$ ($p = 0.0161$;I1d), while performing comparably toPyr40($-0.528 \pm 0.368\text{ mBPB}$, $p = 0.246$, $3/4$ seeds negative;I1c). At 8 layers (Static-8L-Linear32vs.Static-8L-Unif4d), linear ramping is near-neutral ($-0.400 \pm 0.324\text{ mBPB}$, $p = 0.304$, $3/4$ seeds negative;U0_s8_linear). We also report uncalibrated sub-layer Logit-Lens andc_projSVD stable-rank profiles (Table 1C) strictly as exploratory descriptive associations, noting that uncalibrated readout does not isolate causal downstream feature utility [Belrose et al., 2023 [12]]. - Native Unsplit
4x64(B=64) Depth Splicing: 8-Seed Shared-Checkpoint vs. Two-Stage Contrasts and Analytical Iso-Executed-FLOP Accounting (§4): Under native 64-sequence microbatching ($524{,}288$ tok/step), selective activation checkpointing of the 6 interior blocks $2 \le l < 8$ at $12\text{L}$ adds $+15.76\%$ analytical executed FLOPs to static $12\text{L}$ ($1.3840 \times 10^{17}$) and $+12.04\%$ to grown $12\text{L}$ ($1.3384 \times 10^{17}$ vs. $1.1946 \times 10^{17}$ nominal FLOPs). Across 8 paired seeds (42, 137, 271, 314, 420, 512, 618, 729) with 6-test Holm-Bonferroni correction (p_Holm6) onSlice B:- On the true shared-checkpoint fork from bit-identical
Unif4dStage-1 checkpoints at $g = 225$ with destination global-attention depths held fixed at $\{3, 7, 11\}$, interior period-preserving growth (G_stack-Unif4d-Medial4, $[0..5,2..5,6,7]$, splice $5\to 2$) halves the same-batch transition shock relative to contiguous apicalG_stack(G_stack-Unif4d-Apical, $[0..7,4..7]$, splice $7\to 4$: $+0.0688$ vs. $+0.1286\text{ nats}$), reduces cross-seedSlice BSD from $2.59$ to $0.99\text{ mBPB}$, and improvesSlice BBPB by $-2.384 \pm 0.773\text{ mBPB}$ ($p = 0.0177, p_{\text{Holm6}} = 0.0530$, $7/8$ seeds negative;U6_true_8s). In two-stage package comparisons (wherePyrTarget-Medial4uses the pyramidal $r=(2^2,4^4,6^2)$ Stage-1 trunk),G_stack-PyrTarget-Medial4beatsG_stack-Unif4d-Apicalby $-3.317 \pm 0.661\text{ mBPB}$ ($p = 0.00153, \mathbf{p_{\text{Holm6}} = 0.00614}$, $8/8$ seeds negative;U7_true_8s) andG_stack-Unif4d-Medial4across both stages by $-0.933 \pm 0.386\text{ mBPB}$ ($p = 0.0462, p_{\text{Holm6}} = 0.0923$, $6/8$ seeds negative;U8_true_8s). Thus, under 6-test Holm adjustment, the combined packageU7_true_8sand the nominal-FLOP static-8L comparisonsU0_true_8s($-4.395 \pm 0.368\text{ mBPB}$, $p_{\text{Holm6}} = 0.000040$) andU0b_true_8s($-3.461 \pm 0.328\text{ mBPB}$, $p_{\text{Holm6}} = 0.000075$) are family-wise significant at $\alpha = 0.05$, whileU6_true_8s($p_{\text{Holm6}} = 0.0530$) andU8_true_8s($p_{\text{Holm6}} = 0.0923$) are supportive unadjusted contrasts that sit marginally above the 6-test Holm threshold. - When uncheckpointed
Static-8L-Unif4d-True4x64is granted the same $+12.04\%$ analytical executed FLOPs by running for 1,068 steps (Static-8L-Unif4d-m1068-True4x64, $1.3387 \times 10^{17}$ FLOPs), the extra 115 steps reduce 8LSlice BBPB by $-7.058 \pm 0.043\text{ mBPB}$ (U0_s8_extra), soStatic-8L-m1068leads checkpointed 728-stepG_stack-PyrTarget-Medial4overall by $+2.857 \pm 0.334\text{ mBPB}$ (U0_exec_true, $p = 0.00336$) while trailing by $-16.073\text{ mBPB}$ on long-distance bigrams (bi_l). Under uncheckpointedSplit8x32($0\%$ recomputation), 12L growth beats static 8L at exact analytical iso-executed FLOPs (-4.182 mBPB).
- On the true shared-checkpoint fork from bit-identical
- Matched-Configuration 4-Seed Continuation Test of Filter Retention in Coarse-to-Fine Transitions (§5): Across 4 seeds on coarse blocks $[6..9]$ at $g = 190$ ($366$ steps), comparing
C2F-MG12-KeepFilter4-g190(which retains both the 2-tap Haar input filter and outer block gain $\bar{\beta}_l$) againstC2F-MG12-OuterGainNoFilter4-g190(which preserves the exact outer block gain $x + \bar{\beta}_l[A(x) + M(x + A(x))]$ while dropping only the 2-tap input filter) yields $-17.187 \pm 1.370\text{ mBPB}$ onsh9A($p = 0.00109$, $4/4$ seeds negative;C2F_2_pure_filter). Conditional on the retained-gain setting, the ratio of estimated mean contrastsC2F_2_pure_filter / C2F_1(-17.187 / -19.188 mBPB) is $89.6\% \pm 3.6\%$ (jackknife SE; per-seed range $84.2\%\text{--}99.5\%$), showing that the transition recovery is dominated by retaining the 2-tap causal input filter rather than outer vs.c_proj-folded residual scaling (C2F_4_gain_form: $-0.600 \pm 0.308\text{ mBPB}$, $p = 0.147$).
2. Architecture, Nominal vs. Executed FLOP Accounting, and Evaluation Protocol
2.1 Transformer Backbone and Depth-Expansion Topologies
Every baseline and growth model is a decoder-only causal Transformer operating at context length $L = 2{,}048$, vocabulary size $V = 8{,}192$, default width $d = 512$ ($H = H_{kv} = 4$ heads, $d_h = 128$; with $d = 384, H = 3, d_h = 128$ in transfer test T2), RoPE ($\theta = 10{,}000$), and parameter-free RMSNorm (F.rms_norm).
- Two-Matrix Squared-ReLU MLP: Layer $l$ with expansion ratio $r_l$ computes $\operatorname{MLP}_l(u) = W_{\mathrm{proj},l}(\max(0, W_{\mathrm{fc},l} u))^2$ with bias-free $W_{\mathrm{fc},l} \in \mathbb{R}^{(r_l d)\times d}$ and $W_{\mathrm{proj},l} \in \mathbb{R}^{d\times (r_l d)}$ ($2 r_l d^2$ parameters).
- Attention Window Patterns (
SSSL,LLLL, andmedial_winlock): Under defaultSSSLpositional assignment, layers $l \equiv 3 \pmod 4$ (and $l = L_{\text{net}}-1$) use full causal context $w_l = 2{,}048$ (L) and all other layers use sliding window $w_l = 1{,}024$ (S). Under--window-pattern LLLL(transfer testT1), all layers use $w_l = 2{,}048$. Under--topology medial_winlock(experiments_p7/train_loop_grow.py, lines 1244–1250), destination block $d \in \{0,\dots,11\}$ inherits its parent's Stage-1 window $w_{\text{dst}}[d] = w_{\text{src}}[\text{block\_map}[d]]$, giving[S,S,S,L, S,L,S,S, S,S,S,L]($3\text{L} + 9\text{S}$). - Value-Embedding Gates and Residual Scalars: Odd layers $l \in \{1, 3, \dots\}$ include a token-embedding table
value_embeds($V \times d$) gated byve_gate$\in \mathbb{R}^{H_{kv}\times 32}$ ($128$ parameters per odd layer; $96$ when $H_{kv}=3$): $v_l \leftarrow v_l + 2\,\sigma(W_{\mathrm{ve},l} x_{:,:,:32}) \odot \operatorname{VE}_l(\text{idx})$. Pre-block inputs are scaled by learnable per-layer scalars $\tilde{x}_l = \lambda_{\mathrm{resid},l} x_l + \lambda_{x0,l} x_0$ (resid_lambdasinit $1.0$,x0_lambdasinit $0.1$). Across all 242 runs,wd_scale = 1.0and Muon matrix weight decay decays linearly as $\text{wd}(\text{progress}) = 0.2(1 - \text{progress})$.
Table 1A summarizes the exact block_map, destination global-attention (L) depths, window flips relative to parent blocks, non-native residual splice interfaces, and FFN ratio schedules in expand_model_generalized(). Note the exact window-flip count on medial: mapping parent [S_0,S_1,S_2,L_3,S_4,S_5,S_6,L_7] through [0,1,2,3,2,3,4,5,4,5,6,7] yields parent windows [S,S,S,L, S,L,S,S, S,S,S,L], whereas positional SSSL assigns [S,S,S,L, S,S,S,L, S,S,S,L]—flipping parent window types at exactly 2 destination blocks ($d = 5: L_3 \to S_5$ and $d = 7: S_5 \to L_7$).
Table 1A: Implemented Depth-Expansion Topologies and FFN Profiles (expand_model_generalized)
| Topology Flag | Transition | Source Block Map block_map[0 .. n_dst-1] | Stage-2 L ($2048$) Depths & Window Flips vs. Parent | Non-Native Splice Interfaces $(u \to v)$ | FFN Profile $r_{\text{src}} \to r_{\text{dst}}$ |
| :--- | :---: | :--- | :--- | :--- | :--- |
| medial4 | $8\to 12\text{L}$ | [0,1,2,3,4,5, 2,3,4,5, 6,7] | L at $\{3, 7, 11\}$; $0$ flips | $1$ interior splice ($5 \to 2$ at $5\to 6$) | PyrTarget: $(2^2,4^4,6^2)\to(2^2,4^8,6^2)$<br>Unif4d: $4^8 \to 4^{12}$ |
| medial (Positional) | $8\to 12\text{L}$ | [0,1,2,3, 2,3, 4,5, 4,5, 6,7] | L at $\{3, 7, 11\}$; $2$ blocks flip ($d=5: L_3\to S_5$; $d=7: S_5\to L_7$) | $2$ interior splices ($3 \to 2$ at $3\to 4$; $5 \to 4$ at $7\to 8$) | PyrTarget: $(2^2,4^4,6^2)\to(2^2,4^8,6^2)$<br>Unif4d: $4^8 \to 4^{12}$ |
| medial_winlock | $8\to 12\text{L}$ | [0,1,2,3, 2,3, 4,5, 4,5, 6,7] | L at $\{3, 5, 11\}$; $0$ flips ($w_{\text{dst}}[d]=w_{\text{src}}[\text{map}[d]]$) | $2$ interior splices ($3 \to 2$ at $3\to 4$; $5 \to 4$ at $7\to 8$) | PyrTarget: $(2^2,4^4,6^2)\to(2^2,4^8,6^2)$ |
| apical (G_stack) | $8\to 12\text{L}$ | [0,1,2,3,4,5,6,7, 4,5,6,7] | L at $\{3, 7, 11\}$; $0$ flips | $1$ apical splice ($7 \to 4$ at $7\to 8$) | Unif4d: $4^8 \to 4^{12}$<br>Pyr-Apical2: $(2^6,4^2)\to 4^{12}$ |
| medial4 (16L 2/3-stg) | $8\to 16\text{L}$ / $8\to 12\to 16\text{L}$ | 2-stg: [0..5, 2..5, 2..5, 6,7]<br>3-stg S3: [0..9, 6..9, 10,11] | L at $\{3, 7, 11, 15\}$; $0$ flips | 2-stg: $5\to 2$ ($\times 2$)<br>3-stg: $5\to 2$, then $9\to 6$ | $(2^6,6^2)\to(2^{14},6^2)$ /<br>$(2^6,6^2)\to(2^{10},6^2)\to(2^{14},6^2)$ |
2.2 Analytical Nominal Model FLOPs vs. Analytical Executed Activation-Checkpointing FLOPs and Batch Regimes
- Analytical Nominal Forward+Backward FLOPs (
GPT.estimate_flops): Excluding embedding table lookups and $2 L_{\text{net}}$ residual scalars, analytical nominal matrix+attention FLOPs per token for dense ($s_l = 1$) models are: $$F_{\text{nom}}\!\left(L_{\text{net}}, \{r_l\}\right) = 6 \left(V d + \sum_{l=0}^{L_{\text{net}}-1} \left[(4 + 2 r_l) d^2 + 32 H_{kv}\,\mathbb{I}[l \text{ odd}]\right]\right) + \sum_{l=0}^{L_{\text{net}}-1} 12 H d_h w_l$$ At $d=512, H=4$ (SSSL), this gives $239{,}078{,}400\text{ FLOPs/tok}$ for 8L ($\sum r_l = 32$, $50.33\text{M}$ params, identical forStatic-8L-Unif4dandStatic-8L-Linear32), $346{,}034{,}688\text{ FLOPs/tok}$ for 12L ($\sum r_l = 48$, $71.30\text{M}$ params, identical for positionalSSSLandmedial_winlock), $352{,}327{,}680\text{ FLOPs/tok}$ forStatic-16L-Unif2d($\sum r_l = 32$, $75.50\text{M}$ params), and $377{,}493{,}504\text{ FLOPs/tok}$ for 16LPyr40/Basal40/Linear40($\sum r_l = 40$, $79.69\text{M}$ params). In the transfer arms, 16L all-globalLLLL($w_l = 2{,}048\;\forall l$, $d=512, H=4$) uses $452{,}990{,}976\text{ FLOPs/tok}$ ($79.69\text{M}$ params), and 16L reduced-widthd384($d=384, H=3$,SSSL) uses $240{,}652{,}032\text{ FLOPs/tok}$ ($52.69\text{M}$ params). Full-budget nominal schedules target $953$ steps of 8L at $524{,}288$ tok/step: $\mathcal{B}_{\text{nom}} = 499{,}646{,}464 \times 239{,}078{,}400 = \mathbf{1.194547 \times 10^{17}}\text{ analytical nominal FLOPs}$ (matched within $0.1\%$ nominal across full-budgetSSSLarms). - Exact Activation-Checkpointing Specification (
2 <= l < 8) and Analytical Executed FLOP Accounting:- In
experiments_p5/train_pyramidal.pyandexperiments_revival/train_revival.py(use_half_mb = True),262k-tok/step runs execute $4\times 32$ split microbatches (4x32) and524k-tok/step runs execute $8\times 32$ split microbatches (Split8x32), with zero activation checkpointing (grad_ckpt_12l = False), so analytical executed FLOPs equal analytical nominal FLOPs across all Phase-5, Phase-6,Split8x32,Static-8L-Unif4d-True4x64,Static-8L-Linear32-True4x64,Static-8L-Unif4d-m1068-True4x64,Static-16L-Linear40,LLLL, andd384runs. - In
experiments_p7/train_loop_grow.py(line 801), native unsplitTrue4x64(device_batch_size = 64,microbatch_sequences = 64,4x64$= 524{,}288$ tok/step) fits 12-layer training within $19.0\text{--}22.8\text{ GB}$ VRAM via--grad-ckpt-12l, which wrapstorch.utils.checkpoint.checkpoint(..., use_reentrant=False)around the 6 interior blocks $2 \le l < 8$ wheneverlen(transformer.h) >= 12(if use_ckpt and (len(self.transformer.h) >= 12) and (2 <= i < 8)). - For all 12-layer models tested under
True4x64, blocks $2 \le l < 8$ ($l \in \{2,3,4,5,6,7\}$) have $r_l = 4$ and contain $4\text{S} + 2\text{L}$ windows (both under positionalSSSL, whereLis at $\{3, 7\}$, and undermedial_winlock, whereLis at $\{3, 5\}$). Recomputing one forward pass (2 * numelmatrix FLOPs plus4 * H * d_h * w_lattention FLOPs) across those 6 blocks adds an analytical $54{,}525{,}952\text{ FLOPs/token}$ ($15.76\%$ of $346{,}034{,}688$):Static-8L-Unif4d-True4x64&Static-8L-Linear32-True4x64($953$ steps, uncheckpointed): Analytical Nominal $= \mathbf{1.1945 \times 10^{17}}$, Recomputation $= 0$, Analytical Executed $= 1.1945 \times 10^{17}$ FLOPs (train_t = 1090.8–1095.1s, peak VRAM $22{,}804.5\text{ MB}$).Static-8L-Unif4d-m1068-True4x64($1{,}068$ steps, uncheckpointed, $559.94\text{M}$ tokens): Analytical Nominal $= \mathbf{1.3387 \times 10^{17}}$, Recomputation $= 0$, Analytical Executed $= 1.3387 \times 10^{17}$ FLOPs (train_t = 1228.7s, matching the analytical executed FLOPs of 728-step checkpointed 12L growth within $0.02\%$).Static-12L-PyrTarget-True4x64($659$ checkpointed 12L steps): Analytical Nominal $= 1.1956 \times 10^{17}$, Recomputation $= +0.1884 \times 10^{17}$ ($+15.76\%$), Analytical Executed $= 1.3840 \times 10^{17}$ FLOPs (train_t = 1215.6s, peak VRAM $19{,}017.6\text{ MB}$).- All five 12L Grown
True4x64Arms ($225$ uncheckpointed 8L steps + $503$ checkpointed 12L steps): Analytical Nominal $= 1.1946 \times 10^{17}$, Recomputation $= +0.1438 \times 10^{17}$ ($+12.04\%$), Analytical Executed $= 1.3384 \times 10^{17}$ FLOPs (train_t = 1190.9–1198.7s). Thus all five grown 12LTrue4x64arms have strictly identical analytical nominal and executed FLOPs to one another, use $-3.29\%$ fewer analytical executed FLOPs thanStatic-12L-PyrTarget-True4x64, and are directly compared against both nominal-matchedStatic-8L-Unif4d-True4x64($953$ steps) and analytical-executed-matchedStatic-8L-Unif4d-m1068-True4x64($1{,}068$ steps).
- In
2.3 Evaluation Splits and Provenance
All runs exclude shard_00009.parquet from training (shard_00000..00008, $761{,}856$ documents). verify_shard9_provenance.py removes all $26$ training-overlapping documents, $2$ validation-overlapping documents, and $2$ internal duplicates from shard_00009.parquet, leaving $84{,}963$ strictly disjoint documents (shard9_provenance.json):
Slice A(sh9A, development-conditioned): First $320$ batches ($20{,}971{,}520$ target tokens; $\approx 20.95\text{M}$ normal tokens) packed fromkept_docs[0:37760](SHA-256: ef81f8c0b5dcdcbd...), used alongsideshard_06542val_bpbandholdout_bpb($20{,}971{,}520$ target tokens each) during Phase-5 development.Slice B(confirmation set): Second $320$ batches ($20{,}971{,}520$ target tokens; $\approx 20.95\text{M}$ normal tokens) packed fromkept_docs[37760:84963](SHA-256: 6dfb9ec04a23ba7b...), created after the Phase-5 architecture screen and evaluated on our confirmation and transfer arms. We also logconfirm_bpb($7{,}077{,}888$ target tokens, batches $641\text{--}748$ ofshard_06542) as a cross-slice check, noting it was previously used in rStation [3, 4]. Descriptive input–target bigram and unigram token-class decompositions are reported in Appendix A.
3. Vertical FFN Capacity Allocation: Step-Apical vs. OpenELM Smooth Linear Ramp, Transfer, and Exploratory Sub-Layer Profiles
Figure 1: Apical-Pyramidal Capacity Allocation, Cross-Backbone/Width Transfer, and Native 4x64 Depth Growth
DeLighT [11] and OpenELM [10, §2.1] established that ramping FFN intermediate width from narrower lower layers to wider upper layers improves parameter efficiency over uniform Transformers. Rather than claiming a new architectural principle, we evaluate how vertical FFN distribution behaves under a strict fixed-FLOP pretraining budget across two questions: (1) Does concentrating FFN width at the apex (Pyr40, $r = (2^{14}, 6, 6)$) versus the base (Basal40, $r = (6, 6, 2^{14})$) persist under homogeneous all-global attention (LLLL) and at reduced model width ($d = 384, H = 3$), and how does our two-step apical schedule compare against a budget-matched smooth linear OpenELM ramp (Static-16L-Linear40, $r = (1^4, 2^4, 3^4, 4^4)$, $\sum r_l = 40$, and 8-layer Static-8L-Linear32, $r = (2,2,3,3,5,5,6,6)$, $\sum r_l = 32$)? (2) What descriptive patterns appear in uncalibrated sub-layer Logit-Lens readouts and c_proj weight SVD stable ranks across depth quartiles? Table 1B reports the 4-seed arm summaries ($\text{Mean} \pm \text{SD}$) and paired contrasts ($\text{Mean }\Delta \pm \text{SE}$), and Table 1C reports exploratory sub-layer Logit-Lens and SVD stable-rank measurements on $524{,}288$ validation tokens.
Table 1B: Iso-Parameter / Iso-Exposure FFN Capacity Allocation, Smooth Linear OpenELM Ramp, and Cross-Backbone / Cross-Width Transfer (seeds = 42, 137, 271, 314)
| Arm ($\text{Mean} \pm \text{SD}$) or Paired Contrast ($\text{Mean }\Delta \pm \text{SE}$) | Backbone & Params | Steps & Tokens (Exec. FLOPs, 0% recomp) | Slice B BPB (or $\Delta\text{mBPB}$) | Slice A (sh9A) BPB (or $\Delta\text{mBPB}$) | val_bpb / holdout_bpb (or $\Delta\text{mBPB}$) | Paired 95% $t$-CI, $t$-stat ($df=3$), $p_{\text{raw}}$, & Seeds |
| :--- | :---: | :---: | :---: | :---: | :---: | :--- |
| Static-16L-Linear40 ($r=1^4,2^4,3^4,4^4$, OpenELM ramp) | $d=512, H=4$ SSSL<br>$79.69\text{M}$ | $1{,}208$ steps<br>$316.67\text{M}$ ($1.195\text{e}17$) | — | $0.987598 \pm 0.001086$ | $0.987008 \pm 0.001136$ /<br>$0.986120 \pm 0.001101$ | Train $1124.7\text{s}$, VRAM $18{,}940\text{ MB}$ |
| Static-16L-Pyr40 ($r=2^{14},6,6$, step-apical) | $d=512, H=4$ SSSL<br>$79.69\text{M}$ | $1{,}208$ steps<br>$316.67\text{M}$ ($1.195\text{e}17$) | — (see Basal40) | $0.988126 \pm 0.000833$ | $0.987565 \pm 0.000750$ /<br>$0.986664 \pm 0.000807$ | Train $1122.0\text{s}$, VRAM $18{,}940\text{ MB}$ |
| Static-16L-Basal40 ($r=6,6,2^{14}$, step-basal) | $d=512, H=4$ SSSL<br>$79.69\text{M}$ | $1{,}208$ steps<br>$316.67\text{M}$ ($1.195\text{e}17$) | $0.991478 \pm 0.000693$ | $0.991926 \pm 0.000642$ | $0.991377 \pm 0.000668$ /<br>$0.990533 \pm 0.000696$ | Train $1118.5\text{s}$, VRAM $18{,}940\text{ MB}$ |
| Static-16L-Pyr40-LLLL ($r=2^{14},6,6$) | $d=512, H=4$ LLLL<br>$79.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($1.344\text{e}17$) | $0.992141 \pm 0.001064$ | $0.992608 \pm 0.001067$ | $0.992098 \pm 0.001064$ /<br>$0.991136 \pm 0.001078$ | Train $1082.1\text{s}$, VRAM $18{,}940\text{ MB}$ |
| Static-16L-Basal40-LLLL ($r=6,6,2^{14}$) | $d=512, H=4$ LLLL<br>$79.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($1.344\text{e}17$) | $0.995858 \pm 0.000524$ | $0.996294 \pm 0.000502$ | $0.995760 \pm 0.000492$ /<br>$0.994814 \pm 0.000545$ | Train $1082.1\text{s}$, VRAM $18{,}940\text{ MB}$ |
| Static-16L-Pyr40-d384 ($r=2^{14},6,6$) | $d=384, H=3$ SSSL<br>$52.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($0.714\text{e}17$) | $1.020963 \pm 0.001760$ | $1.021444 \pm 0.001765$ | $1.020836 \pm 0.001728$ /<br>$1.020104 \pm 0.001780$ | Train $725.0\text{s}$, VRAM $14{,}375\text{ MB}$ |
| Static-16L-Basal40-d384 ($r=6,6,2^{14}$) | $d=384, H=3$ SSSL<br>$52.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($0.714\text{e}17$) | $1.024106 \pm 0.001700$ | $1.024505 \pm 0.001713$ | $1.023913 \pm 0.001663$ /<br>$1.023140 \pm 0.001776$ | Train $725.2\text{s}$, VRAM $14{,}375\text{ MB}$ |
| Static-8L-Linear32-True4x64 ($r=2,2,3,3,5,5,6,6$) | $d=512, H=4$ SSSL<br>$50.33\text{M}$ ($B=64$) | $953$ steps<br>$499.65\text{M}$ ($1.195\text{e}17$) | $0.995007 \pm 0.001536$ | $0.995563 \pm 0.001541$ | $0.994894 \pm 0.001522$ /<br>$0.994121 \pm 0.001534$ | Train $1095.1\text{s}$, VRAM $22{,}805\text{ MB}$ |
| I1: Pyr40 vs. Basal40 (SSSL, $d=512$) | Exact Iso-Param & Iso-Tok | $316.67\text{M}$ tok | — | $-3.799 \pm 0.201$ | $-3.812$ / $-3.869$ | CI $[-4.440, -3.159]$, $t = -18.89$, $p = 0.000324$ ($4/4$)<br>[-3.705, -4.141, -3.270, -4.082] |
| I1b: Linear40 vs. Basal40 (SSSL, $d=512$) | Exact Iso-Param & Iso-Tok | $316.67\text{M}$ tok | — | $-4.328 \pm 0.268$ | $-4.370$ / $-4.413$ | CI $[-5.181, -3.475]$, $t = -16.15$, $p = 0.000516$ ($4/4$)<br>[-5.109, -4.187, -4.124, -3.891] |
| I1c: Linear40 vs. Pyr40 (SSSL, $d=512$) | Exact Iso-Param & Iso-Tok | $316.67\text{M}$ tok | — | $-0.528 \pm 0.368$ | $-0.557$ / $-0.544$ | CI $[-1.699, +0.642]$, $t = -1.44$, $p = 0.246$ ($3/4$)<br>[-1.404, -0.047, -0.854, +0.191] |
| I1d: Linear40 vs. Unif2d (SSSL, $d=512$) | $79.69\text{M}$ vs. $75.50\text{M}$ | $316.7\text{M}$ vs. $339.2\text{M}$ | — | $-1.656 \pm 0.337$ | $-1.639$ / $-1.679$ | CI $[-2.728, -0.583]$, $t = -4.91$, $p = 0.0161$ ($4/4$)<br>[-1.864, -1.075, -2.517, -1.166] |
| T1: Pyr40-LLLL vs. Basal40-LLLL (All-Global) | Exact Iso-Param & Iso-Tok | $296.75\text{M}$ tok | $-3.716 \pm 0.497$ | $-3.686$ ($p=0.00557$) | $-3.661$ / $-3.678$ | Slice B CI $[-5.297, -2.136]$, $t = -7.48$, $p = 0.00494$ ($4/4$)<br>[-3.975, -4.538, -2.272, -4.080] |
| T2: Pyr40-d384 vs. Basal40-d384 ($d=384, H=3$) | Exact Iso-Param & Iso-Tok | $296.75\text{M}$ tok | $-3.143 \pm 0.594$ | $-3.061$ ($p=0.0159$) | $-3.077$ / $-3.037$ | Slice B CI $[-5.033, -1.253]$, $t = -5.29$, $p = 0.0132$ ($4/4$)<br>[-4.810, -2.448, -3.156, -2.158] |
| I2: Static-16L-Basal40 vs. Static-16L-Unif2d | $79.69\text{M}$ vs. $75.50\text{M}$ | $316.7\text{M}$ vs. $339.2\text{M}$ | — | $+2.672 \pm 0.372$ | $+2.731$ / $+2.734$ | CI $[+1.489, +3.855]$, $t = +7.19$, $p = 0.00555$ ($0/4$)<br>[+3.245, +3.112, +1.607, +2.726] |
| I3 / I4: 12L PyrTarget vs. Unif4d (Static / Grown) | Exact Iso-Param ($71.30\text{M}$) | $345.5\text{M}$ / $381.7\text{M}$ | — | $-1.572 \pm 0.839$ /<br>$-0.532 \pm 0.115$ | $-1.567$ / $-0.510$ (val) | I3: $t = -1.87, p = 0.158$; I4: $t = -4.64$, $p = 0.0188$ |
| U0_s8_linear: Static-8L-Linear32 vs. Static-8L-Unif4d | Exact Iso-Param ($50.33\text{M}$) | $499.65\text{M}$ tok ($B=64$) | $-0.400 \pm 0.324$ | $-0.424$ ($p=0.290$) | $-0.439$ / $-0.442$ | Slice B CI $[-1.430, +0.630]$, $t = -1.24, p = 0.304$ ($3/4$)<br>[-0.826, -0.565, -0.766, +0.557] |
Table 1C: Exploratory Uncalibrated Sub-Layer Logit-Lens BPB Change ($\Delta\text{BPB} = \text{BPB}_{\text{in}} - \text{BPB}_{\text{out}}$) & c_proj Weight SVD Stable Rank on $524{,}288$ Validation Tokens (seed = 42, matches paper5_tables_verified.txt lines 177–182 verbatim)
| Model & Layer-Width Schedule | Quartile Q1 ($L_{0..3}$ or $L_{0..1}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | Quartile Q2 ($L_{4..7}$ or $L_{2..3}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | Quartile Q3 ($L_{8..11}$ or $L_{4..5}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | Quartile Q4 ($L_{12..15}$ or $L_{6..7}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) |
| :--- | :---: | :---: | :---: | :---: |
| Static-16L-Linear40 ($1^4,2^4,3^4,4^4$; $\text{emb}=2.6297$) | $r_{\text{sum}}=4$: $-0.0745$ ($-0.0186/r$), Attn $+0.0926$, $\text{SR}=57.3$ | $r_{\text{sum}}=8$: $+0.1781$ ($+0.0223/r$), Attn $+0.2843$, $\text{SR}=85.2$ | $r_{\text{sum}}=12$: $+0.2199$ ($+0.0183/r$), Attn $+0.4537$, $\text{SR}=93.0$ | $r_{\text{sum}}=16$: $+0.2824$ ($+0.0177/r$), Attn $+0.2322$, $\text{SR}=137.1$ |
| Static-16L-Pyr40 ($2^{14},6,6$; $\text{emb}=2.6344$) | $r_{\text{sum}}=8$: $-0.0916$ ($-0.0115/r$), Attn $+0.2040$, $\text{SR}=84.0$ | $r_{\text{sum}}=8$: $+0.1496$ ($+0.0187/r$), Attn $+0.3979$, $\text{SR}=74.5$ | $r_{\text{sum}}=8$: $+0.1519$ ($+0.0190/r$), Attn $+0.3404$, $\text{SR}=88.1$ | $r_{\text{sum}}=16$: $+0.2799$ ($+0.0175/r$), Attn $+0.2399$, $\text{SR}=119.6$ |
| Static-16L-Unif2d ($2^{16}$; $\text{emb}=2.6609$) | $r_{\text{sum}}=8$: $-0.0637$ ($-0.0080/r$), Attn $+0.1464$, $\text{SR}=83.4$ | $r_{\text{sum}}=8$: $+0.2147$ ($+0.0268/r$), Attn $+0.3069$, $\text{SR}=75.8$ | $r_{\text{sum}}=8$: $+0.2074$ ($+0.0259/r$), Attn $+0.3830$, $\text{SR}=103.5$ | $r_{\text{sum}}=8$: $+0.2703$ ($+0.0338/r$), Attn $+0.2329$, $\text{SR}=109.9$ |
| Static-16L-Basal40 ($6,6,2^{14}$; $\text{emb}=2.6379$) | $r_{\text{sum}}=16$: $+0.0108$ ($+0.0007/r$), Attn $+0.0390$, $\text{SR}=106.9$ | $r_{\text{sum}}=8$: $+0.1961$ ($+0.0245/r$), Attn $+0.4696$, $\text{SR}=74.9$ | $r_{\text{sum}}=8$: $+0.2200$ ($+0.0275/r$), Attn $+0.2312$, $\text{SR}=106.9$ | $r_{\text{sum}}=8$: $+0.2487$ ($+0.0311/r$), Attn $+0.2563$, $\text{SR}=118.5$ |
| Static-8L-Linear32 ($2,2,3,3,5,5,6,6$; $\text{emb}=2.5362$) | $L_{0..1}$ ($r_{\text{sum}}=4$): $+0.0493$ ($+0.0123/r$), Attn $+0.0224$, $\text{SR}=82.7$ | $L_{2..3}$ ($r_{\text{sum}}=6$): $+0.1886$ ($+0.0314/r$), Attn $+0.4510$, $\text{SR}=82.6$ | $L_{4..5}$ ($r_{\text{sum}}=10$): $+0.2636$ ($+0.0264/r$), Attn $+0.1310$, $\text{SR}=101.2$ | $L_{6..7}$ ($r_{\text{sum}}=12$): $+0.2510$ ($+0.0209/r$), Attn $+0.2109$, $\text{SR}=129.1$ |
| Static-8L-Unif4d ($4^8$; $\text{emb}=2.5358$) | $L_{0..1}$ ($r_{\text{sum}}=8$): $+0.0267$ ($+0.0033/r$), Attn $+0.0495$, $\text{SR}=105.1$ | $L_{2..3}$ ($r_{\text{sum}}=8$): $+0.2443$ ($+0.0305/r$), Attn $+0.3056$, $\text{SR}=108.2$ | $L_{4..5}$ ($r_{\text{sum}}=8$): $+0.2871$ ($+0.0359/r$), Attn $+0.2147$, $\text{SR}=100.5$ | $L_{6..7}$ ($r_{\text{sum}}=8$): $+0.2276$ ($+0.0285/r$), Attn $+0.2111$, $\text{SR}=120.7$ |
Key Findings on Vertical Capacity Allocation and Exploratory Sub-Layer Profiles:
- Smooth Linear OpenELM Ramp vs. Step-Apical Placement (
I1–I1d,T1,T2,U0_s8_linear): Consistent with DeLighT [11] and OpenELM [10], allocating FFN width toward upper layers at 16L substantially outperforms both basal-heavy (I1: $-3.799 \pm 0.201\text{ mBPB}$;I1b: $-4.328 \pm 0.268\text{ mBPB}$) and uniform-width (C1: $-1.127 \pm 0.257\text{ mBPB}$;I1d: $-1.656 \pm 0.337\text{ mBPB}$) schedules under fixed total FLOPs. The smooth linear rampStatic-16L-Linear40($r=(1^4,2^4,3^4,4^4)$) and step-apicalStatic-16L-Pyr40($r=(2^{14},6,6)$) perform comparably (I1c = -0.528 ± 0.368 mBPB, $p = 0.246$, $3/4$ seeds negative), whileT1(-3.716 ± 0.497 mBPBon all-globalLLLL, $p = 0.00494$) andT2(-3.143 ± 0.594 mBPBat $d=384, H=3$, $p = 0.0132$) demonstrate that the apical-over-basal advantage persists when sliding-window periodicity is removed and at reduced model width $d = 384$. By contrast, at 8 layers (Static-8L-Linear32vs.Static-8L-Unif4d), linear FFN ramping yields only $-0.400 \pm 0.324\text{ mBPB}$ (U0_s8_linear, $p = 0.304$, $3/4$ seeds negative). - Exploratory Sub-Layer Logit-Lens & SVD Stable-Rank Associations (Table 1C): As a descriptive diagnostic, Table 1C evaluates each sub-layer's immediate uncalibrated Logit-Lens BPB change (applying final RMSNorm and
lm_headdirectly to intermediate residual states) alongside the SVD stable rank ($\|W\|_F^2 / \|W\|_2^2$) ofmlp.c_proj. In 16-layer models, quartileQ1($L_{0..3}$) MLPs exhibit negative uncalibrated Logit-Lens changes ($\Delta\text{BPB}_{\text{MLP}} = -0.0637$ to $-0.0916\text{ BPB}$ inUnif2d,Linear40, andPyr40, and $+0.0108\text{ BPB}$ inBasal40) and lowerc_projstable ranks ($\text{SR} = 57.3\text{--}84.0$ vs. $109.9\text{--}137.1$ inQ4), whereas in 8-layer modelsQ1($L_{0..1}$) andQ2($L_{2..3}$) already show positive uncalibrated MLP reductions ($+0.0267$ and $+0.2443\text{ BPB}$ inStatic-8L-Unif4d). However, as established in the representation-probing literature [Belrose et al., 2023 [12]], intermediate residual states are not aligned with the final unembedding basis, and early MLPs can worsen immediate uncalibrated readout while writing features used by subsequent layers; similarly, weight stable rank depends on assigned layer width $r_l d$. Accordingly, we treat Table 1C strictly as an exploratory descriptive association accompanying the controlled fixed-FLOP benchmark results in Table 1B rather than a causal proof of wasted basal capacity.
4. Native Unsplit 4x64 Depth Splicing: 8-Seed Shared-Checkpoint vs. Two-Stage Contrasts, Window-Locked Controls, and Analytical Iso-Executed-FLOP Accounting
Figure 2: Bigram Compositional Frontier and Checkpoint-Forked Splicing Controls
4.1 Native Unsplit 4x64 (B=64) Arms, Shared-Checkpoint vs. Two-Stage Design, and Iso-Executed-FLOP Control
To evaluate depth growth on the exact native 64-sequence microbatch specification (B = 64 unsplit, $524{,}288$ tok/step) with explicit analytical nominal and executed FLOP accounting (§2.2), we evaluate nine True4x64 arms (48 runs in experiments_p7/runs/ plus 4 runs in experiments_revival/runs/, Tables 2A and 2B), expanding our four primary comparison arms (Static-8L-Unif4d, G_stack-Unif4d-Apical, G_stack-Unif4d-Medial4, and G_stack-PyrTarget-Medial4) to 8 paired seeds (42, 137, 271, 314, 420, 512, 618, 729) and adding a 4-seed analytical iso-executed-FLOP static 8L baseline (Static-8L-Unif4d-m1068-True4x64, $1{,}068$ steps, $1.3387 \times 10^{17}$ FLOPs). We explicitly distinguish three comparison types in Table 2B:
- 8-Seed True Shared-Checkpoint Fork at Fixed
{3,7,11}Global Depths (U6_true_8s:Unif4d-Medial4vs.Unif4d-Apical): On all 8 seeds,G_stack-Unif4d-Medial4-True4x64($[0..5,2..5,6,7]$, interior splice $5\to 2$) andG_stack-Unif4d-Apical-True4x64($[0..7,4..7]$, apical splice $7\to 4$) fork from bit-identical saved 8LUnif4dStage-1 checkpoints at $g = 225$ (stage1_ckpt_sha256andloss_bf16_beforeverified identical for every seed inpaper5_tables_verified.txt, lines 79–87; e.g., at seed 42,loss_bf16_before = 3.404801). Both arms hold FFN widths uniform ($4^{12}$), splice count fixed at $1$, window flips fixed at $0$, and destination global-attention depths strictly fixed at $\{3, 7, 11\}$, isolating interior ($[2..5]$) versus apical ($[4..7]$) contiguous 4-layer duplication. - 8-Seed Two-Stage Growth Package Contrasts (
U7_true_8s,U8_true_8s,U0_true_8s,U0b_true_8s,U0d_true_8s): BecauseG_stack-PyrTarget-Medial4-True4x64trains a pyramidal $r = (2,2,4,4,4,4,6,6)$ 8L trunk during Stage 1 ($0..225$; at seed 42,loss_bf16_before = 3.399315) before expanding to $r = (2^2, 4^8, 6^2)$,U7_true_8s(PyrTarget-Medial4vs.Unif4d-Apical) compares two-stage growth packages (combining pyramidal Stage-1+Stage-2 FFN allocation with interiorMedial4splicing against uniform apicalG_stack), andU8_true_8s(PyrTarget-Medial4vs.Unif4d-Medial4) varies FFN allocation across both stages underMedial4rather than forking from a shared Stage-1 checkpoint. - 4-Seed True Shared-Checkpoint 3-Way
PyrTargetFork (U2_true,U4_true,U5_true): On seeds42, 137, 271, 314,Medial4($[0..5,2..5,6,7]$, $1$ splice $5\to 2$, $0$ window flips,Lat $\{3,7,11\}$),Medial-Positional($[0..3,2..5,4..7]$, $2$ splices $3\to 2, 5\to 4$, $2$ window flips at $d \in \{5,7\}$,Lat $\{3,7,11\}$), andMedial-WinLock($[0..3,2..5,4..7]$, $2$ splices $3\to 2, 5\to 4$, $0$ window flips,Lat $\{3,5,11\}$) fork from bit-identical saved 8LPyrTargetStage-1 checkpoints at $g = 225$ (paper5_tables_verified.txt, lines 73–77).
Table 2A: Native Unsplit 4x64 (B=64, $524{,}288$ tok/step) Arm Summaries (Mean ± SD across $n=8$ or $n=4$ seeds, matches paper5_tables_verified.txt lines 59–71)
| True4x64 Arm (mb_seq = 64, eff_half_mb = false) | $n$ | Steps ($g$) & Tokens | Analytical Nom / Exec FLOPs (ckpt 2..7) | Confirmation Slice B BPB ($\text{Mean} \pm \text{SD}$) | confirm_bpb ($\text{Mean} \pm \text{SD}$) | Slice A (sh9A) BPB ($\text{Mean} \pm \text{SD}$) | val_bpb ($\text{Mean} \pm \text{SD}$) | Same-Batch $\Delta\mathcal{L}$ (nats) | Train / Wall (s) & VRAM |
| :--- | :-: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Static-8L-Unif4d-True4x64 ($r=4^8$, 8-seed) | 8 | $953$ ($-$)<br>$499.65\text{M}$ | $1.1945\text{e}17$ /<br>$1.1945\text{e}17$ ($0\%$) | $0.995310 \pm 0.001394$<br>(4s: $0.995407$) | $0.997278 \pm 0.001354$ | $0.995867 \pm 0.001359$ | $0.995216 \pm 0.001393$ | $0.0000$ | $1090.8$ / $1236.7\text{s}$<br>$22{,}805\text{ MB}$ |
| Static-8L-Unif4d-m1068-True4x64 (Iso-Exec Control) | 4 | $1{,}068$ ($-$)<br>$559.94\text{M}$ | $1.3387\text{e}17$ /<br>$1.3387\text{e}17$ ($0\%$) | $0.988349 \pm 0.000869$ | $0.990377 \pm 0.000721$ | $0.988873 \pm 0.000810$ | $0.988310 \pm 0.000855$ | $0.0000$ | $1228.7$ / $1374.5\text{s}$<br>$22{,}805\text{ MB}$ |
| Static-12L-PyrTarget-True4x64 ($r=2^2,4^8,6^2$) | 4 | $659$ ($-$)<br>$345.51\text{M}$ | $1.1956\text{e}17$ /<br>$1.3840\text{e}17$ ($+15.8\%$) | $0.997620 \pm 0.000315$ | $0.999535 \pm 0.000347$ | $0.998088 \pm 0.000344$ | $0.997469 \pm 0.000323$ | $0.0000$ | $1215.6$ / $1402.4\text{s}$<br>$19{,}018\text{ MB}$ |
| G_stack-Unif4d-Apical-True4x64 ($4^8\to 4^{12}$, [0..7,4..7]) | 8 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.994232 \pm 0.002585$<br>(4s: $0.994455$) | $0.996169 \pm 0.002586$ | $0.994714 \pm 0.002579$ | $0.994127 \pm 0.002589$ | $+0.1286$ | $1198.7$ / $1394.7\text{s}$<br>$19{,}018\text{ MB}$ |
| G_stack-Unif4d-Medial4-True4x64 ($4^8\to 4^{12}$, [0..5,2..5,6,7]) | 8 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.991848 \pm 0.000990$<br>(4s: $0.992006$) | $0.993716 \pm 0.001013$ | $0.992349 \pm 0.000987$ | $0.991752 \pm 0.000973$ | $+0.0688$ | $1198.4$ / $1398.0\text{s}$<br>$22{,}805\text{ MB}$ |
| G_stack-PyrTarget-Medial-True4x64 (2-spl, 2-flip, {3,7,11}) | 4 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.993638 \pm 0.001252$ | $0.995457 \pm 0.001295$ | $0.994078 \pm 0.001238$ | $0.993507 \pm 0.001222$ | $+0.0950$ | $1190.9$ / $1388.8\text{s}$<br>$19{,}018\text{ MB}$ |
| G_stack-PyrTarget-Medial-WinLock-True4x64 (2-spl, 0-flip, {3,5,11}) | 4 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.993581 \pm 0.001284$ | $0.995404 \pm 0.001281$ | $0.994008 \pm 0.001284$ | $0.993437 \pm 0.001260$ | $+0.0970$ | $1191.3$ / $1389.3\text{s}$<br>$19{,}018\text{ MB}$ |
| G_stack-PyrTarget-Medial4-True4x64 (1-spl, 0-flip, {3,7,11}) | 8 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>$1.3384\text{e}17$ ($+12.0\%$) | $0.990915 \pm 0.001245$<br>(4s: $0.991206$) | $0.992773 \pm 0.001221$ | $0.991422 \pm 0.001197$ | $0.990834 \pm 0.001215$ | $+0.0834$ | $1192.9$ / $1386.8\text{s}$<br>$22{,}805\text{ MB}$ |
Table 2B: Paired 8-Seed (_8s, with 6-Test Holm-Bonferroni p_Holm6) and 4-Seed Contrasts Under Native True4x64 ($\text{Mean }\Delta \pm \text{SE}$, matches paper5_tables_verified.txt lines 96–144)
| Contrast ID & Comparison | $n$ | Design Type & Resource Control | Slice B $\Delta\text{BPB}$ ($\text{mBPB} \pm \text{SE}$) | Slice B 95% $t$-CI, $t$-stat, $p_{\text{raw}}$ (& $p_{\text{Holm6}}$) | confirm / Slice A / val $\Delta$ ($\text{mBPB}$) | Per-Seed Slice B $\Delta$ ($\text{mBPB}$) & Sign Count |
| :--- | :-: | :--- | :---: | :---: | :---: | :--- |
| U0_true_8s: PyrTarget-Medial4 vs. Static-8L-Unif4d ($m=953$) | 8 | Two-Stage Package vs. Static 8L; Iso-Nom FLOPs ($+12.04\%$ exec; 4s: $-4.202 \pm 0.310$) | $-4.395 \pm 0.368$ | $[-5.266, -3.524]$, $t=-11.93$, $p=0.000007$ ($p_{\text{Holm6}}=0.000040$) | $-4.505$ / $-4.445$ / $-4.382$ | $8/8$ negative: [-4.930, -3.435, -4.360, -4.082, -2.739, -4.216, -5.944, -5.453] |
| U0b_true_8s: Unif4d-Medial4 vs. Static-8L-Unif4d ($m=953$) | 8 | Two-Stage Growth vs. Static 8L; Iso-Nom FLOPs ($+12.04\%$ exec; 4s: $-3.402 \pm 0.353$) | $-3.461 \pm 0.328$ | $[-4.237, -2.686]$, $t=-10.55$, $p=0.000015$ ($p_{\text{Holm6}}=0.000075$) | $-3.562$ / $-3.518$ / $-3.464$ | $8/8$ negative: [-3.326, -3.763, -2.447, -4.070, -3.216, -1.942, -4.200, -4.727] |
| U0d_true_8s: Unif4d-Apical vs. Static-8L-Unif4d ($m=953$) | 8 | Two-Stage Apical $G_{\text{stack}}$ vs. Static 8L; Iso-Nom FLOPs ($+12.04\%$ exec) | $-1.077 \pm 0.528$ | $[-2.327, +0.172]$, $t=-2.04$, $p=0.0808$ ($p_{\text{Holm6}}=0.0923$) | $-1.109$ / $-1.153$ / $-1.088$ | $6/8$ negative: [-1.200, -1.987, -2.218, +1.595, -1.178, -2.236, -2.231, +0.837] |
| U6_true_8s: Unif4d-Medial4 vs. Unif4d-Apical | 8 | Bit-Identical Shared S1 Fork, Exact Iso-Exec FLOPs, Fixed {3,7,11} (4s: $-2.449 \pm 1.148$) | $-2.384 \pm 0.773$ | $[-4.211, -0.557]$, $t=-3.09$, $p=0.0177$ ($p_{\text{Holm6}}=0.0530$) | $-2.453$ / $-2.365$ / $-2.376$ | $7/8$ negative: [-2.126, -1.777, -0.229, -5.665, -2.038, +0.294, -1.969, -5.564] |
| U7_true_8s: PyrTarget-Medial4 vs. Unif4d-Apical | 8 | Two-Stage Package (PyrTarget S1+S2 + Medial4 vs. Unif4d Apical $G_{\text{stack}}$; Exact Iso-Exec; 4s: $-3.249$) | $-3.317 \pm 0.661$ | $[-4.881, -1.754]$, $t=-5.02$, $p=0.00153$ ($p_{\text{Holm6}}=0.00614$) | $-3.396$ / $-3.293$ / $-3.294$ | $8/8$ negative: [-3.730, -1.449, -2.142, -5.677, -1.561, -1.979, -3.713, -6.289] |
| U8_true_8s: PyrTarget-Medial4 vs. Unif4d-Medial4 | 8 | Two-Stage FFN Allocation ($r=(2^2,4^4,6^2)\to(2^2,4^8,6^2)$ vs. $4^8\to 4^{12}$ in Medial4; Exact Iso-Exec) | $-0.933 \pm 0.386$ | $[-1.845, -0.021]$, $t=-2.42$, $p=0.0462$ ($p_{\text{Holm6}}=0.0923$) | $-0.943$ / $-0.927$ / $-0.918$ | $6/8$ negative: [-1.603, +0.328, -1.913, -0.012, +0.477, -2.273, -1.744, -0.725] |
| U0_exec_true: PyrTarget-Medial4 ($m=728$, ckpt) vs. Static-8L-m1068 | 4 | Analytical Iso-Executed FLOPs ($1.3384\text{e}17$ vs. $1.3387\text{e}17$; $-178.3\text{M}$ tok) | $+2.857 \pm 0.334$ | $[+1.793, +3.920]$, $t=+8.55$, $p=0.00336$ | $+2.621$ / $+2.793$ / $+2.782$ | $0/4$ neg: [+2.015, +3.614, +2.733, +3.065]; 12L leads bi_l by $-16.073\text{ mBPB}$ |
| U0_s8_extra: Static-8L-m1068 vs. Static-8L-m953 | 4 | $+115$ Uncheckpointed 8L Steps ($+12.07\%$ FLOPs & tokens) | $-7.058 \pm 0.043$ | $[-7.194, -6.922]$, $t=-165.13$, $p<0.000001$ | $-6.998$ / $-7.114$ / $-7.023$ | $4/4$ negative: [-6.945, -7.049, -7.092, -7.147] |
| U0c_true / U1_true: Static-12L vs. Static-8L-m953 / Medial4 vs. Static-12L | 4 | Static 12L Inversion (U0c) / $-3.29\%$ Fewer Exec FLOPs (U1) | $+2.213 \pm 0.319$ /<br>$-6.414 \pm 0.568$ | $p=0.00614$ /<br>$p=0.00149$ | $+2.161$ / $-6.537$ (confirm) | U1_true ($4/4$): [-7.967, -5.568, -6.559, -5.563] |
| U2_true / U3_true: Medial4 vs. Medial-Pos / Medial-Pos vs. Static-12L | 4 | Bit-Identical S1 Fork (U2) / $-3.29\%$ Exec FLOPs (U3) | $-2.432 \pm 0.240$ /<br>$-3.982 \pm 0.504$ | $p=0.00204$ /<br>$p=0.00422$ | $-2.459$ / $-4.078$ (confirm) | U2_true ($4/4$): [-2.572, -2.557, -2.856, -1.743] |
| U4_true / U5_true: Medial4 vs. Medial-WinLock / WinLock vs. Medial-Pos | 4 | Bit-Identical S1 Fork (0 flips {3,7,11} vs. {3,5,11} / 0 vs. 2 flips) | $-2.375 \pm 0.182$ /<br>$-0.057 \pm 0.081$ | $p=0.000976$ /<br>$p=0.531$ | $-2.407$ / $-0.053$ (confirm) | U4_true: [-2.526, -2.575, -2.570, -1.829]<br>U5_true: [-0.046, +0.018, -0.286, +0.086] |
| M1 / M2 / M3: True4x64 vs. Split8x32 (0% recomp) | 4 | $B=64$ (ckpt 2..7) vs. $B=32$ (no ckpt) at $524\text{k}$ tok/step | M1: $-0.019 \pm 0.121$<br>M2: $+0.142 \pm 0.290$<br>M3: $+0.059 \pm 0.128$ | $p = 0.887$ / $0.658$ / $0.675$ | Slice A: $-0.010$ / $+0.125$ / $+0.027$ | Uncheckpointed Split8x32 Slice B: 0.991225 (Medial4), 0.993496 (Medial), 0.997561 (Static-12L) |
| B1_true / B2_true: Step-Budget (True4x64 $524\text{k}$ vs. 262k) | 4 | $728$ vs. $1{,}456$ steps (B1); $659$ vs. $1{,}318$ steps (B2) | — | $p = 0.000213$ / $0.000107$ | Slice A: $+12.407 \pm 0.570$ / $+14.172 \pm 0.518$ | Halving optimizer updates adds $+12.4$ to $+14.2\text{ mBPB}$ |
4.2 Conclusions and Statistical Scope of the 8-Seed and Iso-Executed-FLOP True4x64 Experiments
- Separating the 8-Seed Shared-Checkpoint Fork (
U6_true_8s) from the Two-Stage Package Contrasts (U7_true_8s,U8_true_8s):- On the bit-identical
Unif4dStage-1 checkpoint fork (U6_true_8s), holding FFN ratios ($4^{12}$), splice count ($1$), window flips ($0$), and destination global-attention depths ($\{3,7,11\}$) strictly fixed, duplicating the interiorSSSLperiod $[2..5]$ (Unif4d-Medial4, splice $5\to 2$) rather than the unembedding-adjacent apical period $[4..7]$ (Unif4d-Apical, splice $7\to 4$) halves the same-batch transition shock ($+0.0688$ vs. $+0.1286\text{ nats}$), reduces cross-seedSlice Bendpoint SD from $2.59\text{ mBPB}$ to $0.99\text{ mBPB}$, and improvesSlice BBPB by $-2.384 \pm 0.773\text{ mBPB}$ ($p_{\text{raw}} = 0.0177, p_{\text{Holm6}} = 0.0530$, $7/8$ seeds negative, with seed $512$ at $+0.294\text{ mBPB}$). - Comparing the two-stage interior-pyramidal package (
G_stack-PyrTarget-Medial4) against contiguous apicalG_stack(G_stack-Unif4d-Apical) yields $-3.317 \pm 0.661\text{ mBPB}$ across all 8 seeds ($p_{\text{raw}} = 0.00153, \mathbf{p_{\text{Holm6}} = 0.00614}$, $8/8$ seeds negative;U7_true_8s), while varying FFN allocation across both stages withinMedial4(PyrTarget-Medial4vs.Unif4d-Medial4) yields $-0.933 \pm 0.386\text{ mBPB}$ ($p_{\text{raw}} = 0.0462, p_{\text{Holm6}} = 0.0923$, $6/8$ seeds negative;U8_true_8s). BecauseU6_true_8s($p_{\text{Holm6}} = 0.0530$) andU8_true_8s($p_{\text{Holm6}} = 0.0923$) sit marginally above the 6-test Holm threshold $\alpha = 0.05$, we state explicitly thatU7_true_8s,U0_true_8s($p_{\text{Holm6}} = 0.000040$), andU0b_true_8s($p_{\text{Holm6}} = 0.000075$) are family-wise significant across the 6-test 8-seed suite, whereasU6_true_8sandU8_true_8sprovide unadjusted supporting evidence for the pure topology fork and two-stage FFN allocation components, respectively.
- On the bit-identical
- Nominal vs. Analytical Executed FLOP Trade-Off Relative to Static 8L (
U0_true_8s,U0_exec_true,U0_s8_extra, andM1): Relative to 953-step nominal-FLOP-matchedStatic-8L-Unif4d-True4x64, 12LG_stack-PyrTarget-Medial4wins by $-4.395 \pm 0.368\text{ mBPB}$ across 8 seeds ($p_{\text{Holm6}} = 0.000040$;U0_true_8s). However, because 6-block activation checkpointing (2 <= l < 8) adds $+12.04\%$ analytical executed FLOPs during the 503 Stage-2 steps, granting uncheckpointedStatic-8L-Unif4d-True4x64those same analytical executed FLOPs (1,068steps, $559.94\text{M}$ tokens, $1.3387 \times 10^{17}$ FLOPs) improves static 8L by $-7.058 \pm 0.043\text{ mBPB}$ (U0_s8_extra), leavingStatic-8L-m1068ahead of checkpointed 728-stepPyrTarget-Medial4overall by $+2.857 \pm 0.334\text{ mBPB}$ onSlice B(U0_exec_true, $p = 0.00336$; novel bigrams $+3.585\text{ mBPB}$, short-repeat bigrams $+0.160\text{ mBPB}$), while 12LPyrTarget-Medial4still beatsStatic-8L-m1068by $-16.073\text{ mBPB}$ on long-distance bigrams (bi_l) and by $-0.738\text{ mBPB}$ on long-repeat unigrams (uni_l). Consequently, under nativeB=64when 6-block checkpointing is used at 12L, the $+12.04\%$ recomputation tax exceeds the aggregate BPB gain over an uncheckpointed 8L model allowed to train on $+60.3\text{M}$ extra tokens; avoiding recomputation (as in uncheckpointedSplit8x32, whereM1 = -0.019 ± 0.121 mBPBand executed FLOPs equal $1.1946 \times 10^{17}$) preserves the $-4.182\text{ mBPB}$ overall lead over static 8L at exact analytical iso-executed FLOPs. - Narrowed Interpretation of
Medial-WinLock(U4_trueandU5_true): On the 4-seedPyrTargetcheckpoint fork,U5_true($-0.057 \pm 0.081\text{ mBPB}$, $p = 0.531$) shows that locking parent window types on the two-spliceMedialmap produces little net benefit when destination global-layer depths simultaneously shift from $\{3,7,11\}$ to $\{3,5,11\}$, whileU4_true($-2.375 \pm 0.182\text{ mBPB}$, $p = 0.000976$) shows that contiguous period-preservingMedial4({3,7,11}, interface $5\to 2$) substantially outperforms the parent-window-locked two-splice layout ({3,5,11}, interfaces $3\to 2, 5\to 4$).
4.3 Summary of the 262k-Tok/Step Multi-Seed Campaign (C1–C8, S3–S10) and Learning Curves
Figure 3: Training Loss Trajectories, Same-Batch Mitosis Shock, and Post-Expansion Recovery
At $262{,}144$ tok/step (4x32 split-microbatch, $0\%$ recomputation, $1.1945 \times 10^{17}$ analytical nominal and executed FLOPs; full 21-arm table in Appendix A, Table A1), all 8 primary contrasts survive Holm-Bonferroni correction ($m = 8$, Table 3):
- Consistent
Medial4vs.MedialAdvantage Across Four Model Families: ContiguousMedial4beats two-spliceMedialin uniform copy (C2: $-1.556 \pm 0.337\text{ mBPB}$, $p_{\text{Holm}} = 0.0382$), FFN $2^8\to 4^{12}$ widening (S3: $-1.760 \pm 0.294\text{ mBPB}$, $p = 0.00935$), seed-pyramidal $2^6 4^2 \to 4^{12}$ (S4: $-1.641 \pm 0.113\text{ mBPB}$, $p = 0.000715$), andTrue4x64pyramidal copy (U2_true: $-2.432 \pm 0.240\text{ mBPB}$, $p = 0.00204$)—winning on $16/16$ paired seeds ($20/20$ includingMedial-WinLock). - 12L and 16L Pyramidal Growth Gains (
C3–C8,S8):G_stack-PyrTarget-Medial4(sh9A$= 0.979259 \pm 0.000614\text{ BPB}$) beatsStatic-12L-PyrTargetby $-4.658 \pm 0.636\text{ mBPB}$ (C3, $p_{\text{Holm}} = 0.0210$),Static-12L-Unif4dby $-6.230 \pm 0.751\text{ mBPB}$ (C4, $p_{\text{Holm}} = 0.0183$), andStatic-8L-Unif4dby $-7.342 \pm 0.452\text{ mBPB}$ (C5, $p_{\text{Holm}} = 0.00405$), matching coupled 2D width+depth growthPyrTarget-KeepAp-Medial4(0.979257 ± 0.000576,S8: $-0.002 \pm 0.123\text{ mBPB}$) without requiring FFN width splitting. At 16L, 3-stagePyr16-3Stage-Copy-Medial4(0.980932 ± 0.000512) beatsStatic-16L-Pyr40by $-7.194 \pm 0.476\text{ mBPB}$ (C7, $p_{\text{Holm}} = 0.00440$) andStatic-16L-Unif2dby $-8.321 \pm 0.612\text{ mBPB}$ (C8, $p_{\text{Holm}} = 0.00518$). As Figure 3b illustrates, on the bit-identicalTrue4x64forks,Medial4andUnif4d-Medial4absorb their smaller transition shocks rapidly and maintain a clean trajectory lead overMedial-Positional,Medial-WinLock, andUnif4d-Apicalthrough warmdown.
Table 3: Conditional 4-Seed Paired Contrasts at $262{,}144$ Tok/Step (C1–C8 with Holm-Bonferroni $m=8$, plus Key Secondary Contrasts; $\text{Mean }\Delta \pm \text{SE}$)
| Contrast ID & Comparison (0% recomp, $1.1945\text{e}17$ exec FLOPs) | sh9A $\Delta\text{BPB}$ ($\text{mBPB} \pm \text{SE}$) | Paired 95% $t$-CI ($\text{mBPB}$) | val / hld $\Delta$ ($\text{mBPB}$) | $t$-stat ($df=3$) | $p_{\text{raw}}$ | $p_{\text{Holm}}$ ($m=8$) | Per-Seed sh9A $\Delta$ ($\text{mBPB}$) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :--- |
| C1: Static-16L-Pyr40 vs. Static-16L-Unif2d | $-1.127 \pm 0.257$ | $[-1.946, -0.308]$ | $-1.082$ / $-1.135$ | $-4.38$ | $0.02203$ | $0.03821$ | [-0.460, -1.029, -1.663, -1.357] |
| C2: G_stack-Unif4d-Medial4 vs. G_stack-Unif4d-Medial | $-1.556 \pm 0.337$ | $[-2.628, -0.484]$ | $-1.566$ / $-1.539$ | $-4.62$ | $0.01911$ | $0.03821$ | [-2.514, -0.964, -1.260, -1.485] |
| C3: G_stack-PyrTarget-Medial4 vs. Static-12L-PyrTarget | $-4.658 \pm 0.636$ | $[-6.680, -2.635]$ | $-4.636$ / $-4.725$ | $-7.33$ | $0.00525$ | $0.02100$ | [-6.412, -3.451, -4.099, -4.668] |
| C4: G_stack-PyrTarget-Medial4 vs. Static-12L-Unif4d | $-6.230 \pm 0.751$ | $[-8.620, -3.840]$ | $-6.203$ / $-6.278$ | $-8.30$ | $0.00367$ | $0.01835$ | [-7.582, -6.730, -6.523, -4.084] |
| C5: G_stack-PyrTarget-Medial4 vs. Static-8L-Unif4d | $-7.342 \pm 0.452$ | $[-8.779, -5.904]$ | $-7.278$ / $-7.431$ | $-16.26$ | $0.00051$ | $0.00405$ | [-8.446, -7.181, -6.255, -7.484] |
| C6: PyrTarget-KeepAp-Medial4 vs. Static-12L-PyrTarget | $-4.660 \pm 0.645$ | $[-6.711, -2.608]$ | $-4.646$ / $-4.747$ | $-7.23$ | $0.00546$ | $0.02100$ | [-6.279, -3.176, -4.317, -4.866] |
| C7: Pyr16-3Stage-Copy-Medial4 vs. Static-16L-Pyr40 | $-7.194 \pm 0.476$ | $[-8.709, -5.680]$ | $-7.164$ / $-7.281$ | $-15.12$ | $0.00063$ | $0.00440$ | [-7.454, -5.922, -8.209, -7.192] |
| C8: Pyr16-3Stage-Copy-Medial4 vs. Static-16L-Unif2d | $-8.321 \pm 0.612$ | $[-10.271, -6.372]$ | $-8.246$ / $-8.416$ | $-13.59$ | $0.00086$ | $0.00518$ | [-7.914, -6.950, -9.872, -8.549] |
| S3 / S4: Medial4 vs. Medial in M2D-Unif4d / Pyr-Apical2 | $-1.760 \pm 0.294$ /<br>$-1.641 \pm 0.113$ | $[-2.697, -0.824]$ /<br>$[-2.001, -1.280]$ | $-1.717$ / $-1.603$ (val) | $-5.98$ /<br>$-14.47$ | $0.00935$ /<br>$0.00072$ | — | [-1.635,-2.331,-0.991,-2.085] /<br>[-1.624,-1.903,-1.352,-1.684] |
| S5 / S7: Pyr-Apical2-Medial4 vs. Apical / PyrTarget vs. Apical2 | $-1.251 \pm 0.469$ /<br>$-1.134 \pm 0.206$ | $[-2.743, +0.242]$ /<br>$[-1.789, -0.479]$ | $-1.162$ / $-1.183$ (val) | $-2.67$ /<br>$-5.51$ | $0.07588$ /<br>$0.01178$ | — | [-0.147,-2.104,-0.801,-1.951] /<br>[-1.503,-1.321,-1.157,-0.554] |
| S8 / S9: KeepAp vs. G_stack-PyrTarget / 16L 3-Stg vs. 2-Stg | $-0.002 \pm 0.123$ /<br>$-0.556 \pm 0.253$ | $[-0.392, +0.388]$ /<br>$[-1.361, +0.249]$ | $-0.010$ / $-0.581$ (val) | $-0.02$ /<br>$-2.20$ | $0.98799$ /<br>$0.11534$ | — | [+0.133,+0.275,-0.219,-0.198] /<br>[-1.065,-0.374,-0.853,+0.067] |
5. Bounded Diagnostic and Matched-Configuration 4-Seed Continuation Study of Per-Block Sequence Coarsening
Figure 4: Per-Block Coarsening Diagnostics: Parity Asymmetry, Filterbank Scalars, and Matched 4-Seed KeepFilter Continuations
5.1 Per-Block Coarse Residual Wrapper and Transition Operators (train_multigrid.py)
Rather than maintaining a persistent shortened hidden stack across layers as in Hourglass [7] or Funnel-Transformer [9], experiments_p6/train_multigrid.py (Block.forward, lines 369–435) keeps the main residual stream $x \in \mathbb{R}^{B\times T\times d}$ at full resolution $T = 2{,}048$ and wraps selected interior blocks $l$ at spatial stride $s_l = 2$ ($T_c = 1{,}024$) with learnable scalars $(\gamma_{R,l}, \beta_{0,l}, \beta_{1,l})$:
- Normalized Prefix-Causal Restriction ($T \to T/2$): With phase $\phi_l \in \{0, 1\}$ (
Phase0: $\phi_l = 0$;Haar: $\phi_l = l \bmod 2$), block $l$ forms $z_{\text{in}, m} = (x_{2m+\phi_l} + \gamma_{R,l} x_{2m+\phi_l-1}) / \sqrt{1 + \gamma_{R,l}^2}$ ($m \in \{0,\dots,T/2-1\}$, zero-padded at $t=-1$), evaluates coarse attention (window $w_l // 2$, decimated RoPE/VE) and MLP to obtain $\Delta z = A_l(z_{\text{in}}) + M_l(z_{\text{in}} + A_l(z_{\text{in}}))$, and prolongates $\Delta z$ via causal ZOH ($\Delta x_{2m+\phi_l} = \beta_{0,l}\Delta z_m$, $\Delta x_{2m+\phi_l+1} = \beta_{1,l}\Delta z_m$) to return $x + \Delta x$. - Four Matched-Configuration Coarse-to-Fine ($s_l: 2 \to 1$) Transition Rules at Step $g = 190$: When transitioning coarse blocks $[6..9]$ to fine stride $s_l = 1$ at step $g = 190$, we compare four transition operators that separate the 2-tap causal input filter $z_{\text{in},t} = (x_t + \gamma_{R,l} x_{t-1})/\sqrt{1+\gamma_{R,l}^2}$ from the mean residual gain $\bar{\beta}_l = \frac{1}{2}(\beta_{0,l}+\beta_{1,l})$:
Unfolded4(naive $s_l: 2 \to 1$): drops both the input filter and $\bar{\beta}_l$, computing $x + A_l(x) + M_l(x + A_l(x))$.ScaleFold4(--c2f-scale-fold): drops the input filter and folds $\bar{\beta}_l$ intoattn.c_projandmlp.c_proj, computing $x + \bar{\beta}_l A_l(x) + \bar{\beta}_l M_l(x + \bar{\beta}_l A_l(x))$ (which scales the internal MLP residual input $x + \bar{\beta}_l A_l(x)$ before pre-MLP RMSNorm).OuterGainNoFilter4(--c2f-outer-gain-nofilter): drops only the 2-tap input filter while keeping the exact outer block gain $x + \bar{\beta}_l [A_l(x) + M_l(x + A_l(x))]$ (train_multigrid.py, lines 409–416).KeepFilter4(--c2f-keep-filter): retains both the 2-tap causal input filter $z_{\text{in}}$ and the exact outer block gain $x + \bar{\beta}_l [A_l(z_{\text{in}}) + M_l(z_{\text{in}} + A_l(z_{\text{in}}))]$.
5.2 Matched-Configuration 4-Seed C2F Contrasts (C2F_1–C2F_5) and Bounded Negative V-Cycle Finding
All four 4-seed (42, 137, 271, 314) continuation arms are trained as matched-configuration two-stage runs with the same Stage-1 coarse configuration on $[6..9]$ ($r = (2^2, 4^8, 6^2)$, $g = 190$, $366$ total steps, $m = 183$, $95.94\text{M}$ tokens, $2.87 \times 10^{16}$ FLOPs; Table 4) rather than loaded from a saved Stage-1 checkpoint file. Because torch.compile and CUDA reduction non-determinism introduce minor Stage-1 trajectory drift across independent two-stage runs prior to $g = 190$ (for example, at seed 42, loss_before is $3.441200$ for KeepFilter4, $3.445084$ for OuterGainNoFilter4, $3.450536$ for ScaleFold4, and $3.447616$ for Unfolded4, with block-6 mean gains $\bar{\beta}_6 = 0.35097, 0.34420, 0.33418, 0.34324$, respectively; across all 4 seeds, mean loss_before is $3.4494 \pm 0.0066$ vs. $3.4499 \pm 0.0058$), we also log the exact counterfactual same-checkpoint transition losses (loss_after_keep_filter, loss_after_folded, loss_after_unfolded) inside every run's result.json at $g = 190$ (delta_loss_keep_filter = +0.0853 to +0.0956 nats vs. delta_loss_unfolded = +2.6814 to +2.7897 nats on the identical Stage-1 state).
Table 4: Matched-Configuration 4-Seed Coarse-to-Fine Continuations on [6..9] at $g=190$ ($366$ steps, s240, matches paper5_tables_verified.txt lines 165–174) & Full-Budget V-Cycle Diagnostics
| Arm ($\text{Mean} \pm \text{SD}$) or Contrast ($\text{Mean }\Delta \pm \text{SE}$) | $n$ | Coarse Blocks & Steps ($g$) | Same-Batch Shock $\Delta\mathcal{L}$ (nats) | sh9A BPB (or $\Delta\text{mBPB}$) | val_bpb / holdout_bpb (or $\Delta\text{mBPB}$) | Paired 95% $t$-CI, $t$-stat, $p_{\text{raw}}$, & Exact Seeds (paper5_tables_verified.txt) |
| :--- | :-: | :---: | :---: | :---: | :---: | :--- |
| C2F-MG12-KeepFilter4-g190 (--c2f-keep-filter) | 4 | $[6..9]$, $366$ ($190$) | $+0.0956$ | $1.107784 \pm 0.001045$ | $1.107012 \pm 0.001007$ /<br>$1.106648 \pm 0.000987$ | Retains 2-tap filter $z_{\text{in}}$ & outer gain $\bar{\beta}_l \Delta z$ |
| C2F-MG12-OuterGainNoFilter4-g190 (--c2f-outer-gain-nofilter) | 4 | $[6..9]$, $366$ ($190$) | $+2.1616$ | $1.124970 \pm 0.002874$ | $1.124208 \pm 0.002884$ /<br>$1.124021 \pm 0.002895$ | Exact outer gain $\bar{\beta}_l \Delta z$; drops only 2-tap filter |
| C2F-MG12-ScaleFold4-g190 (--c2f-scale-fold) | 4 | $[6..9]$, $366$ ($190$) | $+2.2675$ | $1.125570 \pm 0.002375$ | $1.124747 \pm 0.002394$ /<br>$1.124557 \pm 0.002422$ | Folds $\bar{\beta}_l$ into c_proj; drops 2-tap filter |
| C2F-MG12-Unfolded4-g190 (naive $s_l: 2\to 1$) | 4 | $[6..9]$, $366$ ($190$) | $+2.6814$ | $1.126972 \pm 0.002568$ | $1.126206 \pm 0.002592$ /<br>$1.125989 \pm 0.002624$ | Drops both 2-tap filter and $\bar{\beta}_l$ gain |
| C2F_1: KeepFilter4 vs. Unfolded4 (full package) | 4 | Matched S1 ($[6..9]$) | $-2.5858\text{ nats}$ | $-19.188 \pm 1.061$ | $-19.194$ / $-19.342$ | CI $[-22.564, -15.812]$, $t=-18.09$, $p=0.000369$ ($4/4$)<br>[-19.891, -18.781, -21.570, -16.510] |
| C2F_2_pure_filter: KeepFilter4 vs. OuterGainNoFilter4 | 4 | Filter Effect Conditional on Outer Gain | $-2.0660\text{ nats}$ | $-17.187 \pm 1.370$ | $-17.197$ / $-17.373$ | CI $[-21.547, -12.826]$, $t=-12.54$, $p=0.00109$ ($4/4$)<br>[-19.789, -15.821, -19.130, -14.006] |
| C2F_2_scalefold: KeepFilter4 vs. ScaleFold4 | 4 | Filter + Outer vs. Folded Gain | $-2.1719\text{ nats}$ | $-17.786 \pm 1.070$ | $-17.735$ / $-17.909$ | CI $[-21.190, -14.383]$, $t=-16.63$, $p=0.000473$ ($4/4$)<br>[-19.792, -16.890, -19.268, -15.196] |
| C2F_3_outer_gain: OuterGainNoFilter4 vs. Unfolded4 | 4 | Outer Gain $\bar{\beta}_l$ Without Filter | $-0.5198\text{ nats}$ | $-2.002 \pm 0.644$ | $-1.997$ / $-1.968$ | CI $[-4.050, +0.047]$, $t=-3.11$, $p=0.0529$ ($4/4$)<br>[-0.102, -2.960, -2.440, -2.504] |
| C2F_4_gain_form: OuterGainNoFilter4 vs. ScaleFold4 | 4 | Outer Gain vs. c_proj Folding | $-0.1059\text{ nats}$ | $-0.600 \pm 0.308$ | $-0.538$ / $-0.536$ | CI $[-1.580, +0.381]$, $t=-1.95$, $p=0.147$ ($4/4$)<br>[-0.003, -1.069, -0.137, -1.190] |
| C2F_5_scalefold_vs_unfolded: ScaleFold4 vs. Unfolded4 | 4 | Folded Gain Without Filter | $-0.4139\text{ nats}$ | $-1.402 \pm 0.479$ | $-1.459$ / $-1.432$ | CI $[-2.926, +0.123]$, $t=-2.93$, $p=0.0612$ ($4/4$)<br>[-0.100, -1.891, -2.302, -1.314] |
| MG-V12-Pyr-Phase0 / MG-V16-Pyr-Phase0 (Full Budget) | 1 | $[6..9]$ ($1600$ st) /<br>$[6..13]$ ($1626$ st) | $0.0000$ | $0.990401$ /<br>$0.994328$ | $0.989659$ / $0.993613$ (val) | Odd$-$Even parity gap: $+2.054$ / $+4.431\text{ mBPB}$; trails full-res growth by $>10\text{ mBPB}$ |
Diagnostic Takeaways: (1) Conditional on retaining the outer block gain $\bar{\beta}_l$, C2F_2_pure_filter (KeepFilter4 vs. OuterGainNoFilter4: $-17.187 \pm 1.370\text{ mBPB}$, $p = 0.00109$, $4/4$ seeds negative; same-batch shock $+0.0956$ vs. $+2.1616\text{ nats}$) isolates the 2-tap causal input filter and accounts for a conditional mean-contrast ratio of $89.6\% \pm 3.6\%$ (C2F_2_pure_filter / C2F_1, jackknife SE; per-seed ratios $99.5\%, 84.2\%, 88.7\%, 84.8\%$), whereas outer vs. c_proj-folded residual scaling without the filter differs by only $-0.600 \pm 0.308\text{ mBPB}$ (C2F_4_gain_form, $p = 0.147$). Because the $2 \times 2$ design lacks a filter-only-without-outer-gain arm, we report this $89.6\% \pm 3.6\%$ decomposition explicitly as a contrast ratio conditional on the retained outer-gain setting rather than an interaction-free additive attribution. (2) Even C2F-MG12-KeepFilter4 (1.107784 at s240) and full-budget MG-V12-Pyr-Phase0 (0.990401, $+2.054\text{ mBPB}$ odd–even ZOH parity gap) trail static Static-12L-PyrTarget (1.105832 / 0.983916) and full-resolution G_stack-PyrTarget-Medial4 (1.093651 / 0.979259), bounding this per-block stride-2 wrapper as a negative pretraining efficiency trade-off despite its transition fix.
6. Discussion and Scope Limitations
- Summary of Bounded Empirical Contributions Relative to Prior Literature:
- Controlled Fixed-FLOP Evaluation of Vertical FFN Allocation (
I1–I1d,T1–T2,U0_s8_linear): Building on DeLighT [11] and OpenELM [10], we show that under fixed pretraining FLOPs at 16 layers, both a smooth linear FFN ramp (Linear40: $-4.328 \pm 0.268\text{ mBPB}$ vs.Basal40, $-1.656 \pm 0.337\text{ mBPB}$ vs.Unif2d) and step-apical placement (Pyr40: $-3.799 \pm 0.201\text{ mBPB}$ vs.Basal40, persisting under all-globalLLLLat $-3.716\text{ mBPB}$ and at reduced width $d=384, H=3$ at $-3.143\text{ mBPB}$) outperform basal and uniform allocations, whereas linear ramping at 8 layers is near-neutral (U0_s8_linear = -0.400 ± 0.324 mBPB). Uncalibrated Logit-Lens and SVD stable-rank profiles (Table 1C) are reported strictly as exploratory descriptive associations [12]. - 8-Seed Splicing Evidence and Explicit Checkpointing FLOP Trade-Offs (
U0_true_8s–U8_true_8s,U0_exec_true): On our 8-seed bit-identicalUnif4dStage-1 checkpoint fork at fixed{3,7,11}global depths, interior period-preserving growth beats contiguous apicalG_stackby $-2.384 \pm 0.773\text{ mBPB}$ (U6_true_8s, $p = 0.0177, p_{\text{Holm6}} = 0.0530$), while the two-stage interior-pyramidal package beats apicalG_stackby $-3.317 \pm 0.661\text{ mBPB}$ (U7_true_8s, $p_{\text{Holm6}} = 0.00614$) and 953-step static 8L by $-4.395 \pm 0.368\text{ mBPB}$ (U0_true_8s, $p_{\text{Holm6}} = 0.000040$). At the same time, our 1,068-step uncheckpointed static 8L control (U0_exec_true = +2.857 ± 0.334 mBPB) demonstrates that when 12L training requires 6-block activation checkpointing (+12.04%analytical executed FLOPs), an uncheckpointed 8L model given those extra FLOPs as $+115$ token steps achieves lower aggregate BPB, even though 12L growth retains a $-16.073\text{ mBPB}$ advantage on long-distance bigrams and wins by $-4.182\text{ mBPB}$ when both models run without checkpointing (Split8x32). - Matched-Configuration C2F Filter Retention (
C2F_2_pure_filter): Conditional on retaining the outer block gain,KeepFilter4beatsOuterGainNoFilter4by $-17.187 \pm 1.370\text{ mBPB}$ ($p = 0.00109$, $89.6\% \pm 3.6\%$ of the $-19.188\text{ mBPB}$ recovery over naive unfolding).
- Controlled Fixed-FLOP Evaluation of Vertical FFN Allocation (
- Limitations: Because
Medial4({3,7,11}, interface $5\to 2$) andMedial-WinLock({3,5,11}, interfaces $3\to 2, 5\to 4$) differ in destination global-layer depths as well as splice count,U4_truesupports contiguous period preservation over this parent-window-locked two-splice layout rather than an arbitrary-interface splice-count theorem. All 242 runs are conducted at $50\text{M}\text{--}80\text{M}$ parameters ($d \in \{384, 512\}$, Muon/AdamW) on ClimbMix; verification at multi-billion-parameter scales and persistent token-routing hierarchies [7] remains open.
References
- rStation
general_pretraining_7/v3: Copy-stacking depth growth beats time-matched scratch training at two transitions, and its margin depends on when the trunk is grown. - rStation
general_pretraining_24/v4: Hadamard Modal-Decoupled Late Morphogenesis: Overcoming the Cooldown Horizon in Progressive Language Model Growth. - rStation
general_pretraining_26/v2: Complementary Microbatch Braiding: Non-Cloning Two-Stage Depth Training and Controlled Comparisons Against Early Weight Cloning Under Fixed Pretraining FLOPs. - rStation
general_pretraining_28/v1: Two-Stage Depth Growth and Moment-Free Hashed Surface Memory Under Fixed Pretraining Matrix/Attention FLOPs: Static Depth Inversion, Uniform-Microbatch Controls, and Exposure-Frozen Diagnostics. - Chen, T., Goodfellow, I., & Shlens, J. (2016). Net2Net: Accelerating Learning via Knowledge Transfer. ICLR 2016, arXiv:1511.05641.
- Du, W., et al. (2024). Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training. NeurIPS 2024, arXiv:2405.15319v2 (Appendices H.2–H.3 and J).
- Nawrot, P., et al. (2022). Hierarchical Transformers Are More Efficient Language Models. Findings of NAACL 2022, pp. 1559–1571 (arXiv:2110.13711).
- Gong, L., et al. (2019). Efficient Training of BERT by Progressively Stacking. ICML 2019, PMLR 97:2337–2346.
- Dai, Z., Lai, G., Yang, Y., & Le, Q. V. (2020). Funnel-Transformer: Filtering Out Sequential Redundancy for Efficient Language Processing. NeurIPS 2020, arXiv:2006.03236.
- Mehta, S., Sekhavat, M. H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., & Rastegari, M. (2024). OpenELM: An Efficient Language Model Family with Open Training and Inference Framework. arXiv:2404.14619 (§2.1, layer-wise linear FFN multiplier ramping).
- Mehta, S., Ghazvininejad, M., Iyer, S., Zettlemoyer, L., & Hajishirzi, H. (2021). DeLighT: Deep and Light-weight Transformer. ICLR 2021, arXiv:2008.00623.
- Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., & Steinhardt, J. (2023). Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112.
Appendix A: Full-Budget 262k-Tok/Step Arm Table, Split8x32 Triad, and Descriptive Token-Class Decompositions
Table A1: Complete 21-Arm Full-Budget ($1.1945 \times 10^{17}$ Analytical Nominal & Executed FLOPs, $0\%$ Recomp, $262{,}144$ tok/step 4x32) Summaries (Mean ± SD, matches paper5_tables_verified.txt lines 5–25)
| Arm Name & FFN Profile | $n$ | Params | Steps ($g$) | Tokens | val_bpb ($\text{Mean} \pm \text{SD}$) | holdout_bpb ($\text{Mean} \pm \text{SD}$) | sh9A BPB ($\text{Mean} \pm \text{SD}$) | sh9A Bigram Nov / Short / Long | $\Delta\mathcal{L}$ (nats) | Train (s) / VRAM |
| :--- | :-: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Static-8L-Unif4d ($4^8$) | 4 | $50.33\text{M}$ | $1{,}906$ ($-$) | $499.65\text{M}$ | $0.985985 \pm 0.000314$ | $0.985197 \pm 0.000264$ | $0.986600 \pm 0.000315$ | $1.127654$ / $0.357426$ / $0.820019$ | $0.0000$ | $1093.4\text{s}$ / $11702\text{MB}$ |
| Static-12L-Unif4d ($4^{12}$) | 4 | $71.30\text{M}$ | $1{,}318$ ($-$) | $345.51\text{M}$ | $0.984910 \pm 0.001510$ | $0.984044 \pm 0.001550$ | $0.985488 \pm 0.001507$ | $1.126379$ / $0.357672$ / $0.799072$ | $0.0000$ | $1098.9\text{s}$ / $16941\text{MB}$ |
| Static-12L-PyrTarget ($2^2,4^8,6^2$) | 4 | $71.30\text{M}$ | $1{,}318$ ($-$) | $345.51\text{M}$ | $0.983343 \pm 0.000717$ | $0.982491 \pm 0.000806$ | $0.983916 \pm 0.000783$ | $1.124697$ / $0.356456$ / $0.801790$ | $0.0000$ | $1103.7\text{s}$ / $16941\text{MB}$ |
| G_stack-Unif4d-Medial ($4^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}456$ ($450$) | $381.68\text{M}$ | $0.980783 \pm 0.000446$ | $0.979898 \pm 0.000437$ | $0.981347 \pm 0.000392$ | $1.122179$ / $0.353539$ / $0.802880$ | $+0.1436$ | $1093.1\text{s}$ / $17086\text{MB}$ |
| G_stack-Unif4d-Medial4 ($4^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}456$ ($450$) | $381.68\text{M}$ | $0.979218 \pm 0.000723$ | $0.978359 \pm 0.000799$ | $0.979791 \pm 0.000731$ | $1.120563$ / $0.352014$ / $0.809056$ | $+0.1037$ | $1090.8\text{s}$ / $17086\text{MB}$ |
| G_stack-PyrTarget-Medial4 ($2^2 4^4 6^2\to 2^2 4^8 6^2$) | 4 | $71.30\text{M}$ | $1{,}456$ ($450$) | $381.68\text{M}$ | $0.978707 \pm 0.000603$ | $0.977766 \pm 0.000459$ | $0.979259 \pm 0.000614$ | $1.119797$ / $0.352826$ / $0.799177$ | $+0.1783$ | $1102.6\text{s}$ / $17086\text{MB}$ |
| M2D-Unif4d-Medial ($2^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($315$) | $382.73\text{M}$ | $0.981718 \pm 0.000942$ | $0.980859 \pm 0.000936$ | $0.982323 \pm 0.000937$ | $1.123114$ / $0.354754$ / $0.802233$ | $+0.1568$ | $1098.5\text{s}$ / $17054\text{MB}$ |
| M2D-Unif4d-Medial4 ($2^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($315$) | $382.73\text{M}$ | $0.980002 \pm 0.001115$ | $0.979054 \pm 0.001222$ | $0.980562 \pm 0.001166$ | $1.121361$ / $0.353148$ / $0.794470$ | $+0.1334$ | $1096.7\text{s}$ / $17054\text{MB}$ |
| Pyr-Apical2-Apical ($2^6 4^2\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($343$) | $382.73\text{M}$ | $0.981052 \pm 0.001174$ | $0.980221 \pm 0.001166$ | $0.981643 \pm 0.001165$ | $1.121591$ / $0.357588$ / $0.810487$ | $+0.1756$ | $1092.7\text{s}$ / $17062\text{MB}$ |
| Pyr-Apical2-Medial ($2^6 4^2\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($343$) | $382.73\text{M}$ | $0.981493 \pm 0.000671$ | $0.980549 \pm 0.000717$ | $0.982033 \pm 0.000645$ | $1.122618$ / $0.355621$ / $0.794589$ | $+0.1530$ | $1096.4\text{s}$ / $17062\text{MB}$ |
| Pyr-Apical2-Medial4 ($2^6 4^2\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($343$) | $382.73\text{M}$ | $0.979890 \pm 0.000676$ | $0.978880 \pm 0.000677$ | $0.980392 \pm 0.000657$ | $1.121140$ / $0.353495$ / $0.785159$ | $+0.1426$ | $1096.0\text{s}$ / $17062\text{MB}$ |
| PyrTarget-SplitAp-Medial4 ($2^6 3^2\to 2^2 4^8 6^2$) | 4 | $71.30\text{M}$ | $1{,}460$ ($328$) | $382.73\text{M}$ | $0.978926 \pm 0.001707$ | $0.977953 \pm 0.001819$ | $0.979450 \pm 0.001759$ | $1.119964$ / $0.353297$ / $0.794008$ | $+0.1151$ | $1108.9\text{s}$ / $17058\text{MB}$ |
| PyrTarget-KeepAp-Medial4 ($2^6 6^2\to 2^2 4^8 6^2$) | 4 | $71.30\text{M}$ | $1{,}460$ ($375$) | $382.73\text{M}$ | $0.978697 \pm 0.000647$ | $0.977744 \pm 0.000602$ | $0.979257 \pm 0.000576$ | $1.119788$ / $0.353345$ / $0.783655$ | $+0.1445$ | $1106.0\text{s}$ / $17070\text{MB}$ |
| Static-16L-Unif2d ($2^{16}$) | 4 | $75.50\text{M}$ | $1{,}294$ ($-$) | $339.21\text{M}$ | $0.988647 \pm 0.001297$ | $0.987799 \pm 0.001287$ | $0.989254 \pm 0.001271$ | $1.131080$ / $0.356924$ / $0.812456$ | $0.0000$ | $1131.6\text{s}$ / $17860\text{MB}$ |
| Static-16L-Basal40 ($6^2,2^{14}$) | 4 | $79.69\text{M}$ | $1{,}208$ ($-$) | $316.67\text{M}$ | $0.991377 \pm 0.000668$ | $0.990533 \pm 0.000696$ | $0.991926 \pm 0.000642$ | $1.133720$ / $0.359893$ / $0.810332$ | $0.0000$ | $1118.5\text{s}$ / $18940\text{MB}$ |
| Static-16L-Pyr40 ($2^{14},6^2$) | 4 | $79.69\text{M}$ | $1{,}208$ ($-$) | $316.67\text{M}$ | $0.987565 \pm 0.000750$ | $0.986664 \pm 0.000807$ | $0.988126 \pm 0.000833$ | $1.129301$ / $0.358648$ / $0.813951$ | $0.0000$ | $1122.0\text{s}$ / $18940\text{MB}$ |
| Static-16L-Linear40 ($1^4,2^4,3^4,4^4$) | 4 | $79.69\text{M}$ | $1{,}208$ ($-$) | $316.67\text{M}$ | $0.987008 \pm 0.001136$ | $0.986120 \pm 0.001101$ | $0.987598 \pm 0.001086$ | $1.128686$ / $0.358568$ / $0.811452$ | $0.0000$ | $1124.7\text{s}$ / $18940\text{MB}$ |
| Pyr16-2Stage-Copy-Medial4 ($2^6 6^2\to 2^{14} 6^2$) | 4 | $79.69\text{M}$ | $1{,}360$ ($353$) | $356.52\text{M}$ | $0.980981 \pm 0.000245$ | $0.979979 \pm 0.000214$ | $0.981488 \pm 0.000250$ | $1.122771$ / $0.352298$ / $0.782633$ | $+0.4246$ | $1106.5\text{s}$ / $19069\text{MB}$ |
| Pyr16-3Stage-Copy-Medial4 ($2^6 6^2\to 2^{10} 6^2\to 2^{14} 6^2$) | 4 | $79.69\text{M}$ | $1{,}360$ ($216,490$) | $356.52\text{M}$ | $0.980401 \pm 0.000503$ | $0.979382 \pm 0.000480$ | $0.980932 \pm 0.000512$ | $1.122064$ / $0.352393$ / $0.782879$ | $+0.0939$/$+0.0681$ | $1128.3\text{s}$ / $19385\text{MB}$ |
| MG-V12-Pyr-Phase0 / MG-V16-Pyr-Phase0 | 1 | $71.3$/$79.7\text{M}$ | $1600$ / $1626$ | $419.4$/$426.3\text{M}$ | $0.989659$ / $0.993613$ | $0.988883$ / $0.992912$ | $0.990401$ / $0.994328$ | $1.132300$ / $1.136725$ (nov) | $0.0000$ | $1143$/$1206\text{s}$ |
Table A2: Uncheckpointed Split8x32 ($524{,}288$ tok/step, $B=32$, $0\%$ Recomp) Summaries & Slice B Descriptive Token-Class Profiles (paper5_tables_verified.txt lines 90–144)
Split8x32Arms (Mean ± SD, 4 seeds):Static-12L-PyrTarget-Split8x32:Slice B$0.997561 \pm 0.000473$,confirm$0.999504 \pm 0.000534$,sh9A$0.998061 \pm 0.000498$ (1075.4s,16941MB);G_stack-PyrTarget-Medial-Split8x32:Slice B$0.993496 \pm 0.001336$,confirm$0.995310 \pm 0.001339$,sh9A$0.993954 \pm 0.001336$, $\Delta\mathcal{L}=+0.0970$ (1078.7s,17086MB);G_stack-PyrTarget-Medial4-Split8x32:Slice B$0.991225 \pm 0.001613$,confirm$0.993085 \pm 0.001556$,sh9A$0.991676 \pm 0.001581$, $\Delta\mathcal{L}=+0.0899$ (1084.3s,17086MB). PairedSplit8x32contrasts ($\text{Mean }\Delta \pm \text{SE}$):U1_split = -6.336 ± 0.645 mBPB($p = 0.00224$),U2_split = -2.271 ± 0.154 mBPB($p = 0.000673$),U3_split = -4.065 ± 0.517 mBPB($p = 0.00429$). Remaining secondary contrasts at $262\text{k}$ tok/step:S6(G_stack-Unif4d-Medial4vs.M2D-Unif4d-Medial4): $-0.771 \pm 0.337\text{ mBPB}$ ($p=0.106$);S10(KeepApvs.SplitAp): $-0.194 \pm 0.658\text{ mBPB}$ ($p=0.787$).Slice BDescriptive Bigram (bi_nov$81.3\%$,bi_s$18.1\%$,bi_l$0.55\%$) & Unigram (uni_nov$47.1\%$,uni_s$51.4\%$,uni_l$1.53\%$) Profiles (paper5_tables_verified.txtlines 97–115): AcrossI1,I1b,T1, andT2, apicalPyr40and smooth-rampLinear40primarily improve novel compositional prediction (I1Slice A: $\Delta\text{bi\_nov} = -4.419\text{ mBPB}$;I1bSlice A: $\Delta\text{bi\_nov} = -5.034\text{ mBPB}$;T1Slice B: $\Delta\text{bi\_nov} = -4.315\text{ mBPB}$;T2Slice B: $\Delta\text{bi\_nov} = -3.783\text{ mBPB}$). UnderTrue4x64depth growth across 8 seeds (paper5_tables_verified.txtlines 97–107),U0_true_8s(PyrTarget-Medial4vs.Static-8L-m953) gives $\Delta\text{bi\_nov} = -4.470$, $\Delta\text{bi\_s} = -3.859$, $\Delta\text{bi\_l} = -10.855\text{ mBPB}$ (4-seed subset: $-4.259, -3.420, -21.502\text{ mBPB}$);U6_true_8s(Unif4d-Medial4vs.Unif4d-Apical) gives $\Delta\text{bi\_nov} = -1.842$, $\Delta\text{bi\_s} = -4.550$, $\Delta\text{bi\_l} = -11.234\text{ mBPB}$ (4-seed subset: $-1.978, -4.317, -10.600\text{ mBPB}$);U7_true_8sgives $\Delta\text{bi\_nov} = -3.065$, $\Delta\text{bi\_s} = -4.149$, $\Delta\text{bi\_l} = -13.236\text{ mBPB}$ (4-seed subset: $-3.046, -3.576, -22.571\text{ mBPB}$); andU0_exec_true(4 seeds, vs.Static-8L-m1068) gives $\Delta\text{bi\_nov} = +3.585$, $\Delta\text{bi\_s} = +0.160$, $\Delta\text{bi\_l} = -16.073\text{ mBPB}$.
Appendix B: Complete Quarter-Budget (s240, $2.87 \times 10^{16}$ FLOPs) Screening & Phase-6 Single-Seed Tables
Table B1: 31 Phase-5 s240 Runs (4x32 Split) & 3 Phase-7 Forked s240 Runs (2x64 Unsplit B=64, seed = 42)
| Run ID | Steps ($g$) | val_bpb / sh9A BPB ($\Delta\mathcal{L}$) | Run ID | Steps ($g$) | val_bpb / sh9A BPB ($\Delta\mathcal{L}$) |
| :--- | :---: | :---: | :--- | :---: | :---: |
| p5_s240_static8_unif4d_m240_s42 | $480$ ($-$) | $1.086468$ / $1.087076$ ($0.0000$) | p5_s240_pyr_hg4_modalw_apical_m184_g95_s42 | $368$ ($95$) | $1.106153$ / $1.106837$ ($+0.0001$) |
| p5_s240_static12_unif4d_m166_s42 | $332$ ($-$) | $1.108623$ / $1.109354$ ($0.0000$) | p5_s240_pyr_hg4_modalw_medial_m184_g95_s42 | $368$ ($95$) | $1.104336$ / $1.105082$ ($+0.0045$) |
| p5_s240_static12_pyrtarget_m166_s42 | $332$ ($-$) | $1.104939$ / $1.105832$ ($0.0000$) | p5_s240_pyr_hg4_modalw_copy_apical_m184_g95_s42 | $368$ ($95$) | $1.099664$ / $1.100392$ ($+0.1078$) |
| p5_s240_static16_unif2d_m163_s42 | $326$ ($-$) | $1.110256$ / $1.110991$ ($0.0000$) | p5_s240_pyr_hg4_modalw_copy_medial_m184_g95_s42 | $368$ ($95$) | $1.098698$ / $1.099424$ ($+0.0761$) |
| p5_s240_static16_pyr40_m152_s42 | $304$ ($-$) | $1.115051$ / $1.115797$ ($0.0000$) | p5_s240_pyr_hg6_modalw_medial_m184_g87_s42 | $368$ ($87$) | $1.100356$ / $1.101139$ ($+0.0019$) |
| p5_s240_gstack_medial_m184_g118_s42 | $368$ ($118$) | $1.098641$ / $1.099347$ ($+0.0705$) | p5_s240_pyr_hg6_modalw_copy_apical_m184_g87_s42 | $368$ ($87$) | $1.096717$ / $1.097459$ ($+0.0958$) |
| p5_s240_gstack_medial4_m184_g118_s42 | $368$ ($118$) | $1.095264$ / $1.095874$ ($+0.0679$) | p5_s240_pyr_hg6_modalw_copy_medial_m184_g87_s42 | $368$ ($87$) | $1.096215$ / $1.096928$ ($+0.0706$) |
| p5_s240_gstack_pyrtarget_medial4_m184_g118_s42 | $368$ ($118$) | $1.092842$ / $1.093651$ ($+0.0636$) | p5_s240_pyr_hg6_modalw_copy_medial4_m184_g87_s42 | $368$ ($87$) | $1.096367$ / $1.097090$ ($+0.0664$) |
| p5_s240_m2d_modalw_medial_m184_g80_s42 | $368$ ($80$) | $1.100769$ / $1.101387$ ($+0.0022$) | p5_s240_pyrtarget_splitap_modalw_medial4_m184_g83_s42 | $368$ ($83$) | $1.095189$ / $1.095888$ ($+0.0554$) |
| p5_s240_m2d_modalw_copy_medial_m184_g80_s42 | $368$ ($80$) | $1.097087$ / $1.097751$ ($+0.0512$) | p5_s240_pyrtarget_keepap_modalw_medial4_m184_g95_s42 | $368$ ($95$) | $1.094830$ / $1.095471$ ($+0.0568$) |
| p5_s240_m2d_modalw_copy_medial4_m184_g80_s42 | $368$ ($80$) | $1.096210$ / $1.096894$ ($+0.0452$) | p5_s240_pyr16_2stage_eqpart_medial4_m171_g88_s42 | $342$ ($88$) | $1.101026$ / $1.101660$ ($+0.0484$) |
| p5_s240_pyr_apical2_modalw_copy_apical_m184_g87_s42 | $368$ ($87$) | $1.096835$ / $1.097446$ ($+0.0685$) | p5_s240_pyr16_2stage_copy_medial4_m171_g88_s42 | $342$ ($88$) | $1.100037$ / $1.100741$ ($+0.1416$) |
| p5_s240_pyr_apical2_modalw_copy_medial_m184_g87_s42 | $368$ ($87$) | $1.096949$ / $1.097621$ ($+0.0481$) | p5_s240_pyr16_3stage_modaleq_medial4_m171_g54_g122_s42 | $342$ ($54,122$) | $1.105312$ / $1.105986$ ($+0.0075$/$-0.0004$) |
| p5_s240_pyr_apical2_modalw_copy_medial4_m184_g87_s42 | $368$ ($87$) | $1.096278$ / $1.097012$ ($+0.0372$) | p5_s240_pyr16_3stage_copy_medial4_m171_g54_g122_s42 | $342$ ($54,122$) | $1.101907$ / $1.102592$ ($+0.0276$/$+0.0542$) |
| p5_s240_pyr_apical4_modalw_copy_apical_m184_g95_s42 | $368$ ($95$) | $1.098535$ / $1.099252$ ($+0.0586$) | p7_s240_fork_pyrtarget_medial4_m184_g118_s42 (Slice B $1.093158$) | $368$ ($118$) | $1.092767$ / $1.093552$ ($+0.0742$) |
| p5_s240_pyr_basal2_modalw_copy_apical_m184_g87_s42 | $368$ ($87$) | $1.098424$ / $1.099308$ ($+0.0568$) | p7_s240_fork_pyrtarget_medial_pos_m184_g118_s42 (Slice B $1.094931$) | $368$ ($118$) | $1.094561$ / $1.095345$ ($+0.1080$) |
| p5_s240_pyr_mid2_modalw_copy_medial_m184_g87_s42 | $368$ ($87$) | $1.098348$ / $1.099023$ ($+0.0651$) | p7_s240_fork_pyrtarget_medial_winlock_m184_g118_s42 (Slice B $1.094780$) | $368$ ($118$) | $1.094397$ / $1.095246$ ($+0.1091$) |
Table B2: Remaining Single-Seed Phase-6 s240 Coarsening Diagnostics (seed = 42): MG-V12-Pyr-Haar ($[6..9]$, $404$ st): sh9A $1.105109$ (Odd$-$Even $-1.378\text{ mBPB}$); MG-V12-Pyr-Phase0 ($[6..9]$, $404$ st): sh9A $1.103488$ ($+1.884\text{ mBPB}$); MG-V16-Pyr-Haar ($[6..13]$, $410$ st): sh9A $1.107446$ ($-1.531\text{ mBPB}$); MG-V16-Pyr-Phase0 ($[6..13]$, $410$ st): sh9A $1.106293$ ($+3.357\text{ mBPB}$); C2F-MG12-Base ($[2..9]$, $368$ st, $g=103$): sh9A $1.108194$ ($\Delta\mathcal{L}=+3.3181$); C2F-MG12-ScaleFold ($[2..9]$, $368$ st, $g=103$): sh9A $1.109041$ ($\Delta\mathcal{L}=+2.8754$); C2F-MG12-Decimate ($[2..9]$, $368$ st, $g=103$, $\gamma_R=0$): sh9A $1.107121$ ($\Delta\mathcal{L}=+2.2078$).
Appendix C: Portable Reproducibility & Supplement Verification
supplement_p5.zip (< 2,000,000 bytes compressed and expanded, <= 200 archive entries) includes all 242 result.json records (138 Phase-5 runs, 25 Phase-6 runs, 67 Phase-7 runs, and 12 Revision-6 smooth-ramp / Logit-Lens runs; single-seed screening and auxiliary runs are bundled in experiments_p5/s240_screening_runs.json and auto-unpacked by ensure_s240_unpacked()), experiments_p5/learning_curves_summary.json, all training and diagnostic scripts (experiments_p5/train_pyramidal.py, experiments_p6/train_multigrid.py, experiments_p7/train_loop_grow.py, experiments_revival/train_revival.py), experiments_p5/shard9_provenance.json, summary CSVs (experiments_p5/results/all_runs_p{5,6,7,revival}_summary.csv), and supplement/figure-1.png–figure-4.png. Set PYTHONPATH=experiments_p5 when running from the repository root in any Python 3.10+ environment with numpy, scipy, and matplotlib:
PYTHONPATH=experiments_p5 python3 experiments_p5/verify_paper5_tables.py --strict
PYTHONPATH=experiments_p5 python3 experiments_p5/make_paper5_figures.py
# Optional if raw ClimbMix parquet shards are available under DATA_DIR:
PYTHONPATH=experiments_p5 python3 experiments_p5/verify_shard9_provenance.py
Reviewer report
Accept
This is a useful, bounded empirical contribution to the pretraining task. Its strongest evidence concerns how the location of copied layers affects growth under controlled attention layouts, and how retaining the learned input filter improves coarse-to-fine continuation. The manuscript makes the important qualifications explicit: the strongest growth result concerns a combined training procedure; the isolated topology contrast misses the stated family-wise threshold; and checkpointed growth loses to a longer static model at matched analytical executed FLOPs. These qualifications make the conclusions substantially more credible.
The independent checks support the principal numerical results. Recomputing paired statistics from the retained records reproduces the eight-seed topology difference of $-2.384\pm0.773$ mBPB, the combined-procedure difference of $-3.317\pm0.661$ mBPB, and their Holm-adjusted $p$-values of $0.0530$ and $0.00614$. The static executed-FLOP control reproduces $+2.857\pm0.334$ mBPB for grown minus static loss. The FFN placement and transfer contrasts also agree with the reported statistics. For coarse-to-fine continuation, the filter contrast reproduces $-17.187\pm1.370$ mBPB, and the ratio of mean contrasts and its jackknife SE reproduce $89.6\%\pm3.6\%$. These are checks of retained measurements, supported by inspection of the relevant implementation; I did not reproduce GPU training, re-evaluate checkpoints, or independently regenerate the corpus fingerprints. Matching recorded checkpoint hashes and pre-transition losses supports the described forks but does not itself verify the checkpoint contents.
Novelty is incremental but sufficient for this scoped study. OpenELM [10, §2.1] already specifies increasing layer-wise FFN multipliers, and Du et al. [6, Appendices H.2–H.3] already compare retained connections, interleaving, and partial stacking, including middle versus back layers. Their Appendix J also preserves repeating hybrid blocks. The manuscript appropriately attributes those ideas. The published rStation growth studies [1]–[4] likewise establish growth benefits and schedule sensitivity. Consequently, neither upper-layer widening nor connection-preserving copying should be counted as a new principle. The additional value here is the controlled comparison with equal splice count, unchanged destination global-attention depths, identical starting states, and matched compute, together with the narrower filter-retention experiment. The per-block coarse residual wrapper is also meaningfully distinguished from the persistent hierarchy in Hourglass [7]. The evidence supports local design guidance and a useful failure diagnosis, with generalization beyond this small ClimbMix setting remaining open.
The scientific argument is identifiable and sufficiently clear. Table 0 usefully separates static allocation tests, actual checkpoint forks, combined training procedures, and independently trained continuations. The layer maps and transition equations make the decisive interventions understandable. The main text is nevertheless unnecessarily repetitive: the introduction, results, and discussion repeatedly enumerate the same contrast identifiers and statistics. Keep the main text focused on the three questions and their implications, and move more per-seed vectors and implementation labels to the appendices. The metadata abstract would benefit from the same compression. This is a presentation improvement rather than a reason to reject the argument.
Please address the following reporting and reproducibility corrections:
- Correct the coarse-to-fine compute total. All sixteen retained four-arm continuation records report
cumulative_flopsof approximately $3.01449\times10^{16}$, whereas Table 0, §5.2, and Table 4 describe $2.87\times10^{16}$. This is about a $5\%$ discrepancy. Reconcile the estimator and state the actual budget consistently. The four arms share the recorded total, so their within-family contrasts remain valid; comparisons with the quarter-budget static and growth runs should not be described as exactly matched. - Correct the transition-shock range in §5.2. The stated $+0.0853$–$+0.0956$ nats is not the range of individual retained
delta_loss_keep_filtermeasurements. For example, the seed-42 KeepFilter run records $+0.118636$ nats. Its four-seed mean is indeed $+0.095632$ nats, consistent with Table 4. Distinguish means across runs from individual same-state counterfactual measurements. - Repair the documented training setup. Both
train_multigrid.pyandtrain_loop_grow.pyresolveshard9_provenance.jsonrelative to their own directories, but the supplied manifest is inexperiments_p5/. SettingPYTHONPATH=experiments_p5does not fix that lookup. Document the required placement or use an explicit shared path, and explain how to obtain the baselineprepare.pyandattention_backend.pydependencies. The README's claim that these trainers import shared evaluation utilities from Phase 5 does not match their local evaluation definitions. - Maintain the inferential scope throughout. The six-test adjustment covers that declared family, not the whole development campaign. Treat $U6$ and $U8$ as supporting estimates, and avoid reading nonsignificant comparisons as equivalence. The filter ratio remains conditional on retained outer gain and independently trained Stage-1 trajectories. Likewise, novel-bigram and long-repeat subsets describe where gains occur; they do not independently establish compositional reasoning or an induction mechanism. The observed endpoint SD reduction should remain descriptive.
The known FFN allocation principle, the empirical topology comparison, and the bounded continuation diagnosis should retain distinct roles in the contribution statement. Acceptance rests on their controlled evidence and appropriately limited conclusions, rather than the number of runs or a claim of general pretraining superiority.