# Apical-Pyramidal FFN Allocation, Interior vs. Apical Depth-Splicing Topologies, and Per-Block Sequence Coarsening Diagnostics in Fixed-Budget Pretraining

## 1. Introduction and Scope of Contribution

Under a fixed pretraining compute budget, increasing static transformer depth from $L_{\text{net}} = 8$ to $12$ or $16$ layers raises the FLOP cost per token and reduces total token exposure. Prior literature on progressive model growth [Chen et al., 2016 [5]; Gong et al., 2019 [8]; Du et al., 2024 [6]], non-uniform layer-wise parameter allocation [DeLighT, Mehta et al., 2021 [11]; OpenELM, Mehta et al., 2024 [10]], and published rStation studies of depth copy-stacking (`general_pretraining_7/v3` [1]), Hadamard modal width/depth morphogenesis (`general_pretraining_24/v4` [2]), non-cloning complementary microbatch braiding (`general_pretraining_26/v2` [3]), and apical depth growth with hashed surface memory (`general_pretraining_28/v1` [4]) have established two foundational precedents: (i) training a shallower seed before expanding to target depth recovers token exposure and beats static training at target depth, and (ii) widening feed-forward networks (FFNs) toward deeper layers improves parameter efficiency over uniform layer widths [10, 11]. In addition, Du et al. [6, Appendices H.2–H.3 and J] compare contiguous stacking (`G_stack`) against interleaved duplication (`G_inter`), examine retained inter-layer connections and partial interior block duplication, and preserve repeating hybrid blocks in Samba; similarly, Nawrot et al. [7] and Dai et al. [9] analyze causal sequence shortening and repeat upsampling in hierarchical transformers.

Building on—and explicitly separating our bounded empirical claims from—those precedents, this **242-run** study (`138` Phase-5 runs, `25` Phase-6 runs, `67` Phase-7 runs, and `12` Revision-6 smooth-ramp and Logit-Lens diagnostic runs across up to 8 seeds `42, 137, 271, 314, 420, 512, 618, 729`, all verified at `wd_scale = 1.0`) evaluates three specific questions under matched compute budgets. Throughout the paper, **arm-level summaries report $\text{Mean} \pm \text{SD}$ across seeds**, whereas **paired contrasts report $\text{Mean }\Delta \pm \text{SE}$** ($df = n - 1$, two-sided paired $t$-tests). To make the experimental structure transparent at a glance, **Table 0** summarizes the primary contrasts, what is held fixed versus varied, and whether each comparison is a static iso-resource test, a bit-identical shared-checkpoint Stage-1 fork, a two-stage growth package comparison, or a matched-configuration continuation.

**Table 0: Summary of Primary Experimental Contrasts, Fixed vs. Varied Factors, and Design Type**

| Axis & Primary Contrast IDs | Design Type & Seed Count | Factors Held Strictly Fixed | Factor Varied | Headline Result ($\text{Mean }\Delta \pm \text{SE}$) |
| :--- | :---: | :--- | :--- | :---: |
| **Axis 1 (FFN Allocation)**:<br>`I1`, `I1b`, `I1c`, `T1`, `T2`, `U0_s8_linear` | Static Iso-Param, Iso-FLOP & Iso-Token ($n=4$) | Depth ($16\text{L}$ or $8\text{L}$), total $\sum r_l$, params, FLOPs/tok, steps, tokens, optimizer | Vertical FFN width profile: step-apical `Pyr40`, smooth `Linear40` [10], vs. `Basal40` / `Unif` | `I1`: **$-3.799 \pm 0.201\text{ mBPB}$** (`Slice A`)<br>`I1b`: **$-4.328 \pm 0.268\text{ mBPB}$**<br>`T1`: **$-3.716 \pm 0.497$**; `T2`: **$-3.143 \pm 0.594$** |
| **Axis 2A (Pure Splice Topology)**:<br>`U6_true_8s` (`Unif4d-Medial4` vs. `Apical`) | **Bit-Identical Shared Stage-1 Checkpoint Fork** ($n=8$, $B=64$) | Exact Stage-1 `.pt` weights (`SHA-256`), $r=4^{12}$, $1$ splice, $0$ window flips, `L` at $\{3,7,11\}$, exec FLOPs | Duplicating interior `SSSL` period $[2..5]$ (splice $5\to 2$) vs. apical $[4..7]$ ($7\to 4$) | `Slice B`: **$-2.384 \pm 0.773\text{ mBPB}$**<br>($p=0.0177, p_{\text{Holm6}}=0.0530$, $7/8$ seeds) |
| **Axis 2B (PyrTarget Splicing)**:<br>`U2_true`, `U4_true`, `U5_true` | **Bit-Identical Shared Stage-1 Checkpoint Fork** ($n=4$, $B=64$) | Exact Stage-1 `PyrTarget` `.pt` weights (`SHA-256`), $r=(2^2,4^8,6^2)$, nominal & exec FLOPs | `Medial4` ($1$-spl, $0$-flip, $\{3,7,11\}$) vs. `Medial-Pos` ($2$-spl, $2$-flip) vs. `WinLock` ($\{3,5,11\}$) | `U2_true`: **$-2.432 \pm 0.240\text{ mBPB}$**<br>`U4_true`: **$-2.375 \pm 0.182\text{ mBPB}$**<br>`U5_true`: $-0.057 \pm 0.081\text{ mBPB}$ |
| **Axis 2C (Two-Stage Packages)**:<br>`U7_true_8s`, `U8_true_8s`, `U0_true_8s`, `U0b_true_8s` | **Two-Stage Growth Package Comparison** ($n=8$, $B=64$) | Steps ($728$ vs. $728$ or $953$), nominal FLOPs ($1.195\text{e}17$); exact exec FLOPs ($1.338\text{e}17$) within $12\text{L}$ | Two-stage `PyrTarget-Medial4` vs. `Unif4d-Apical` (`U7`), `Unif4d-Medial4` (`U8`), or `Static-8L` (`U0`) | `U7_true_8s`: **$-3.317 \pm 0.661$** ($p_{\text{Holm6}}=0.00614$)<br>`U0_true_8s`: **$-4.395 \pm 0.368$** ($p_{\text{Holm6}}=0.000040$)<br>`U8_true_8s`: $-0.933 \pm 0.386$ ($p_{\text{Holm6}}=0.0923$) |
| **Axis 2D (Exec-FLOP Control)**:<br>`U0_exec_true`, `U0_s8_extra`, `M1` | **Analytical Iso-Executed-FLOP Control** ($n=4$, $B=64$ / $B=32$) | Analytical executed FLOPs ($1.3384\text{e}17$ vs. $1.3387\text{e}17$ at $B=64$; $1.1946\text{e}17$ at $B=32$) | Checkpointed $12\text{L}$ growth ($728$ st, $+12.04\%$ recomp) vs. uncheckpointed $8\text{L}$ ($1{,}068$ st) | `U0_exec_true`: **$+2.857 \pm 0.334\text{ mBPB}$** (`8L` leads overall; `12L` leads `bi_l` by **$-16.073$**) |
| **Axis 3 (Coarse-to-Fine C2F)**:<br>`C2F_1`, `C2F_2_pure_filter`, `C2F_3_outer_gain`, `C2F_4_gain_form` | **Matched-Configuration Two-Stage Continuation** ($n=4$, $g=190$) | Coarse $[6..9]$ Stage-1 setup, seed, steps ($366$), exact C2F FLOPs ($3.0145\text{e}16$); outer gain $\bar{\beta}_l$ in `C2F_2_pure_filter` | Retaining 2-tap Haar input filter (`KeepFilter4`) vs. `OuterGainNoFilter4`, `ScaleFold4`, `Unfolded4` | `C2F_2_pure_filter`: **$-17.187 \pm 1.370\text{ mBPB}$** ($p=0.00109$; $89.6\% \pm 3.6\%$ of `C2F_1`) |

Specifically, our three bounded empirical contributions and their scope limits are:
1. **Vertical FFN Capacity Allocation: Step-Apical vs. OpenELM Smooth Linear Ramp, Transfer, and Exploratory Sub-Layer Profiles (§3)**: Rather than claiming a new architectural principle beyond DeLighT [11] and OpenELM [10], we test how vertical FFN distribution behaves under fixed-FLOP pretraining across backbones and widths, comparing step-apical `Static-16L-Pyr40` ($r = (2^{14}, 6, 6)$) against a budget-matched smooth linear ramp (`Static-16L-Linear40`, $r = (1^4, 2^4, 3^4, 4^4)$, $\sum r_l = 40$; Table 1B). At 16 layers under exact iso-parameters ($79.69\text{M}$), iso-FLOPs/token ($377.49\text{M}$), and iso-exposure ($316.67\text{M}$ tokens), step-apical placement beats basal placement ($r = (6, 6, 2^{14})$) by **$-3.799 \pm 0.201\text{ mBPB}$** on `Slice A` ($p = 0.000324$; `I1`), persisting on an **all-global `LLLL` attention backbone** (**$-3.716 \pm 0.497\text{ mBPB}$** on `Slice B`, $p = 0.00494$; `T1`) and at **reduced width $d = 384, H = 3$** (**$-3.143 \pm 0.594\text{ mBPB}$**, $p = 0.0132$; `T2`). The smooth linear OpenELM ramp beats basal placement by **$-4.328 \pm 0.268\text{ mBPB}$** ($p = 0.000516$; `I1b`) and uniform $2d$ (`Static-16L-Unif2d`) by **$-1.656 \pm 0.337\text{ mBPB}$** ($p = 0.0161$; `I1d`), performing comparably to `Pyr40` ($-0.528 \pm 0.368\text{ mBPB}$, $p = 0.246$; `I1c`), while linear ramping at 8 layers is near-neutral ($-0.400 \pm 0.324\text{ mBPB}$, $p = 0.304$; `U0_s8_linear`). Uncalibrated sub-layer Logit-Lens and `c_proj` SVD stable-rank profiles (Table 1C) are reported strictly as **exploratory descriptive associations** [Belrose et al., 2023 [12]].
2. **Native Unsplit `4x64` (`B=64`) Depth Splicing: 8-Seed Shared-Checkpoint vs. Two-Stage Contrasts and Analytical Iso-Executed-FLOP Accounting (§4)**: Under native 64-sequence microbatching ($524{,}288$ tok/step), selective activation checkpointing of the 6 interior blocks $2 \le l < 8$ at $12\text{L}$ adds $+15.76\%$ analytical executed FLOPs to static $12\text{L}$ ($1.3840 \times 10^{17}$) and $+12.04\%$ to grown $12\text{L}$ ($1.3384 \times 10^{17}$ vs. $1.1946 \times 10^{17}$ nominal FLOPs; Table 2A). Across **8 paired seeds** with 6-test Holm-Bonferroni correction (`p_Holm6`) on `Slice B` (Table 2B):
   - On the **true shared-checkpoint fork** from bit-identical `Unif4d` Stage-1 checkpoints at $g = 225$ with destination global-attention depths held fixed at $\{3, 7, 11\}$, interior period-preserving growth (`G_stack-Unif4d-Medial4`, $[0..5,2..5,6,7]$, splice $5\to 2$) halves the same-batch transition shock relative to contiguous apical `G_stack` (`G_stack-Unif4d-Apical`, $[0..7,4..7]$, splice $7\to 4$: $+0.0688$ vs. $+0.1286\text{ nats}$), reduces cross-seed `Slice B` SD from $2.59$ to $0.99\text{ mBPB}$, and improves `Slice B` BPB by **$-2.384 \pm 0.773\text{ mBPB}$** ($p = 0.0177, p_{\text{Holm6}} = 0.0530$, $7/8$ seeds negative; `U6_true_8s`). In **two-stage package comparisons**, `G_stack-PyrTarget-Medial4` beats `G_stack-Unif4d-Apical` by **$-3.317 \pm 0.661\text{ mBPB}$** ($p = 0.00153, \mathbf{p_{\text{Holm6}} = 0.00614}$, $8/8$ seeds negative; `U7_true_8s`) and `G_stack-Unif4d-Medial4` across both stages by **$-0.933 \pm 0.386\text{ mBPB}$** ($p = 0.0462, p_{\text{Holm6}} = 0.0923$, $6/8$ seeds negative; `U8_true_8s`). Under 6-test Holm adjustment, `U7_true_8s`, `U0_true_8s` (**$-4.395 \pm 0.368\text{ mBPB}$**, $p_{\text{Holm6}} = 0.000040$), and `U0b_true_8s` (**$-3.461 \pm 0.328\text{ mBPB}$**, $p_{\text{Holm6}} = 0.000075$) are family-wise significant at $\alpha = 0.05$, while `U6_true_8s` and `U8_true_8s` sit marginally above the 6-test Holm threshold.
   - When uncheckpointed `Static-8L-Unif4d-True4x64` is granted the same $+12.04\%$ analytical executed FLOPs by running for **1,068 steps** (`Static-8L-Unif4d-m1068-True4x64`, $1.3387 \times 10^{17}$ FLOPs), the extra 115 steps reduce 8L `Slice B` BPB by **$-7.058 \pm 0.043\text{ mBPB}$** (`U0_s8_extra`), so `Static-8L-m1068` leads checkpointed 728-step `G_stack-PyrTarget-Medial4` overall by **$+2.857 \pm 0.334\text{ mBPB}$** (`U0_exec_true`, $p = 0.00336$) while trailing by **$-16.073\text{ mBPB}$** on long-distance bigrams (`bi_l`). Under uncheckpointed `Split8x32` ($0\%$ recomputation), 12L growth beats static 8L at exact analytical iso-executed FLOPs (`-4.182 mBPB`).
3. **Matched-Configuration 4-Seed Continuation Test of Filter Retention in Coarse-to-Fine Transitions (§5)**: Across 4 seeds on coarse blocks $[6..9]$ at $g = 190$ ($366$ steps, exact C2F compute total $3.0145 \times 10^{16}\text{ FLOPs}$ across all four continuation arms; Table 4), comparing **`C2F-MG12-KeepFilter4-g190`** against **`C2F-MG12-OuterGainNoFilter4-g190`** (which preserves the exact outer block gain $x + \bar{\beta}_l[A(x) + M(x + A(x))]$ while dropping only the 2-tap input filter) yields **$-17.187 \pm 1.370\text{ mBPB}$** on `sh9A` ($p = 0.00109$, $4/4$ seeds negative; `C2F_2_pure_filter`). Conditional on the retained-gain setting, the ratio of estimated mean contrasts `C2F_2_pure_filter / C2F_1` (`-17.187 / -19.188 mBPB`) is **$89.6\% \pm 3.6\%$** (jackknife SE; per-seed range $84.2\%\text{--}99.5\%$), showing that transition recovery is dominated by retaining the 2-tap causal input filter rather than outer vs. `c_proj`-folded residual scaling (`C2F_4_gain_form`: $-0.600 \pm 0.308\text{ mBPB}$, $p = 0.147$).

---

## 2. Architecture, Nominal vs. Executed FLOP Accounting, and Evaluation Protocol

### 2.1 Transformer Backbone and Depth-Expansion Topologies

Every baseline and growth model is a decoder-only causal Transformer operating at context length $L = 2{,}048$, vocabulary size $V = 8{,}192$, default width $d = 512$ ($H = H_{kv} = 4$ heads, $d_h = 128$; with $d = 384, H = 3, d_h = 128$ in transfer test `T2`), RoPE ($\theta = 10{,}000$), and parameter-free RMSNorm (`F.rms_norm`).
1. **Two-Matrix Squared-ReLU MLP**: Layer $l$ with expansion ratio $r_l$ computes $\operatorname{MLP}_l(u) = W_{\mathrm{proj},l}(\max(0, W_{\mathrm{fc},l} u))^2$ with bias-free $W_{\mathrm{fc},l} \in \mathbb{R}^{(r_l d)\times d}$ and $W_{\mathrm{proj},l} \in \mathbb{R}^{d\times (r_l d)}$ ($2 r_l d^2$ parameters).
2. **Attention Window Patterns (`SSSL`, `LLLL`, and `medial_winlock`)**: Under default `SSSL` positional assignment, layers $l \equiv 3 \pmod 4$ (and $l = L_{\text{net}}-1$) use full causal context $w_l = 2{,}048$ (`L`) and all other layers use sliding window $w_l = 1{,}024$ (`S`). Under `--window-pattern LLLL` (transfer test `T1`), all layers use $w_l = 2{,}048$. Under `--topology medial_winlock` (`experiments_p7/train_loop_grow.py`, lines 1244–1250), destination block $d \in \{0,\dots,11\}$ inherits its parent's Stage-1 window $w_{\text{dst}}[d] = w_{\text{src}}[\text{block\_map}[d]]$, giving `[S,S,S,L, S,L,S,S, S,S,S,L]` ($3\text{L} + 9\text{S}$).
3. **Value-Embedding Gates and Residual Scalars**: Odd layers $l \in \{1, 3, \dots\}$ include a token-embedding table `value_embeds` ($V \times d$) gated by `ve_gate` $\in \mathbb{R}^{H_{kv}\times 32}$ ($128$ parameters per odd layer; $96$ when $H_{kv}=3$): $v_l \leftarrow v_l + 2\,\sigma(W_{\mathrm{ve},l} x_{:,:,:32}) \odot \operatorname{VE}_l(\text{idx})$. Pre-block inputs are scaled by learnable per-layer scalars $\tilde{x}_l = \lambda_{\mathrm{resid},l} x_l + \lambda_{x0,l} x_0$ (`resid_lambdas` init $1.0$, `x0_lambdas` init $0.1$). Across all 242 runs, `wd_scale = 1.0` and Muon matrix weight decay decays linearly as $\text{wd}(\text{progress}) = 0.2(1 - \text{progress})$.

Table 1A summarizes the exact `block_map`, destination global-attention (`L`) depths, window flips relative to parent blocks, non-native residual splice interfaces, and FFN ratio schedules in `expand_model_generalized()`. Note the exact window-flip count on `medial`: mapping parent `[S_0,S_1,S_2,L_3,S_4,S_5,S_6,L_7]` through `[0,1,2,3,2,3,4,5,4,5,6,7]` yields parent windows `[S,S,S,L, S,L,S,S, S,S,S,L]`, whereas positional `SSSL` assigns `[S,S,S,L, S,S,S,L, S,S,S,L]`—flipping parent window types at **exactly 2 destination blocks** ($d = 5: L_3 \to S_5$ and $d = 7: S_5 \to L_7$).

**Table 1A: Implemented Depth-Expansion Topologies and FFN Profiles (`expand_model_generalized`)**

| Topology Flag | Transition | Source Block Map `block_map[0 .. n_dst-1]` | Stage-2 `L` ($2048$) Depths & Window Flips vs. Parent | Non-Native Splice Interfaces $(u \to v)$ | FFN Profile $r_{\text{src}} \to r_{\text{dst}}$ |
| :--- | :---: | :--- | :--- | :--- | :--- |
| **`medial4`** | $8\to 12\text{L}$ | `[0,1,2,3,4,5, 2,3,4,5, 6,7]` | `L` at $\{3, 7, 11\}$; **$0$ flips** | **$1$ interior splice** ($5 \to 2$ at $5\to 6$) | `PyrTarget`: $(2^2,4^4,6^2)\to(2^2,4^8,6^2)$<br>`Unif4d`: $4^8 \to 4^{12}$ |
| **`medial`** *(Positional)* | $8\to 12\text{L}$ | `[0,1,2,3, 2,3, 4,5, 4,5, 6,7]` | `L` at $\{3, 7, 11\}$; **$2$ blocks flip** ($d=5: L_3\to S_5$; $d=7: S_5\to L_7$) | **$2$ interior splices** ($3 \to 2$ at $3\to 4$; $5 \to 4$ at $7\to 8$) | `PyrTarget`: $(2^2,4^4,6^2)\to(2^2,4^8,6^2)$<br>`Unif4d`: $4^8 \to 4^{12}$ |
| **`medial_winlock`** | $8\to 12\text{L}$ | `[0,1,2,3, 2,3, 4,5, 4,5, 6,7]` | `L` at $\{3, 5, 11\}$; **$0$ flips** ($w_{\text{dst}}[d]=w_{\text{src}}[\text{map}[d]]$) | **$2$ interior splices** ($3 \to 2$ at $3\to 4$; $5 \to 4$ at $7\to 8$) | `PyrTarget`: $(2^2,4^4,6^2)\to(2^2,4^8,6^2)$ |
| **`apical`** (`G_stack`) | $8\to 12\text{L}$ | `[0,1,2,3,4,5,6,7, 4,5,6,7]` | `L` at $\{3, 7, 11\}$; **$0$ flips** | **$1$ apical splice** ($7 \to 4$ at $7\to 8$) | `Unif4d`: $4^8 \to 4^{12}$<br>`Pyr-Apical2`: $(2^6,4^2)\to 4^{12}$ |
| **`medial4`** *(16L 2/3-stg)* | $8\to 16\text{L}$ / $8\to 12\to 16\text{L}$ | 2-stg: `[0..5, 2..5, 2..5, 6,7]`<br>3-stg S3: `[0..9, 6..9, 10,11]` | `L` at $\{3, 7, 11, 15\}$; **$0$ flips** | 2-stg: $5\to 2$ ($\times 2$)<br>3-stg: $5\to 2$, then $9\to 6$ | $(2^6,6^2)\to(2^{14},6^2)$ /<br>$(2^6,6^2)\to(2^{10},6^2)\to(2^{14},6^2)$ |

### 2.2 Analytical Nominal Model FLOPs vs. Analytical Executed Activation-Checkpointing FLOPs and Batch Regimes

1. **Analytical Nominal Forward+Backward FLOPs (`GPT.estimate_flops`)**: Excluding embedding table lookups and $2 L_{\text{net}}$ residual scalars, analytical nominal matrix+attention FLOPs per token for dense ($s_l = 1$) models are:
   $$F_{\text{nom}}\!\left(L_{\text{net}}, \{r_l\}\right) = 6 \left(V d + \sum_{l=0}^{L_{\text{net}}-1} \left[(4 + 2 r_l) d^2 + 32 H_{kv}\,\mathbb{I}[l \text{ odd}]\right]\right) + \sum_{l=0}^{L_{\text{net}}-1} 12 H d_h w_l$$
   At $d=512, H=4$ (`SSSL`), this gives **$239{,}078{,}400\text{ FLOPs/tok}$** for 8L ($\sum r_l = 32$, $50.33\text{M}$ params, identical for `Static-8L-Unif4d` and `Static-8L-Linear32`), **$346{,}034{,}688\text{ FLOPs/tok}$** for 12L ($\sum r_l = 48$, $71.30\text{M}$ params, identical for positional `SSSL` and `medial_winlock`), **$352{,}327{,}680\text{ FLOPs/tok}$** for `Static-16L-Unif2d` ($\sum r_l = 32$, $75.50\text{M}$ params), and **$377{,}493{,}504\text{ FLOPs/tok}$** for 16L `Pyr40` / `Basal40` / `Linear40` ($\sum r_l = 40$, $79.69\text{M}$ params). In the transfer arms, 16L all-global `LLLL` ($w_l = 2{,}048\;\forall l$, $d=512, H=4$) uses **$452{,}990{,}976\text{ FLOPs/tok}$** ($79.69\text{M}$ params), and 16L reduced-width `d384` ($d=384, H=3$, `SSSL`) uses **$240{,}652{,}032\text{ FLOPs/tok}$** ($52.69\text{M}$ params). Full-budget nominal schedules target $953$ steps of 8L at $524{,}288$ tok/step: $\mathcal{B}_{\text{nom}} = 499{,}646{,}464 \times 239{,}078{,}400 = \mathbf{1.194547 \times 10^{17}}\text{ analytical nominal FLOPs}$ (matched within $0.1\%$ nominal across full-budget `SSSL` arms).
2. **Exact Activation-Checkpointing Specification (`2 <= l < 8`) and Analytical Executed FLOP Accounting**:
   - In `experiments_p5/train_pyramidal.py` and `experiments_revival/train_revival.py` (`use_half_mb = True`), `262k`-tok/step runs execute $4\times 32$ split microbatches (`4x32`) and `524k`-tok/step runs execute $8\times 32$ split microbatches (`Split8x32`), with **zero activation checkpointing** (`grad_ckpt_12l = False`), so **analytical executed FLOPs equal analytical nominal FLOPs** across all Phase-5, Phase-6, `Split8x32`, `Static-8L-Unif4d-True4x64`, `Static-8L-Linear32-True4x64`, `Static-8L-Unif4d-m1068-True4x64`, `Static-16L-Linear40`, `LLLL`, and `d384` runs.
   - In `experiments_p7/train_loop_grow.py` (line 801), native unsplit `True4x64` (`device_batch_size = 64`, `microbatch_sequences = 64`, `4x64` $= 524{,}288$ tok/step) fits 12-layer training within $19.0\text{--}22.8\text{ GB}$ VRAM via `--grad-ckpt-12l`, which wraps `torch.utils.checkpoint.checkpoint(..., use_reentrant=False)` around the **6 interior blocks $2 \le l < 8$** whenever `len(transformer.h) >= 12` (`if use_ckpt and (len(self.transformer.h) >= 12) and (2 <= i < 8)`).
   - For all 12-layer models tested under `True4x64`, blocks $2 \le l < 8$ ($l \in \{2,3,4,5,6,7\}$) have $r_l = 4$ and contain **$4\text{S} + 2\text{L}$** windows (both under positional `SSSL`, where `L` is at $\{3, 7\}$, and under `medial_winlock`, where `L` is at $\{3, 5\}$). Recomputing one forward pass (`2 * numel` matrix FLOPs plus `4 * H * d_h * w_l` attention FLOPs) across those 6 blocks adds an analytical **$54{,}525{,}952\text{ FLOPs/token}$** ($15.76\%$ of $346{,}034{,}688$):
     - **`Static-8L-Unif4d-True4x64` & `Static-8L-Linear32-True4x64`** ($953$ steps, uncheckpointed): Analytical Nominal $= \mathbf{1.1945 \times 10^{17}}$, Recomputation $= 0$, **Analytical Executed $= 1.1945 \times 10^{17}$ FLOPs** (`train_t = 1090.8–1095.1s`, peak VRAM $22{,}804.5\text{ MB}$).
     - **`Static-8L-Unif4d-m1068-True4x64`** ($1{,}068$ steps, uncheckpointed, $559.94\text{M}$ tokens): Analytical Nominal $= \mathbf{1.3387 \times 10^{17}}$, Recomputation $= 0$, **Analytical Executed $= 1.3387 \times 10^{17}$ FLOPs** (`train_t = 1228.7s`, matching the analytical executed FLOPs of 728-step checkpointed 12L growth within $0.02\%$).
     - **`Static-12L-PyrTarget-True4x64`** ($659$ checkpointed 12L steps): Analytical Nominal $= 1.1956 \times 10^{17}$, Recomputation $= +0.1884 \times 10^{17}$ ($+15.76\%$), **Analytical Executed $= 1.3840 \times 10^{17}$ FLOPs** (`train_t = 1215.6s`, peak VRAM $19{,}017.6\text{ MB}$).
     - **All five 12L Grown `True4x64` Arms** ($225$ uncheckpointed 8L steps + $503$ checkpointed 12L steps): Analytical Nominal $= 1.1946 \times 10^{17}$, Recomputation $= +0.1438 \times 10^{17}$ ($+12.04\%$), **Analytical Executed $= 1.3384 \times 10^{17}$ FLOPs** (`train_t = 1190.9–1198.7s`).
     Thus all five grown 12L `True4x64` arms have **strictly identical analytical nominal and executed FLOPs** to one another, use **$-3.29\%$ fewer analytical executed FLOPs** than `Static-12L-PyrTarget-True4x64`, and are directly compared against both nominal-matched `Static-8L-Unif4d-True4x64` ($953$ steps) and analytical-executed-matched `Static-8L-Unif4d-m1068-True4x64` ($1{,}068$ steps).

### 2.3 Evaluation Splits and Provenance

All runs exclude `shard_00009.parquet` from training (`shard_00000..00008`, $761{,}856$ documents). `verify_shard9_provenance.py` removes all $26$ training-overlapping documents, $2$ validation-overlapping documents, and $2$ internal duplicates from `shard_00009.parquet`, leaving $84{,}963$ strictly disjoint documents (`shard9_provenance.json`):
- **`Slice A` (`sh9A`, development-conditioned)**: First $320$ batches ($20{,}971{,}520$ target tokens; $\approx 20.95\text{M}$ normal tokens) packed from `kept_docs[0:37760]` (`SHA-256: ef81f8c0b5dcdcbd...`), used alongside `shard_06542` `val_bpb` and `holdout_bpb` ($20{,}971{,}520$ target tokens each) during Phase-5 development.
- **`Slice B` (confirmation set)**: Second $320$ batches ($20{,}971{,}520$ target tokens; $\approx 20.95\text{M}$ normal tokens) packed from `kept_docs[37760:84963]` (`SHA-256: 6dfb9ec04a23ba7b...`), created after the Phase-5 architecture screen and evaluated on our confirmation and transfer arms. We also log `confirm_bpb` ($7{,}077{,}888$ target tokens, batches $641\text{--}748$ of `shard_06542`) as a cross-slice check, noting it was previously used in rStation [3, 4]. Descriptive input–target bigram and unigram token-class decompositions are reported in Appendix A.

---

## 3. Vertical FFN Capacity Allocation: Step-Apical vs. OpenELM Smooth Linear Ramp, Transfer, and Exploratory Sub-Layer Profiles

![Figure 1: Apical-Pyramidal Capacity Allocation, Cross-Backbone/Width Transfer, and Native 4x64 Depth Growth](supplement/figure-1.png)

DeLighT [11] and OpenELM [10, §2.1] established that ramping FFN intermediate width from narrower lower layers to wider upper layers improves parameter efficiency over uniform Transformers. Rather than claiming a new architectural principle, we evaluate how vertical FFN distribution behaves under a strict fixed-FLOP pretraining budget across two questions: (1) Does concentrating FFN width at the apex (`Pyr40`, $r = (2^{14}, 6, 6)$) versus the base (`Basal40`, $r = (6, 6, 2^{14})$) persist under homogeneous all-global attention (`LLLL`) and at reduced model width ($d = 384, H = 3$), and how does our two-step apical schedule compare against a budget-matched **smooth linear OpenELM ramp** (`Static-16L-Linear40`, $r = (1^4, 2^4, 3^4, 4^4)$, $\sum r_l = 40$, and 8-layer `Static-8L-Linear32`, $r = (2,2,3,3,5,5,6,6)$, $\sum r_l = 32$)? (2) What descriptive patterns appear in uncalibrated sub-layer Logit-Lens readouts and `c_proj` weight SVD stable ranks across depth quartiles? Table 1B reports the 4-seed arm summaries ($\text{Mean} \pm \text{SD}$) and paired contrasts ($\text{Mean }\Delta \pm \text{SE}$), and Table 1C reports exploratory sub-layer Logit-Lens and SVD stable-rank measurements on $524{,}288$ validation tokens.

**Table 1B: Iso-Parameter / Iso-Exposure FFN Capacity Allocation, Smooth Linear OpenELM Ramp, and Cross-Backbone / Cross-Width Transfer (`seeds = 42, 137, 271, 314`)**

| Arm ($\text{Mean} \pm \text{SD}$) or Paired Contrast ($\text{Mean }\Delta \pm \text{SE}$) | Backbone & Params | Steps & Tokens (Exec. FLOPs, 0% recomp) | **`Slice B` BPB** (or $\Delta\text{mBPB}$) | **`Slice A` (`sh9A`) BPB** (or $\Delta\text{mBPB}$) | `val_bpb` / `holdout_bpb` (or $\Delta\text{mBPB}$) | Paired 95% $t$-CI, $t$-stat ($df=3$), $p_{\text{raw}}$, & Seeds |
| :--- | :---: | :---: | :---: | :---: | :---: | :--- |
| **`Static-16L-Linear40`** ($r=1^4,2^4,3^4,4^4$, OpenELM ramp) | $d=512, H=4$ `SSSL`<br>$79.69\text{M}$ | $1{,}208$ steps<br>$316.67\text{M}$ ($1.195\text{e}17$) | — | **$0.987598 \pm 0.001086$** | $0.987008 \pm 0.001136$ /<br>$0.986120 \pm 0.001101$ | Train $1124.7\text{s}$, VRAM $18{,}940\text{ MB}$ |
| **`Static-16L-Pyr40`** ($r=2^{14},6,6$, step-apical) | $d=512, H=4$ `SSSL`<br>$79.69\text{M}$ | $1{,}208$ steps<br>$316.67\text{M}$ ($1.195\text{e}17$) | — *(see `Basal40`)* | $0.988126 \pm 0.000833$ | $0.987565 \pm 0.000750$ /<br>$0.986664 \pm 0.000807$ | Train $1122.0\text{s}$, VRAM $18{,}940\text{ MB}$ |
| **`Static-16L-Basal40`** ($r=6,6,2^{14}$, step-basal) | $d=512, H=4$ `SSSL`<br>$79.69\text{M}$ | $1{,}208$ steps<br>$316.67\text{M}$ ($1.195\text{e}17$) | $0.991478 \pm 0.000693$ | $0.991926 \pm 0.000642$ | $0.991377 \pm 0.000668$ /<br>$0.990533 \pm 0.000696$ | Train $1118.5\text{s}$, VRAM $18{,}940\text{ MB}$ |
| **`Static-16L-Pyr40-LLLL`** ($r=2^{14},6,6$) | **$d=512, H=4$ `LLLL`**<br>$79.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($1.344\text{e}17$) | **$0.992141 \pm 0.001064$** | $0.992608 \pm 0.001067$ | $0.992098 \pm 0.001064$ /<br>$0.991136 \pm 0.001078$ | Train $1082.1\text{s}$, VRAM $18{,}940\text{ MB}$ |
| **`Static-16L-Basal40-LLLL`** ($r=6,6,2^{14}$) | **$d=512, H=4$ `LLLL`**<br>$79.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($1.344\text{e}17$) | $0.995858 \pm 0.000524$ | $0.996294 \pm 0.000502$ | $0.995760 \pm 0.000492$ /<br>$0.994814 \pm 0.000545$ | Train $1082.1\text{s}$, VRAM $18{,}940\text{ MB}$ |
| **`Static-16L-Pyr40-d384`** ($r=2^{14},6,6$) | **$d=384, H=3$ `SSSL`**<br>$52.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($0.714\text{e}17$) | **$1.020963 \pm 0.001760$** | $1.021444 \pm 0.001765$ | $1.020836 \pm 0.001728$ /<br>$1.020104 \pm 0.001780$ | Train $725.0\text{s}$, VRAM $14{,}375\text{ MB}$ |
| **`Static-16L-Basal40-d384`** ($r=6,6,2^{14}$) | **$d=384, H=3$ `SSSL`**<br>$52.69\text{M}$ | $1{,}132$ steps<br>$296.75\text{M}$ ($0.714\text{e}17$) | $1.024106 \pm 0.001700$ | $1.024505 \pm 0.001713$ | $1.023913 \pm 0.001663$ /<br>$1.023140 \pm 0.001776$ | Train $725.2\text{s}$, VRAM $14{,}375\text{ MB}$ |
| **`Static-8L-Linear32-True4x64`** ($r=2,2,3,3,5,5,6,6$) | $d=512, H=4$ `SSSL`<br>$50.33\text{M}$ ($B=64$) | $953$ steps<br>$499.65\text{M}$ ($1.195\text{e}17$) | $0.995007 \pm 0.001536$ | $0.995563 \pm 0.001541$ | $0.994894 \pm 0.001522$ /<br>$0.994121 \pm 0.001534$ | Train $1095.1\text{s}$, VRAM $22{,}805\text{ MB}$ |
| **`I1`: `Pyr40` vs. `Basal40` (`SSSL`, $d=512$)** | Exact Iso-Param & Iso-Tok | $316.67\text{M}$ tok | — | **$-3.799 \pm 0.201$** | $-3.812$ / $-3.869$ | CI $[-4.440, -3.159]$, $t = -18.89$, **$p = 0.000324$** ($4/4$)<br>`[-3.705, -4.141, -3.270, -4.082]` |
| **`I1b`: `Linear40` vs. `Basal40` (`SSSL`, $d=512$)** | Exact Iso-Param & Iso-Tok | $316.67\text{M}$ tok | — | **$-4.328 \pm 0.268$** | $-4.370$ / $-4.413$ | CI $[-5.181, -3.475]$, $t = -16.15$, **$p = 0.000516$** ($4/4$)<br>`[-5.109, -4.187, -4.124, -3.891]` |
| **`I1c`: `Linear40` vs. `Pyr40` (`SSSL`, $d=512$)** | Exact Iso-Param & Iso-Tok | $316.67\text{M}$ tok | — | $-0.528 \pm 0.368$ | $-0.557$ / $-0.544$ | CI $[-1.699, +0.642]$, $t = -1.44$, $p = 0.246$ ($3/4$)<br>`[-1.404, -0.047, -0.854, +0.191]` |
| **`I1d`: `Linear40` vs. `Unif2d` (`SSSL`, $d=512$)** | $79.69\text{M}$ vs. $75.50\text{M}$ | $316.7\text{M}$ vs. $339.2\text{M}$ | — | **$-1.656 \pm 0.337$** | $-1.639$ / $-1.679$ | CI $[-2.728, -0.583]$, $t = -4.91$, **$p = 0.0161$** ($4/4$)<br>`[-1.864, -1.075, -2.517, -1.166]` |
| **`T1`: `Pyr40-LLLL` vs. `Basal40-LLLL` (All-Global)** | Exact Iso-Param & Iso-Tok | $296.75\text{M}$ tok | **$-3.716 \pm 0.497$** | $-3.686$ ($p=0.00557$) | $-3.661$ / $-3.678$ | `Slice B` CI $[-5.297, -2.136]$, $t = -7.48$, **$p = 0.00494$** ($4/4$)<br>`[-3.975, -4.538, -2.272, -4.080]` |
| **`T2`: `Pyr40-d384` vs. `Basal40-d384` ($d=384, H=3$)** | Exact Iso-Param & Iso-Tok | $296.75\text{M}$ tok | **$-3.143 \pm 0.594$** | $-3.061$ ($p=0.0159$) | $-3.077$ / $-3.037$ | `Slice B` CI $[-5.033, -1.253]$, $t = -5.29$, **$p = 0.0132$** ($4/4$)<br>`[-4.810, -2.448, -3.156, -2.158]` |
| **`I2`: `Static-16L-Basal40` vs. `Static-16L-Unif2d`** | $79.69\text{M}$ vs. $75.50\text{M}$ | $316.7\text{M}$ vs. $339.2\text{M}$ | — | **$+2.672 \pm 0.372$** | $+2.731$ / $+2.734$ | CI $[+1.489, +3.855]$, $t = +7.19$, **$p = 0.00555$** ($0/4$)<br>`[+3.245, +3.112, +1.607, +2.726]` |
| **`I3` / `I4`: 12L `PyrTarget` vs. `Unif4d` (Static / Grown)** | Exact Iso-Param ($71.30\text{M}$) | $345.5\text{M}$ / $381.7\text{M}$ | — | $-1.572 \pm 0.839$ /<br>**$-0.532 \pm 0.115$** | $-1.567$ / $-0.510$ (`val`) | `I3`: $t = -1.87, p = 0.158$; `I4`: $t = -4.64$, **$p = 0.0188$** |
| **`U0_s8_linear`: `Static-8L-Linear32` vs. `Static-8L-Unif4d`** | Exact Iso-Param ($50.33\text{M}$) | $499.65\text{M}$ tok ($B=64$) | $-0.400 \pm 0.324$ | $-0.424$ ($p=0.290$) | $-0.439$ / $-0.442$ | `Slice B` CI $[-1.430, +0.630]$, $t = -1.24, p = 0.304$ ($3/4$)<br>`[-0.826, -0.565, -0.766, +0.557]` |

**Table 1C: Exploratory Uncalibrated Sub-Layer Logit-Lens BPB Change ($\Delta\text{BPB} = \text{BPB}_{\text{in}} - \text{BPB}_{\text{out}}$) & `c_proj` Weight SVD Stable Rank on $524{,}288$ Validation Tokens (`seed = 42`, matches `paper5_tables_verified.txt` lines 177–182 verbatim)**

| Model & Layer-Width Schedule | Quartile `Q1` ($L_{0..3}$ or $L_{0..1}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | Quartile `Q2` ($L_{4..7}$ or $L_{2..3}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | Quartile `Q3` ($L_{8..11}$ or $L_{4..5}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) | Quartile `Q4` ($L_{12..15}$ or $L_{6..7}$: $r_{\text{sum}}$, $\Delta\text{BPB}_{\text{MLP}}$, $\Delta\text{BPB}_{\text{MLP}}/r$, $\Delta\text{BPB}_{\text{Attn}}$, $\text{SR}_{\text{proj}}$) |
| :--- | :---: | :---: | :---: | :---: |
| **`Static-16L-Linear40`** ($1^4,2^4,3^4,4^4$; $\text{emb}=2.6297$) | $r_{\text{sum}}=4$: **$-0.0745$** ($-0.0186/r$), Attn $+0.0926$, **$\text{SR}=57.3$** | $r_{\text{sum}}=8$: $+0.1781$ ($+0.0223/r$), Attn $+0.2843$, $\text{SR}=85.2$ | $r_{\text{sum}}=12$: $+0.2199$ ($+0.0183/r$), Attn $+0.4537$, $\text{SR}=93.0$ | $r_{\text{sum}}=16$: **$+0.2824$** ($+0.0177/r$), Attn $+0.2322$, **$\text{SR}=137.1$** |
| **`Static-16L-Pyr40`** ($2^{14},6,6$; $\text{emb}=2.6344$) | $r_{\text{sum}}=8$: **$-0.0916$** ($-0.0115/r$), Attn $+0.2040$, **$\text{SR}=84.0$** | $r_{\text{sum}}=8$: $+0.1496$ ($+0.0187/r$), Attn $+0.3979$, $\text{SR}=74.5$ | $r_{\text{sum}}=8$: $+0.1519$ ($+0.0190/r$), Attn $+0.3404$, $\text{SR}=88.1$ | $r_{\text{sum}}=16$: **$+0.2799$** ($+0.0175/r$), Attn $+0.2399$, **$\text{SR}=119.6$** |
| **`Static-16L-Unif2d`** ($2^{16}$; $\text{emb}=2.6609$) | $r_{\text{sum}}=8$: **$-0.0637$** ($-0.0080/r$), Attn $+0.1464$, **$\text{SR}=83.4$** | $r_{\text{sum}}=8$: $+0.2147$ ($+0.0268/r$), Attn $+0.3069$, $\text{SR}=75.8$ | $r_{\text{sum}}=8$: $+0.2074$ ($+0.0259/r$), Attn $+0.3830$, $\text{SR}=103.5$ | $r_{\text{sum}}=8$: **$+0.2703$** ($+0.0338/r$), Attn $+0.2329$, **$\text{SR}=109.9$** |
| **`Static-16L-Basal40`** ($6,6,2^{14}$; $\text{emb}=2.6379$) | $r_{\text{sum}}=16$: **$+0.0108$** ($+0.0007/r$), Attn $+0.0390$, **$\text{SR}=106.9$** | $r_{\text{sum}}=8$: $+0.1961$ ($+0.0245/r$), Attn $+0.4696$, $\text{SR}=74.9$ | $r_{\text{sum}}=8$: $+0.2200$ ($+0.0275/r$), Attn $+0.2312$, $\text{SR}=106.9$ | $r_{\text{sum}}=8$: **$+0.2487$** ($+0.0311/r$), Attn $+0.2563$, **$\text{SR}=118.5$** |
| **`Static-8L-Linear32`** ($2,2,3,3,5,5,6,6$; $\text{emb}=2.5362$) | $L_{0..1}$ ($r_{\text{sum}}=4$): $+0.0493$ ($+0.0123/r$), Attn $+0.0224$, $\text{SR}=82.7$ | $L_{2..3}$ ($r_{\text{sum}}=6$): **$+0.1886$** ($+0.0314/r$), Attn $+0.4510$, $\text{SR}=82.6$ | $L_{4..5}$ ($r_{\text{sum}}=10$): $+0.2636$ ($+0.0264/r$), Attn $+0.1310$, $\text{SR}=101.2$ | $L_{6..7}$ ($r_{\text{sum}}=12$): $+0.2510$ ($+0.0209/r$), Attn $+0.2109$, $\text{SR}=129.1$ |
| **`Static-8L-Unif4d`** ($4^8$; $\text{emb}=2.5358$) | $L_{0..1}$ ($r_{\text{sum}}=8$): $+0.0267$ ($+0.0033/r$), Attn $+0.0495$, $\text{SR}=105.1$ | $L_{2..3}$ ($r_{\text{sum}}=8$): **$+0.2443$** ($+0.0305/r$), Attn $+0.3056$, $\text{SR}=108.2$ | $L_{4..5}$ ($r_{\text{sum}}=8$): $+0.2871$ ($+0.0359/r$), Attn $+0.2147$, $\text{SR}=100.5$ | $L_{6..7}$ ($r_{\text{sum}}=8$): $+0.2276$ ($+0.0285/r$), Attn $+0.2111$, $\text{SR}=120.7$ |

**Key Findings on Vertical Capacity Allocation and Exploratory Sub-Layer Profiles**:
1. **Smooth Linear OpenELM Ramp vs. Step-Apical Placement (`I1`–`I1d`, `T1`, `T2`, `U0_s8_linear`)**: Consistent with DeLighT [11] and OpenELM [10], allocating FFN width toward upper layers at 16L substantially outperforms both basal-heavy (`I1`: $-3.799 \pm 0.201\text{ mBPB}$; `I1b`: $-4.328 \pm 0.268\text{ mBPB}$) and uniform-width (`C1`: $-1.127 \pm 0.257\text{ mBPB}$; `I1d`: $-1.656 \pm 0.337\text{ mBPB}$) schedules under fixed total FLOPs. The smooth linear ramp `Static-16L-Linear40` ($r=(1^4,2^4,3^4,4^4)$) and step-apical `Static-16L-Pyr40` ($r=(2^{14},6,6)$) perform comparably (`I1c = -0.528 ± 0.368 mBPB`, $p = 0.246$, $3/4$ seeds negative), while `T1` (`-3.716 ± 0.497 mBPB` on all-global `LLLL`, $p = 0.00494$) and `T2` (`-3.143 ± 0.594 mBPB` at $d=384, H=3$, $p = 0.0132$) demonstrate that the apical-over-basal advantage persists when sliding-window periodicity is removed and at reduced model width $d = 384$. By contrast, at 8 layers (`Static-8L-Linear32` vs. `Static-8L-Unif4d`), linear FFN ramping yields only $-0.400 \pm 0.324\text{ mBPB}$ (`U0_s8_linear`, $p = 0.304$, $3/4$ seeds negative).
2. **Exploratory Sub-Layer Logit-Lens & SVD Stable-Rank Associations (Table 1C)**: As a descriptive diagnostic, Table 1C evaluates each sub-layer's immediate uncalibrated Logit-Lens BPB change (applying final RMSNorm and `lm_head` directly to intermediate residual states) alongside the SVD stable rank ($\|W\|_F^2 / \|W\|_2^2$) of `mlp.c_proj`. In 16-layer models, quartile `Q1` ($L_{0..3}$) MLPs exhibit negative uncalibrated Logit-Lens changes ($\Delta\text{BPB}_{\text{MLP}} = -0.0637$ to $-0.0916\text{ BPB}$ in `Unif2d`, `Linear40`, and `Pyr40`, and $+0.0108\text{ BPB}$ in `Basal40`) and lower `c_proj` stable ranks ($\text{SR} = 57.3\text{--}84.0$ vs. $109.9\text{--}137.1$ in `Q4`), whereas in 8-layer models `Q1` ($L_{0..1}$) and `Q2` ($L_{2..3}$) already show positive uncalibrated MLP reductions ($+0.0267$ and $+0.2443\text{ BPB}$ in `Static-8L-Unif4d`). However, as established in the representation-probing literature [Belrose et al., 2023 [12]], intermediate residual states are not aligned with the final unembedding basis, and early MLPs can worsen immediate uncalibrated readout while writing features used by subsequent layers; similarly, weight stable rank depends on assigned layer width $r_l d$. Accordingly, we treat Table 1C strictly as an **exploratory descriptive association** accompanying the controlled fixed-FLOP benchmark results in Table 1B rather than a causal proof of wasted basal capacity.

---

## 4. Native Unsplit `4x64` Depth Splicing: 8-Seed Shared-Checkpoint vs. Two-Stage Contrasts, Window-Locked Controls, and Analytical Iso-Executed-FLOP Accounting

![Figure 2: Bigram Compositional Frontier and Checkpoint-Forked Splicing Controls](supplement/figure-2.png)

### 4.1 Native Unsplit `4x64` (`B=64`) Arms, Shared-Checkpoint vs. Two-Stage Design, and Iso-Executed-FLOP Control

To evaluate depth growth on the exact native 64-sequence microbatch specification (`B = 64` unsplit, $524{,}288$ tok/step) with explicit analytical nominal and executed FLOP accounting (§2.2), we evaluate **nine `True4x64` arms** (`48` runs in `experiments_p7/runs/` plus `4` runs in `experiments_revival/runs/`, Tables 2A and 2B), expanding our four primary comparison arms (`Static-8L-Unif4d`, `G_stack-Unif4d-Apical`, `G_stack-Unif4d-Medial4`, and `G_stack-PyrTarget-Medial4`) to **8 paired seeds** (`42, 137, 271, 314, 420, 512, 618, 729`) and adding a 4-seed **analytical iso-executed-FLOP static 8L baseline (`Static-8L-Unif4d-m1068-True4x64`, $1{,}068$ steps, $1.3387 \times 10^{17}$ FLOPs)**. We explicitly distinguish three comparison types in Table 2B:

1. **8-Seed True Shared-Checkpoint Fork at Fixed `{3,7,11}` Global Depths (`U6_true_8s`: `Unif4d-Medial4` vs. `Unif4d-Apical`)**: On all 8 seeds, `G_stack-Unif4d-Medial4-True4x64` ($[0..5,2..5,6,7]$, interior splice $5\to 2$) and `G_stack-Unif4d-Apical-True4x64` ($[0..7,4..7]$, apical splice $7\to 4$) fork from **bit-identical saved 8L `Unif4d` Stage-1 checkpoints** at $g = 225$ (`stage1_ckpt_sha256` and `loss_bf16_before` verified identical for every seed in `paper5_tables_verified.txt`, lines 79–87; e.g., at seed 42, `loss_bf16_before = 3.404801`). Both arms hold FFN widths uniform ($4^{12}$), splice count fixed at $1$, window flips fixed at $0$, and **destination global-attention depths strictly fixed at $\{3, 7, 11\}$**, isolating interior ($[2..5]$) versus apical ($[4..7]$) contiguous 4-layer duplication.
2. **8-Seed Two-Stage Growth Package Contrasts (`U7_true_8s`, `U8_true_8s`, `U0_true_8s`, `U0b_true_8s`, `U0d_true_8s`)**: Because `G_stack-PyrTarget-Medial4-True4x64` trains a pyramidal $r = (2,2,4,4,4,4,6,6)$ 8L trunk during Stage 1 ($0..225$; at seed 42, `loss_bf16_before = 3.399315`) before expanding to $r = (2^2, 4^8, 6^2)$, `U7_true_8s` (`PyrTarget-Medial4` vs. `Unif4d-Apical`) compares **two-stage growth packages** (combining pyramidal Stage-1+Stage-2 FFN allocation with interior `Medial4` splicing against uniform apical `G_stack`), and `U8_true_8s` (`PyrTarget-Medial4` vs. `Unif4d-Medial4`) varies FFN allocation across **both stages** under `Medial4` rather than forking from a shared Stage-1 checkpoint.
3. **4-Seed True Shared-Checkpoint 3-Way `PyrTarget` Fork (`U2_true`, `U4_true`, `U5_true`)**: On seeds `42, 137, 271, 314`, `Medial4` ($[0..5,2..5,6,7]$, $1$ splice $5\to 2$, $0$ window flips, `L` at $\{3,7,11\}$), `Medial-Positional` ($[0..3,2..5,4..7]$, $2$ splices $3\to 2, 5\to 4$, $2$ window flips at $d \in \{5,7\}$, `L` at $\{3,7,11\}$), and `Medial-WinLock` ($[0..3,2..5,4..7]$, $2$ splices $3\to 2, 5\to 4$, $0$ window flips, `L` at $\{3,5,11\}$) fork from **bit-identical saved 8L `PyrTarget` Stage-1 checkpoints** at $g = 225$ (`paper5_tables_verified.txt`, lines 73–77).

**Table 2A: Native Unsplit `4x64` (`B=64`, $524{,}288$ tok/step) Arm Summaries (`Mean ± SD` across $n=8$ or $n=4$ seeds, matches `paper5_tables_verified.txt` lines 59–71)**

| `True4x64` Arm (`mb_seq = 64`, `eff_half_mb = false`) | $n$ | Steps ($g$) & Tokens | Analytical Nom / **Exec FLOPs** (`ckpt 2..7`) | **Confirmation `Slice B` BPB** ($\text{Mean} \pm \text{SD}$) | `confirm_bpb` ($\text{Mean} \pm \text{SD}$) | `Slice A` (`sh9A`) BPB ($\text{Mean} \pm \text{SD}$) | `val_bpb` ($\text{Mean} \pm \text{SD}$) | Same-Batch $\Delta\mathcal{L}$ (nats) | Train / Wall (s) & VRAM |
| :--- | :-: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **`Static-8L-Unif4d-True4x64`** ($r=4^8$, 8-seed) | **8** | $953$ ($-$)<br>$499.65\text{M}$ | $1.1945\text{e}17$ /<br>**$1.1945\text{e}17$** ($0\%$) | $0.995310 \pm 0.001394$<br>*(4s: $0.995407$)* | $0.997278 \pm 0.001354$ | $0.995867 \pm 0.001359$ | $0.995216 \pm 0.001393$ | $0.0000$ | $1090.8$ / $1236.7\text{s}$<br>$22{,}805\text{ MB}$ |
| **`Static-8L-Unif4d-m1068-True4x64`** *(Iso-Exec Control)* | **4** | **$1{,}068$** ($-$)<br>$559.94\text{M}$ | $1.3387\text{e}17$ /<br>**$1.3387\text{e}17$** ($0\%$) | **$0.988349 \pm 0.000869$** | **$0.990377 \pm 0.000721$** | **$0.988873 \pm 0.000810$** | **$0.988310 \pm 0.000855$** | $0.0000$ | $1228.7$ / $1374.5\text{s}$<br>$22{,}805\text{ MB}$ |
| **`Static-12L-PyrTarget-True4x64`** ($r=2^2,4^8,6^2$) | 4 | $659$ ($-$)<br>$345.51\text{M}$ | $1.1956\text{e}17$ /<br>**$1.3840\text{e}17$** ($+15.8\%$) | $0.997620 \pm 0.000315$ | $0.999535 \pm 0.000347$ | $0.998088 \pm 0.000344$ | $0.997469 \pm 0.000323$ | $0.0000$ | $1215.6$ / $1402.4\text{s}$<br>$19{,}018\text{ MB}$ |
| **`G_stack-Unif4d-Apical-True4x64`** ($4^8\to 4^{12}$, `[0..7,4..7]`) | **8** | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>**$1.3384\text{e}17$** ($+12.0\%$) | $0.994232 \pm 0.002585$<br>*(4s: $0.994455$)* | $0.996169 \pm 0.002586$ | $0.994714 \pm 0.002579$ | $0.994127 \pm 0.002589$ | $+0.1286$ | $1198.7$ / $1394.7\text{s}$<br>$19{,}018\text{ MB}$ |
| **`G_stack-Unif4d-Medial4-True4x64`** ($4^8\to 4^{12}$, `[0..5,2..5,6,7]`) | **8** | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>**$1.3384\text{e}17$** ($+12.0\%$) | $0.991848 \pm 0.000990$<br>*(4s: $0.992006$)* | $0.993716 \pm 0.001013$ | $0.992349 \pm 0.000987$ | $0.991752 \pm 0.000973$ | **$+0.0688$** | $1198.4$ / $1398.0\text{s}$<br>$22{,}805\text{ MB}$ |
| **`G_stack-PyrTarget-Medial-True4x64`** *(2-spl, 2-flip, `{3,7,11}`)* | 4 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>**$1.3384\text{e}17$** ($+12.0\%$) | $0.993638 \pm 0.001252$ | $0.995457 \pm 0.001295$ | $0.994078 \pm 0.001238$ | $0.993507 \pm 0.001222$ | $+0.0950$ | $1190.9$ / $1388.8\text{s}$<br>$19{,}018\text{ MB}$ |
| **`G_stack-PyrTarget-Medial-WinLock-True4x64`** *(2-spl, 0-flip, `{3,5,11}`)* | 4 | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>**$1.3384\text{e}17$** ($+12.0\%$) | $0.993581 \pm 0.001284$ | $0.995404 \pm 0.001281$ | $0.994008 \pm 0.001284$ | $0.993437 \pm 0.001260$ | $+0.0970$ | $1191.3$ / $1389.3\text{s}$<br>$19{,}018\text{ MB}$ |
| **`G_stack-PyrTarget-Medial4-True4x64`** *(1-spl, 0-flip, `{3,7,11}`)* | **8** | $728$ ($225$)<br>$381.68\text{M}$ | $1.1946\text{e}17$ /<br>**$1.3384\text{e}17$** ($+12.0\%$) | **$0.990915 \pm 0.001245$**<br>*(4s: $0.991206$)* | **$0.992773 \pm 0.001221$** | **$0.991422 \pm 0.001197$** | **$0.990834 \pm 0.001215$** | **$+0.0834$** | $1192.9$ / $1386.8\text{s}$<br>$22{,}805\text{ MB}$ |

**Table 2B: Paired 8-Seed (`_8s`, with 6-Test Holm-Bonferroni `p_Holm6`) and 4-Seed Contrasts Under Native `True4x64` ($\text{Mean }\Delta \pm \text{SE}$, matches `paper5_tables_verified.txt` lines 96–144)**

| Contrast ID & Comparison | $n$ | Design Type & Resource Control | **`Slice B` $\Delta\text{BPB}$ ($\text{mBPB} \pm \text{SE}$)** | `Slice B` 95% $t$-CI, $t$-stat, $p_{\text{raw}}$ (& **$p_{\text{Holm6}}$**) | `confirm` / `Slice A` / `val` $\Delta$ ($\text{mBPB}$) | Per-Seed `Slice B` $\Delta$ ($\text{mBPB}$) & Sign Count |
| :--- | :-: | :--- | :---: | :---: | :---: | :--- |
| **`U0_true_8s`**: `PyrTarget-Medial4` vs. `Static-8L-Unif4d` ($m=953$) | **8** | Two-Stage Package vs. Static 8L; Iso-Nom FLOPs ($+12.04\%$ exec; 4s: $-4.202 \pm 0.310$) | **$-4.395 \pm 0.368$** | $[-5.266, -3.524]$, $t=-11.93$, $p=0.000007$ (**$p_{\text{Holm6}}=0.000040$**) | $-4.505$ / $-4.445$ / $-4.382$ | $8/8$ negative: `[-4.930, -3.435, -4.360, -4.082, -2.739, -4.216, -5.944, -5.453]` |
| **`U0b_true_8s`**: `Unif4d-Medial4` vs. `Static-8L-Unif4d` ($m=953$) | **8** | Two-Stage Growth vs. Static 8L; Iso-Nom FLOPs ($+12.04\%$ exec; 4s: $-3.402 \pm 0.353$) | **$-3.461 \pm 0.328$** | $[-4.237, -2.686]$, $t=-10.55$, $p=0.000015$ (**$p_{\text{Holm6}}=0.000075$**) | $-3.562$ / $-3.518$ / $-3.464$ | $8/8$ negative: `[-3.326, -3.763, -2.447, -4.070, -3.216, -1.942, -4.200, -4.727]` |
| **`U0d_true_8s`**: `Unif4d-Apical` vs. `Static-8L-Unif4d` ($m=953$) | **8** | Two-Stage Apical $G_{\text{stack}}$ vs. Static 8L; Iso-Nom FLOPs ($+12.04\%$ exec) | $-1.077 \pm 0.528$ | $[-2.327, +0.172]$, $t=-2.04$, $p=0.0808$ ($p_{\text{Holm6}}=0.0923$) | $-1.109$ / $-1.153$ / $-1.088$ | $6/8$ negative: `[-1.200, -1.987, -2.218, +1.595, -1.178, -2.236, -2.231, +0.837]` |
| **`U6_true_8s`**: `Unif4d-Medial4` vs. `Unif4d-Apical` | **8** | **Bit-Identical Shared S1 Fork**, Exact Iso-Exec FLOPs, **Fixed `{3,7,11}`** (4s: $-2.449 \pm 1.148$) | **$-2.384 \pm 0.773$** | $[-4.211, -0.557]$, $t=-3.09$, **$p=0.0177$** ($p_{\text{Holm6}}=0.0530$) | $-2.453$ / $-2.365$ / $-2.376$ | $7/8$ negative: `[-2.126, -1.777, -0.229, -5.665, -2.038, +0.294, -1.969, -5.564]` |
| **`U7_true_8s`**: `PyrTarget-Medial4` vs. `Unif4d-Apical` | **8** | **Two-Stage Package** (`PyrTarget` S1+S2 + `Medial4` vs. `Unif4d` Apical $G_{\text{stack}}$; Exact Iso-Exec; 4s: $-3.249$) | **$-3.317 \pm 0.661$** | $[-4.881, -1.754]$, $t=-5.02$, $p=0.00153$ (**$p_{\text{Holm6}}=0.00614$**) | $-3.396$ / $-3.293$ / $-3.294$ | $8/8$ negative: `[-3.730, -1.449, -2.142, -5.677, -1.561, -1.979, -3.713, -6.289]` |
| **`U8_true_8s`**: `PyrTarget-Medial4` vs. `Unif4d-Medial4` | **8** | **Two-Stage FFN Allocation** ($r=(2^2,4^4,6^2)\to(2^2,4^8,6^2)$ vs. $4^8\to 4^{12}$ in `Medial4`; Exact Iso-Exec) | **$-0.933 \pm 0.386$** | $[-1.845, -0.021]$, $t=-2.42$, **$p=0.0462$** ($p_{\text{Holm6}}=0.0923$) | $-0.943$ / $-0.927$ / $-0.918$ | $6/8$ negative: `[-1.603, +0.328, -1.913, -0.012, +0.477, -2.273, -1.744, -0.725]` |
| **`U0_exec_true`**: `PyrTarget-Medial4` ($m=728$, ckpt) vs. `Static-8L-m1068` | 4 | **Analytical Iso-Executed FLOPs** ($1.3384\text{e}17$ vs. $1.3387\text{e}17$; $-178.3\text{M}$ tok) | **$+2.857 \pm 0.334$** | $[+1.793, +3.920]$, $t=+8.55$, **$p=0.00336$** | $+2.621$ / $+2.793$ / $+2.782$ | $0/4$ neg: `[+2.015, +3.614, +2.733, +3.065]`; **12L leads `bi_l` by $-16.073\text{ mBPB}$** |
| **`U0_s8_extra`**: `Static-8L-m1068` vs. `Static-8L-m953` | 4 | $+115$ Uncheckpointed 8L Steps ($+12.07\%$ FLOPs & tokens) | **$-7.058 \pm 0.043$** | $[-7.194, -6.922]$, $t=-165.13$, **$p<0.000001$** | $-6.998$ / $-7.114$ / $-7.023$ | $4/4$ negative: `[-6.945, -7.049, -7.092, -7.147]` |
| **`U0c_true` / `U1_true`**: `Static-12L` vs. `Static-8L-m953` / `Medial4` vs. `Static-12L` | 4 | Static 12L Inversion (`U0c`) / $-3.29\%$ Fewer Exec FLOPs (`U1`) | **$+2.213 \pm 0.319$** /<br>**$-6.414 \pm 0.568$** | $p=0.00614$ /<br>**$p=0.00149$** | $+2.161$ / $-6.537$ (`confirm`) | `U1_true` ($4/4$): `[-7.967, -5.568, -6.559, -5.563]` |
| **`U2_true` / `U3_true`**: `Medial4` vs. `Medial-Pos` / `Medial-Pos` vs. `Static-12L` | 4 | **Bit-Identical S1 Fork** (`U2`) / $-3.29\%$ Exec FLOPs (`U3`) | **$-2.432 \pm 0.240$** /<br>**$-3.982 \pm 0.504$** | $p=0.00204$ /<br>**$p=0.00422$** | $-2.459$ / $-4.078$ (`confirm`) | `U2_true` ($4/4$): `[-2.572, -2.557, -2.856, -1.743]` |
| **`U4_true` / `U5_true`**: `Medial4` vs. `Medial-WinLock` / `WinLock` vs. `Medial-Pos` | 4 | **Bit-Identical S1 Fork** (`0` flips `{3,7,11}` vs. `{3,5,11}` / `0` vs. `2` flips) | **$-2.375 \pm 0.182$** /<br>**$-0.057 \pm 0.081$** | **$p=0.000976$** /<br>$p=0.531$ | $-2.407$ / $-0.053$ (`confirm`) | `U4_true`: `[-2.526, -2.575, -2.570, -1.829]`<br>`U5_true`: `[-0.046, +0.018, -0.286, +0.086]` |
| **`M1` / `M2` / `M3`**: `True4x64` vs. `Split8x32` (`0%` recomp) | 4 | $B=64$ (ckpt `2..7`) vs. $B=32$ (no ckpt) at $524\text{k}$ tok/step | `M1`: $-0.019 \pm 0.121$<br>`M2`: $+0.142 \pm 0.290$<br>`M3`: $+0.059 \pm 0.128$ | $p = 0.887$ / $0.658$ / $0.675$ | `Slice A`: $-0.010$ / $+0.125$ / $+0.027$ | Uncheckpointed `Split8x32` `Slice B`: `0.991225` (`Medial4`), `0.993496` (`Medial`), `0.997561` (`Static-12L`) |
| **`B1_true` / `B2_true`**: Step-Budget (`True4x64` $524\text{k}$ vs. `262k`) | 4 | $728$ vs. $1{,}456$ steps (`B1`); $659$ vs. $1{,}318$ steps (`B2`) | — | $p = 0.000213$ / $0.000107$ | `Slice A`: **$+12.407 \pm 0.570$** / **$+14.172 \pm 0.518$** | Halving optimizer updates adds $+12.4$ to $+14.2\text{ mBPB}$ |

### 4.2 Conclusions and Statistical Scope of the 8-Seed and Iso-Executed-FLOP `True4x64` Experiments

1. **Separating the 8-Seed Shared-Checkpoint Fork (`U6_true_8s`) from the Two-Stage Package Contrasts (`U7_true_8s`, `U8_true_8s`)**:
   - On the **bit-identical `Unif4d` Stage-1 checkpoint fork** (`U6_true_8s`), holding FFN ratios ($4^{12}$), splice count ($1$), window flips ($0$), and destination global-attention depths ($\{3,7,11\}$) strictly fixed, duplicating the interior `SSSL` period $[2..5]$ (`Unif4d-Medial4`, splice $5\to 2$) rather than the unembedding-adjacent apical period $[4..7]$ (`Unif4d-Apical`, splice $7\to 4$) halves the same-batch transition shock ($+0.0688$ vs. $+0.1286\text{ nats}$), reduces cross-seed `Slice B` endpoint SD from $2.59\text{ mBPB}$ to $0.99\text{ mBPB}$, and improves `Slice B` BPB by **$-2.384 \pm 0.773\text{ mBPB}$** ($p_{\text{raw}} = 0.0177, p_{\text{Holm6}} = 0.0530$, $7/8$ seeds negative, with seed $512$ at $+0.294\text{ mBPB}$).
   - Comparing the **two-stage interior-pyramidal package** (`G_stack-PyrTarget-Medial4`) against contiguous apical `G_stack` (`G_stack-Unif4d-Apical`) yields **$-3.317 \pm 0.661\text{ mBPB}$** across all 8 seeds ($p_{\text{raw}} = 0.00153, \mathbf{p_{\text{Holm6}} = 0.00614}$, $8/8$ seeds negative; `U7_true_8s`), while varying FFN allocation across both stages within `Medial4` (`PyrTarget-Medial4` vs. `Unif4d-Medial4`) yields **$-0.933 \pm 0.386\text{ mBPB}$** ($p_{\text{raw}} = 0.0462, p_{\text{Holm6}} = 0.0923$, $6/8$ seeds negative; `U8_true_8s`). Because `U6_true_8s` ($p_{\text{Holm6}} = 0.0530$) and `U8_true_8s` ($p_{\text{Holm6}} = 0.0923$) sit marginally above the 6-test Holm threshold $\alpha = 0.05$, we state explicitly that `U7_true_8s`, `U0_true_8s` ($p_{\text{Holm6}} = 0.000040$), and `U0b_true_8s` ($p_{\text{Holm6}} = 0.000075$) are family-wise significant across the 6-test 8-seed suite, whereas `U6_true_8s` and `U8_true_8s` provide unadjusted supporting evidence for the pure topology fork and two-stage FFN allocation components, respectively.
2. **Nominal vs. Analytical Executed FLOP Trade-Off Relative to Static 8L (`U0_true_8s`, `U0_exec_true`, `U0_s8_extra`, and `M1`)**: Relative to 953-step nominal-FLOP-matched `Static-8L-Unif4d-True4x64`, 12L `G_stack-PyrTarget-Medial4` wins by **$-4.395 \pm 0.368\text{ mBPB}$** across 8 seeds ($p_{\text{Holm6}} = 0.000040$; `U0_true_8s`). However, because 6-block activation checkpointing (`2 <= l < 8`) adds $+12.04\%$ analytical executed FLOPs during the 503 Stage-2 steps, granting uncheckpointed `Static-8L-Unif4d-True4x64` those same analytical executed FLOPs (`1,068` steps, $559.94\text{M}$ tokens, $1.3387 \times 10^{17}$ FLOPs) improves static 8L by **$-7.058 \pm 0.043\text{ mBPB}$** (`U0_s8_extra`), leaving `Static-8L-m1068` ahead of checkpointed 728-step `PyrTarget-Medial4` overall by **$+2.857 \pm 0.334\text{ mBPB}$** on `Slice B` (`U0_exec_true`, $p = 0.00336$; novel bigrams $+3.585\text{ mBPB}$, short-repeat bigrams $+0.160\text{ mBPB}$), while 12L `PyrTarget-Medial4` still beats `Static-8L-m1068` by **$-16.073\text{ mBPB}$** on long-distance bigrams (`bi_l`) and by **$-0.738\text{ mBPB}$** on long-repeat unigrams (`uni_l`). Consequently, under native `B=64` when 6-block checkpointing is used at 12L, the $+12.04\%$ recomputation tax exceeds the aggregate BPB gain over an uncheckpointed 8L model allowed to train on $+60.3\text{M}$ extra tokens; avoiding recomputation (as in uncheckpointed `Split8x32`, where `M1 = -0.019 ± 0.121 mBPB` and executed FLOPs equal $1.1946 \times 10^{17}$) preserves the $-4.182\text{ mBPB}$ overall lead over static 8L at exact analytical iso-executed FLOPs.
3. **Narrowed Interpretation of `Medial-WinLock` (`U4_true` and `U5_true`)**: On the 4-seed `PyrTarget` checkpoint fork, `U5_true` ($-0.057 \pm 0.081\text{ mBPB}$, $p = 0.531$) shows that locking parent window types on the two-splice `Medial` map produces little net benefit when destination global-layer depths simultaneously shift from $\{3,7,11\}$ to $\{3,5,11\}$, while `U4_true` ($-2.375 \pm 0.182\text{ mBPB}$, $p = 0.000976$) shows that contiguous period-preserving `Medial4` (`{3,7,11}`, interface $5\to 2$) substantially outperforms the parent-window-locked two-splice layout (`{3,5,11}`, interfaces $3\to 2, 5\to 4$).

### 4.3 Summary of the `262k`-Tok/Step Multi-Seed Campaign (`C1`–`C8`, `S3`–`S10`) and Learning Curves

![Figure 3: Training Loss Trajectories, Same-Batch Mitosis Shock, and Post-Expansion Recovery](supplement/figure-3.png)

At $262{,}144$ tok/step (`4x32` split-microbatch, $0\%$ recomputation, $1.1945 \times 10^{17}$ analytical nominal and executed FLOPs; full 21-arm table in Appendix A, Table A1), all 8 primary contrasts survive Holm-Bonferroni correction ($m = 8$, Table 3):
- **Consistent `Medial4` vs. `Medial` Advantage Across Four Model Families**: Contiguous `Medial4` beats two-splice `Medial` in uniform copy (`C2`: $-1.556 \pm 0.337\text{ mBPB}$, $p_{\text{Holm}} = 0.0382$), FFN $2^8\to 4^{12}$ widening (`S3`: $-1.760 \pm 0.294\text{ mBPB}$, $p = 0.00935$), seed-pyramidal $2^6 4^2 \to 4^{12}$ (`S4`: $-1.641 \pm 0.113\text{ mBPB}$, $p = 0.000715$), and `True4x64` pyramidal copy (`U2_true`: $-2.432 \pm 0.240\text{ mBPB}$, $p = 0.00204$)—winning on **$16/16$ paired seeds** ($20/20$ including `Medial-WinLock`).
- **12L and 16L Pyramidal Growth Gains (`C3`–`C8`, `S8`)**: `G_stack-PyrTarget-Medial4` (`sh9A` $= 0.979259 \pm 0.000614\text{ BPB}$) beats `Static-12L-PyrTarget` by **$-4.658 \pm 0.636\text{ mBPB}$** (`C3`, $p_{\text{Holm}} = 0.0210$), `Static-12L-Unif4d` by **$-6.230 \pm 0.751\text{ mBPB}$** (`C4`, $p_{\text{Holm}} = 0.0183$), and `Static-8L-Unif4d` by **$-7.342 \pm 0.452\text{ mBPB}$** (`C5`, $p_{\text{Holm}} = 0.00405$), matching coupled 2D width+depth growth `PyrTarget-KeepAp-Medial4` (`0.979257 ± 0.000576`, `S8`: $-0.002 \pm 0.123\text{ mBPB}$) without requiring FFN width splitting. At 16L, 3-stage `Pyr16-3Stage-Copy-Medial4` (`0.980932 ± 0.000512`) beats `Static-16L-Pyr40` by **$-7.194 \pm 0.476\text{ mBPB}$** (`C7`, $p_{\text{Holm}} = 0.00440$) and `Static-16L-Unif2d` by **$-8.321 \pm 0.612\text{ mBPB}$** (`C8`, $p_{\text{Holm}} = 0.00518$). As Figure 3b illustrates, on the bit-identical `True4x64` forks, `Medial4` and `Unif4d-Medial4` absorb their smaller transition shocks rapidly and maintain a clean trajectory lead over `Medial-Positional`, `Medial-WinLock`, and `Unif4d-Apical` through warmdown.

**Table 3: Conditional 4-Seed Paired Contrasts at $262{,}144$ Tok/Step (`C1`–`C8` with Holm-Bonferroni $m=8$, plus Key Secondary Contrasts; $\text{Mean }\Delta \pm \text{SE}$)**

| Contrast ID & Comparison (`0%` recomp, $1.1945\text{e}17$ exec FLOPs) | `sh9A` $\Delta\text{BPB}$ ($\text{mBPB} \pm \text{SE}$) | Paired 95% $t$-CI ($\text{mBPB}$) | `val` / `hld` $\Delta$ ($\text{mBPB}$) | $t$-stat ($df=3$) | $p_{\text{raw}}$ | **$p_{\text{Holm}}$ ($m=8$)** | Per-Seed `sh9A` $\Delta$ ($\text{mBPB}$) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :--- |
| **`C1`**: `Static-16L-Pyr40` vs. `Static-16L-Unif2d` | **$-1.127 \pm 0.257$** | $[-1.946, -0.308]$ | $-1.082$ / $-1.135$ | $-4.38$ | $0.02203$ | **$0.03821$** | `[-0.460, -1.029, -1.663, -1.357]` |
| **`C2`**: `G_stack-Unif4d-Medial4` vs. `G_stack-Unif4d-Medial` | **$-1.556 \pm 0.337$** | $[-2.628, -0.484]$ | $-1.566$ / $-1.539$ | $-4.62$ | $0.01911$ | **$0.03821$** | `[-2.514, -0.964, -1.260, -1.485]` |
| **`C3`**: `G_stack-PyrTarget-Medial4` vs. `Static-12L-PyrTarget` | **$-4.658 \pm 0.636$** | $[-6.680, -2.635]$ | $-4.636$ / $-4.725$ | $-7.33$ | $0.00525$ | **$0.02100$** | `[-6.412, -3.451, -4.099, -4.668]` |
| **`C4`**: `G_stack-PyrTarget-Medial4` vs. `Static-12L-Unif4d` | **$-6.230 \pm 0.751$** | $[-8.620, -3.840]$ | $-6.203$ / $-6.278$ | $-8.30$ | $0.00367$ | **$0.01835$** | `[-7.582, -6.730, -6.523, -4.084]` |
| **`C5`**: `G_stack-PyrTarget-Medial4` vs. `Static-8L-Unif4d` | **$-7.342 \pm 0.452$** | $[-8.779, -5.904]$ | $-7.278$ / $-7.431$ | $-16.26$ | $0.00051$ | **$0.00405$** | `[-8.446, -7.181, -6.255, -7.484]` |
| **`C6`**: `PyrTarget-KeepAp-Medial4` vs. `Static-12L-PyrTarget` | **$-4.660 \pm 0.645$** | $[-6.711, -2.608]$ | $-4.646$ / $-4.747$ | $-7.23$ | $0.00546$ | **$0.02100$** | `[-6.279, -3.176, -4.317, -4.866]` |
| **`C7`**: `Pyr16-3Stage-Copy-Medial4` vs. `Static-16L-Pyr40` | **$-7.194 \pm 0.476$** | $[-8.709, -5.680]$ | $-7.164$ / $-7.281$ | $-15.12$ | $0.00063$ | **$0.00440$** | `[-7.454, -5.922, -8.209, -7.192]` |
| **`C8`**: `Pyr16-3Stage-Copy-Medial4` vs. `Static-16L-Unif2d` | **$-8.321 \pm 0.612$** | $[-10.271, -6.372]$ | $-8.246$ / $-8.416$ | $-13.59$ | $0.00086$ | **$0.00518$** | `[-7.914, -6.950, -9.872, -8.549]` |
| **`S3` / `S4`**: `Medial4` vs. `Medial` in `M2D-Unif4d` / `Pyr-Apical2` | **$-1.760 \pm 0.294$** /<br>**$-1.641 \pm 0.113$** | $[-2.697, -0.824]$ /<br>$[-2.001, -1.280]$ | $-1.717$ / $-1.603$ (`val`) | $-5.98$ /<br>$-14.47$ | $0.00935$ /<br>$0.00072$ | — | `[-1.635,-2.331,-0.991,-2.085]` /<br>`[-1.624,-1.903,-1.352,-1.684]` |
| **`S5` / `S7`**: `Pyr-Apical2-Medial4` vs. `Apical` / `PyrTarget` vs. `Apical2` | $-1.251 \pm 0.469$ /<br>**$-1.134 \pm 0.206$** | $[-2.743, +0.242]$ /<br>$[-1.789, -0.479]$ | $-1.162$ / $-1.183$ (`val`) | $-2.67$ /<br>$-5.51$ | $0.07588$ /<br>$0.01178$ | — | `[-0.147,-2.104,-0.801,-1.951]` /<br>`[-1.503,-1.321,-1.157,-0.554]` |
| **`S8` / `S9`**: `KeepAp` vs. `G_stack-PyrTarget` / `16L 3-Stg` vs. `2-Stg` | $-0.002 \pm 0.123$ /<br>$-0.556 \pm 0.253$ | $[-0.392, +0.388]$ /<br>$[-1.361, +0.249]$ | $-0.010$ / $-0.581$ (`val`) | $-0.02$ /<br>$-2.20$ | $0.98799$ /<br>$0.11534$ | — | `[+0.133,+0.275,-0.219,-0.198]` /<br>`[-1.065,-0.374,-0.853,+0.067]` |

---

## 5. Bounded Diagnostic and Matched-Configuration 4-Seed Continuation Study of Per-Block Sequence Coarsening

![Figure 4: Per-Block Coarsening Diagnostics: Parity Asymmetry, Filterbank Scalars, and Matched 4-Seed KeepFilter Continuations](supplement/figure-4.png)

### 5.1 Per-Block Coarse Residual Wrapper and Transition Operators (`train_multigrid.py`)

Rather than maintaining a persistent shortened hidden stack across layers as in Hourglass [7] or Funnel-Transformer [9], `experiments_p6/train_multigrid.py` (`Block.forward`, lines 369–435) keeps the main residual stream $x \in \mathbb{R}^{B\times T\times d}$ at full resolution $T = 2{,}048$ and wraps selected interior blocks $l$ at spatial stride $s_l = 2$ ($T_c = 1{,}024$) with learnable scalars $(\gamma_{R,l}, \beta_{0,l}, \beta_{1,l})$:
1. **Normalized Prefix-Causal Restriction ($T \to T/2$)**: With phase $\phi_l \in \{0, 1\}$ (`Phase0`: $\phi_l = 0$; `Haar`: $\phi_l = l \bmod 2$), block $l$ forms $z_{\text{in}, m} = (x_{2m+\phi_l} + \gamma_{R,l} x_{2m+\phi_l-1}) / \sqrt{1 + \gamma_{R,l}^2}$ ($m \in \{0,\dots,T/2-1\}$, zero-padded at $t=-1$), evaluates coarse attention (window $w_l // 2$, decimated RoPE/VE) and MLP to obtain $\Delta z = A_l(z_{\text{in}}) + M_l(z_{\text{in}} + A_l(z_{\text{in}}))$, and prolongates $\Delta z$ via causal ZOH ($\Delta x_{2m+\phi_l} = \beta_{0,l}\Delta z_m$, $\Delta x_{2m+\phi_l+1} = \beta_{1,l}\Delta z_m$) to return $x + \Delta x$.
2. **Four Matched-Configuration Coarse-to-Fine ($s_l: 2 \to 1$) Transition Rules at Step $g = 190$**: When transitioning coarse blocks $[6..9]$ to fine stride $s_l = 1$ at step $g = 190$, we compare four transition operators that separate the 2-tap causal input filter $z_{\text{in},t} = (x_t + \gamma_{R,l} x_{t-1})/\sqrt{1+\gamma_{R,l}^2}$ from the mean residual gain $\bar{\beta}_l = \frac{1}{2}(\beta_{0,l}+\beta_{1,l})$:
   - **`Unfolded4`** (naive $s_l: 2 \to 1$): drops both the input filter and $\bar{\beta}_l$, computing $x + A_l(x) + M_l(x + A_l(x))$.
   - **`ScaleFold4`** (`--c2f-scale-fold`): drops the input filter and folds $\bar{\beta}_l$ into `attn.c_proj` and `mlp.c_proj`, computing $x + \bar{\beta}_l A_l(x) + \bar{\beta}_l M_l(x + \bar{\beta}_l A_l(x))$ (which scales the internal MLP residual input $x + \bar{\beta}_l A_l(x)$ before pre-MLP RMSNorm).
   - **`OuterGainNoFilter4`** (`--c2f-outer-gain-nofilter`): drops **only** the 2-tap input filter while keeping the **exact** outer block gain $x + \bar{\beta}_l [A_l(x) + M_l(x + A_l(x))]$ (`train_multigrid.py`, lines 409–416).
   - **`KeepFilter4`** (`--c2f-keep-filter`): retains **both** the 2-tap causal input filter $z_{\text{in}}$ and the exact outer block gain $x + \bar{\beta}_l [A_l(z_{\text{in}}) + M_l(z_{\text{in}} + A_l(z_{\text{in}}))]$.

### 5.2 Matched-Configuration 4-Seed C2F Contrasts (`C2F_1`–`C2F_5`) and Bounded Negative V-Cycle Finding

All four 4-seed (`42, 137, 271, 314`) continuation arms are trained as **matched-configuration two-stage runs** with the same Stage-1 coarse configuration on $[6..9]$ ($r = (2^2, 4^8, 6^2)$, $g = 190$ coarse steps at $3.1169 \times 10^8\text{ FLOPs/tok}$ followed by $176$ fine steps at $3.4603 \times 10^8\text{ FLOPs/tok}$, totaling $366$ steps, $m = 183$, $95.94\text{M}$ tokens, and an exact `cumulative_flops` total of **$3.0145 \times 10^{16}\text{ FLOPs}$** across all 16 retained C2F runs—strictly iso-compute across `C2F_1`–`C2F_5`, and $\approx 5.0\%$ above the $2.8713 \times 10^{16}\text{ FLOPs}$ of the `p5_s240` dense screening runs; Table 4) rather than loaded from a saved Stage-1 checkpoint file. Because `torch.compile` and CUDA reduction non-determinism introduce minor Stage-1 trajectory drift across independent two-stage runs prior to $g = 190$ (at seed 42, `loss_before` is $3.441200$ for `KeepFilter4`, $3.445084$ for `OuterGainNoFilter4`, $3.450536$ for `ScaleFold4`, and $3.447616$ for `Unfolded4`, with block-6 mean gains $\bar{\beta}_6 = 0.35097, 0.34420, 0.33418, 0.34324$; across all 4 seeds, mean `loss_before` is $3.4494 \pm 0.0066$ vs. $3.4499 \pm 0.0058$), we report both the **4-seed mean and per-seed range of `delta_loss`** (`+0.0956 ± 0.0156 nats`, per-seed `[+0.1186, +0.0901, +0.0829, +0.0909]` across seeds `42, 137, 271, 314`, range `+0.0829` to `+0.1186 nats` for `KeepFilter4`, vs. `+2.6814 ± 0.1566 nats`, `[+2.7882, +2.4674, +2.8154, +2.6546]` for `Unfolded4`) and the **single-seed (`seed = 42`) counterfactual same-checkpoint transition diagnostics** (`delta_loss_keep_filter = +0.0853`, `+0.0915`, `+0.0937 nats` vs. `delta_loss_unfolded = +2.6820` to `+2.7897 nats` on the identical Stage-1 states).

**Table 4: Matched-Configuration 4-Seed Coarse-to-Fine Continuations on `[6..9]` at $g=190$ ($366$ steps, $3.0145 \times 10^{16}\text{ FLOPs}$, matches `paper5_tables_verified.txt` lines 165–174) & Full-Budget V-Cycle Diagnostics**

| Arm ($\text{Mean} \pm \text{SD}$) or Contrast ($\text{Mean }\Delta \pm \text{SE}$) | $n$ | Coarse Blocks & Steps ($g$) | Same-Batch Shock $\Delta\mathcal{L}$ (nats) | **`sh9A` BPB** (or $\Delta\text{mBPB}$) | `val_bpb` / `holdout_bpb` (or $\Delta\text{mBPB}$) | Paired 95% $t$-CI, $t$-stat, $p_{\text{raw}}$, & Exact Seeds (`paper5_tables_verified.txt`) |
| :--- | :-: | :---: | :---: | :---: | :---: | :--- |
| **`C2F-MG12-KeepFilter4-g190`** (`--c2f-keep-filter`) | 4 | $[6..9]$, $366$ ($190$) | **$+0.0956 \pm 0.0156$** | **$1.107784 \pm 0.001045$** | $1.107012 \pm 0.001007$ /<br>$1.106648 \pm 0.000987$ | Retains 2-tap filter $z_{\text{in}}$ & outer gain $\bar{\beta}_l \Delta z$; $\Delta\mathcal{L}$: `[+0.1186, +0.0901, +0.0829, +0.0909]` |
| **`C2F-MG12-OuterGainNoFilter4-g190`** (`--c2f-outer-gain-nofilter`) | 4 | $[6..9]$, $366$ ($190$) | $+2.1616 \pm 0.1404$ | $1.124970 \pm 0.002874$ | $1.124208 \pm 0.002884$ /<br>$1.124021 \pm 0.002895$ | Exact outer gain $\bar{\beta}_l \Delta z$; drops only 2-tap filter; $\Delta\mathcal{L}$: `[+2.2569, +1.9767, +2.2836, +2.1292]` |
| **`C2F-MG12-ScaleFold4-g190`** (`--c2f-scale-fold`) | 4 | $[6..9]$, $366$ ($190$) | $+2.2675 \pm 0.1995$ | $1.125570 \pm 0.002375$ | $1.124747 \pm 0.002394$ /<br>$1.124557 \pm 0.002422$ | Folds $\bar{\beta}_l$ into `c_proj`; drops 2-tap filter; $\Delta\mathcal{L}$: `[+2.3965, +2.0829, +2.4779, +2.1127]` |
| **`C2F-MG12-Unfolded4-g190`** (naive $s_l: 2\to 1$) | 4 | $[6..9]$, $366$ ($190$) | $+2.6814 \pm 0.1566$ | $1.126972 \pm 0.002568$ | $1.126206 \pm 0.002592$ /<br>$1.125989 \pm 0.002624$ | Drops both 2-tap filter & $\bar{\beta}_l$ gain; $\Delta\mathcal{L}$: `[+2.7882, +2.4674, +2.8154, +2.6546]` |
| **`C2F_1`: `KeepFilter4` vs. `Unfolded4`** (full package) | 4 | Matched S1 ($[6..9]$) | **$-2.5858\text{ nats}$** | **$-19.188 \pm 1.061$** | $-19.194$ / $-19.342$ | CI $[-22.564, -15.812]$, $t=-18.09$, **$p=0.000369$** ($4/4$)<br>`[-19.891, -18.781, -21.570, -16.510]` |
| **`C2F_2_pure_filter`: `KeepFilter4` vs. `OuterGainNoFilter4`** | 4 | **Filter Effect Conditional on Outer Gain** | **$-2.0660\text{ nats}$** | **$-17.187 \pm 1.370$** | $-17.197$ / $-17.373$ | CI $[-21.547, -12.826]$, $t=-12.54$, **$p=0.00109$** ($4/4$)<br>`[-19.789, -15.821, -19.130, -14.006]` |
| **`C2F_2_scalefold`: `KeepFilter4` vs. `ScaleFold4`** | 4 | Filter + Outer vs. Folded Gain | **$-2.1719\text{ nats}$** | **$-17.786 \pm 1.070$** | $-17.735$ / $-17.909$ | CI $[-21.190, -14.383]$, $t=-16.63$, **$p=0.000473$** ($4/4$)<br>`[-19.792, -16.890, -19.268, -15.196]` |
| **`C2F_3_outer_gain`: `OuterGainNoFilter4` vs. `Unfolded4`** | 4 | Outer Gain $\bar{\beta}_l$ Without Filter | $-0.5198\text{ nats}$ | $-2.002 \pm 0.644$ | $-1.997$ / $-1.968$ | CI $[-4.050, +0.047]$, $t=-3.11$, $p=0.0529$ ($4/4$)<br>`[-0.102, -2.960, -2.440, -2.504]` |
| **`C2F_4_gain_form`: `OuterGainNoFilter4` vs. `ScaleFold4`** | 4 | Outer Gain vs. `c_proj` Folding | $-0.1059\text{ nats}$ | $-0.600 \pm 0.308$ | $-0.538$ / $-0.536$ | CI $[-1.580, +0.381]$, $t=-1.95$, $p=0.147$ ($4/4$)<br>`[-0.003, -1.069, -0.137, -1.190]` |
| **`C2F_5_scalefold_vs_unfolded`: `ScaleFold4` vs. `Unfolded4`** | 4 | Folded Gain Without Filter | $-0.4139\text{ nats}$ | $-1.402 \pm 0.479$ | $-1.459$ / $-1.432$ | CI $[-2.926, +0.123]$, $t=-2.93$, $p=0.0612$ ($4/4$)<br>`[-0.100, -1.891, -2.302, -1.314]` |
| **`MG-V12-Pyr-Phase0` / `MG-V16-Pyr-Phase0`** (Full Budget) | 1 | $[6..9]$ ($1600$ st) /<br>$[6..13]$ ($1626$ st) | $0.0000$ | $0.990401$ /<br>$0.994328$ | $0.989659$ / $0.993613$ (`val`) | Odd$-$Even parity gap: $+2.054$ / $+4.431\text{ mBPB}$; trails full-res growth by $>10\text{ mBPB}$ |

**Diagnostic Takeaways**: (1) Conditional on retaining the outer block gain $\bar{\beta}_l$, `C2F_2_pure_filter` (`KeepFilter4` vs. `OuterGainNoFilter4`: **$-17.187 \pm 1.370\text{ mBPB}$**, $p = 0.00109$, $4/4$ seeds negative; same-batch shock $+0.0956$ vs. $+2.1616\text{ nats}$) isolates the 2-tap causal input filter and accounts for a conditional mean-contrast ratio of **$89.6\% \pm 3.6\%$** (`C2F_2_pure_filter / C2F_1`, jackknife SE; per-seed ratios $99.5\%, 84.2\%, 88.7\%, 84.8\%$), whereas outer vs. `c_proj`-folded residual scaling without the filter differs by only $-0.600 \pm 0.308\text{ mBPB}$ (`C2F_4_gain_form`, $p = 0.147$). Because the $2 \times 2$ design lacks a filter-only-without-outer-gain arm, we report this $89.6\% \pm 3.6\%$ decomposition explicitly as a contrast ratio conditional on the retained outer-gain setting rather than an interaction-free additive attribution. (2) Even `C2F-MG12-KeepFilter4` (`1.107784` at $3.0145 \times 10^{16}\text{ FLOPs}$, $\approx 5.0\%$ more FLOPs than the $2.8713 \times 10^{16}\text{ FLOPs}$ `p5_s240` runs) and full-budget `MG-V12-Pyr-Phase0` (`0.990401`, $+2.054\text{ mBPB}$ odd–even ZOH parity gap) trail static `Static-12L-PyrTarget` (`1.105832` / `0.983916`) and full-resolution `G_stack-PyrTarget-Medial4` (`1.093651` / `0.979259`), bounding this per-block stride-2 wrapper as a negative pretraining efficiency trade-off despite its transition fix.

---

## 6. Discussion and Scope Limitations

1. **Summary of Bounded Empirical Contributions Relative to Prior Literature**:
   - **Controlled Fixed-FLOP Evaluation of Vertical FFN Allocation (§3, Table 1B)**: Building on DeLighT [11] and OpenELM [10], we show that under fixed pretraining FLOPs at 16 layers, both a smooth linear FFN ramp (`Linear40`: $-4.328 \pm 0.268\text{ mBPB}$ vs. `Basal40`, $-1.656 \pm 0.337\text{ mBPB}$ vs. `Unif2d`) and step-apical placement (`Pyr40`: $-3.799 \pm 0.201\text{ mBPB}$ vs. `Basal40`, persisting under all-global `LLLL` at $-3.716\text{ mBPB}$ and at reduced width $d=384, H=3$ at $-3.143\text{ mBPB}$) outperform basal and uniform allocations, whereas linear ramping at 8 layers is near-neutral (`U0_s8_linear = -0.400 ± 0.324 mBPB`). Uncalibrated Logit-Lens and SVD stable-rank profiles (Table 1C) are reported strictly as exploratory descriptive associations [12].
   - **8-Seed Splicing Evidence and Explicit Checkpointing FLOP Trade-Offs (§4, Tables 2A–2B)**: On our 8-seed bit-identical `Unif4d` Stage-1 checkpoint fork at fixed `{3,7,11}` global depths, interior period-preserving growth beats contiguous apical `G_stack` by $-2.384 \pm 0.773\text{ mBPB}$ (`U6_true_8s`, $p = 0.0177, p_{\text{Holm6}} = 0.0530$), while the two-stage interior-pyramidal package beats apical `G_stack` by $-3.317 \pm 0.661\text{ mBPB}$ (`U7_true_8s`, $p_{\text{Holm6}} = 0.00614$) and 953-step static 8L by $-4.395 \pm 0.368\text{ mBPB}$ (`U0_true_8s`, $p_{\text{Holm6}} = 0.000040$). At the same time, our 1,068-step uncheckpointed static 8L control (`U0_exec_true = +2.857 ± 0.334 mBPB`) demonstrates that when 12L training requires 6-block activation checkpointing (`+12.04%` analytical executed FLOPs), an uncheckpointed 8L model given those extra FLOPs as $+115$ token steps achieves lower aggregate BPB, even though 12L growth retains a $-16.073\text{ mBPB}$ advantage on long-distance bigrams and wins by $-4.182\text{ mBPB}$ when both models run without checkpointing (`Split8x32`).
   - **Matched-Configuration C2F Filter Retention (§5, Table 4)**: Across 4 seeds at exact iso-compute ($3.0145 \times 10^{16}\text{ FLOPs}$) and conditional on retaining the outer block gain, `KeepFilter4` beats `OuterGainNoFilter4` by $-17.187 \pm 1.370\text{ mBPB}$ ($p = 0.00109$, $89.6\% \pm 3.6\%$ of the $-19.188\text{ mBPB}$ recovery over naive unfolding).
2. **Limitations**: Because `Medial4` (`{3,7,11}`, interface $5\to 2$) and `Medial-WinLock` (`{3,5,11}`, interfaces $3\to 2, 5\to 4$) differ in destination global-layer depths as well as splice count, `U4_true` supports contiguous period preservation over this parent-window-locked two-splice layout rather than an arbitrary-interface splice-count theorem. All 242 runs are conducted at $50\text{M}\text{--}80\text{M}$ parameters ($d \in \{384, 512\}$, Muon/AdamW) on ClimbMix; verification at multi-billion-parameter scales and persistent token-routing hierarchies [7] remains open.

---

## References
1. **rStation `general_pretraining_7/v3`**: *Copy-stacking depth growth beats time-matched scratch training at two transitions, and its margin depends on when the trunk is grown*.
2. **rStation `general_pretraining_24/v4`**: *Hadamard Modal-Decoupled Late Morphogenesis: Overcoming the Cooldown Horizon in Progressive Language Model Growth*.
3. **rStation `general_pretraining_26/v2`**: *Complementary Microbatch Braiding: Non-Cloning Two-Stage Depth Training and Controlled Comparisons Against Early Weight Cloning Under Fixed Pretraining FLOPs*.
4. **rStation `general_pretraining_28/v1`**: *Two-Stage Depth Growth and Moment-Free Hashed Surface Memory Under Fixed Pretraining Matrix/Attention FLOPs: Static Depth Inversion, Uniform-Microbatch Controls, and Exposure-Frozen Diagnostics*.
5. **Chen, T., Goodfellow, I., & Shlens, J. (2016)**. *Net2Net: Accelerating Learning via Knowledge Transfer*. ICLR 2016, arXiv:1511.05641.
6. **Du, W., et al. (2024)**. *Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training*. NeurIPS 2024, arXiv:2405.15319v2 (Appendices H.2–H.3 and J).
7. **Nawrot, P., et al. (2022)**. *Hierarchical Transformers Are More Efficient Language Models*. Findings of NAACL 2022, pp. 1559–1571 (arXiv:2110.13711).
8. **Gong, L., et al. (2019)**. *Efficient Training of BERT by Progressively Stacking*. ICML 2019, PMLR 97:2337–2346.
9. **Dai, Z., Lai, G., Yang, Y., & Le, Q. V. (2020)**. *Funnel-Transformer: Filtering Out Sequential Redundancy for Efficient Language Processing*. NeurIPS 2020, arXiv:2006.03236.
10. **Mehta, S., Sekhavat, M. H., Cao, Q., Horton, M., Jin, Y., Sun, C., Mirzadeh, I., Najibi, M., Belenko, D., Zatloukal, P., & Rastegari, M. (2024)**. *OpenELM: An Efficient Language Model Family with Open Training and Inference Framework*. arXiv:2404.14619 (§2.1, layer-wise linear FFN multiplier ramping).
11. **Mehta, S., Ghazvininejad, M., Iyer, S., Zettlemoyer, L., & Hajishirzi, H. (2021)**. *DeLighT: Deep and Light-weight Transformer*. ICLR 2021, arXiv:2008.00623.
12. **Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., & Steinhardt, J. (2023)**. *Eliciting Latent Predictions from Transformers with the Tuned Lens*. arXiv:2303.08112.

---

## Appendix A: Full-Budget `262k`-Tok/Step Arm Table, `Split8x32` Triad, and Descriptive Token-Class Decompositions

**Table A1: Complete 21-Arm Full-Budget ($1.1945 \times 10^{17}$ Analytical Nominal & Executed FLOPs, $0\%$ Recomp, $262{,}144$ tok/step `4x32`) Summaries (`Mean ± SD`, matches `paper5_tables_verified.txt` lines 5–25)**

| Arm Name & FFN Profile | $n$ | Params | Steps ($g$) | Tokens | `val_bpb` ($\text{Mean} \pm \text{SD}$) | `holdout_bpb` ($\text{Mean} \pm \text{SD}$) | `sh9A` BPB ($\text{Mean} \pm \text{SD}$) | `sh9A` Bigram Nov / Short / Long | $\Delta\mathcal{L}$ (nats) | Train (s) / VRAM |
| :--- | :-: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| `Static-8L-Unif4d` ($4^8$) | 4 | $50.33\text{M}$ | $1{,}906$ ($-$) | $499.65\text{M}$ | $0.985985 \pm 0.000314$ | $0.985197 \pm 0.000264$ | $0.986600 \pm 0.000315$ | $1.127654$ / $0.357426$ / $0.820019$ | $0.0000$ | $1093.4\text{s}$ / $11702\text{MB}$ |
| `Static-12L-Unif4d` ($4^{12}$) | 4 | $71.30\text{M}$ | $1{,}318$ ($-$) | $345.51\text{M}$ | $0.984910 \pm 0.001510$ | $0.984044 \pm 0.001550$ | $0.985488 \pm 0.001507$ | $1.126379$ / $0.357672$ / $0.799072$ | $0.0000$ | $1098.9\text{s}$ / $16941\text{MB}$ |
| `Static-12L-PyrTarget` ($2^2,4^8,6^2$) | 4 | $71.30\text{M}$ | $1{,}318$ ($-$) | $345.51\text{M}$ | $0.983343 \pm 0.000717$ | $0.982491 \pm 0.000806$ | $0.983916 \pm 0.000783$ | $1.124697$ / $0.356456$ / $0.801790$ | $0.0000$ | $1103.7\text{s}$ / $16941\text{MB}$ |
| `G_stack-Unif4d-Medial` ($4^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}456$ ($450$) | $381.68\text{M}$ | $0.980783 \pm 0.000446$ | $0.979898 \pm 0.000437$ | $0.981347 \pm 0.000392$ | $1.122179$ / $0.353539$ / $0.802880$ | $+0.1436$ | $1093.1\text{s}$ / $17086\text{MB}$ |
| `G_stack-Unif4d-Medial4` ($4^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}456$ ($450$) | $381.68\text{M}$ | $0.979218 \pm 0.000723$ | $0.978359 \pm 0.000799$ | $0.979791 \pm 0.000731$ | $1.120563$ / $0.352014$ / $0.809056$ | $+0.1037$ | $1090.8\text{s}$ / $17086\text{MB}$ |
| `G_stack-PyrTarget-Medial4` ($2^2 4^4 6^2\to 2^2 4^8 6^2$) | 4 | $71.30\text{M}$ | $1{,}456$ ($450$) | $381.68\text{M}$ | $0.978707 \pm 0.000603$ | $0.977766 \pm 0.000459$ | **$0.979259 \pm 0.000614$** | $1.119797$ / $0.352826$ / $0.799177$ | $+0.1783$ | $1102.6\text{s}$ / $17086\text{MB}$ |
| `M2D-Unif4d-Medial` ($2^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($315$) | $382.73\text{M}$ | $0.981718 \pm 0.000942$ | $0.980859 \pm 0.000936$ | $0.982323 \pm 0.000937$ | $1.123114$ / $0.354754$ / $0.802233$ | $+0.1568$ | $1098.5\text{s}$ / $17054\text{MB}$ |
| `M2D-Unif4d-Medial4` ($2^8\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($315$) | $382.73\text{M}$ | $0.980002 \pm 0.001115$ | $0.979054 \pm 0.001222$ | $0.980562 \pm 0.001166$ | $1.121361$ / $0.353148$ / $0.794470$ | $+0.1334$ | $1096.7\text{s}$ / $17054\text{MB}$ |
| `Pyr-Apical2-Apical` ($2^6 4^2\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($343$) | $382.73\text{M}$ | $0.981052 \pm 0.001174$ | $0.980221 \pm 0.001166$ | $0.981643 \pm 0.001165$ | $1.121591$ / $0.357588$ / $0.810487$ | $+0.1756$ | $1092.7\text{s}$ / $17062\text{MB}$ |
| `Pyr-Apical2-Medial` ($2^6 4^2\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($343$) | $382.73\text{M}$ | $0.981493 \pm 0.000671$ | $0.980549 \pm 0.000717$ | $0.982033 \pm 0.000645$ | $1.122618$ / $0.355621$ / $0.794589$ | $+0.1530$ | $1096.4\text{s}$ / $17062\text{MB}$ |
| `Pyr-Apical2-Medial4` ($2^6 4^2\to 4^{12}$) | 4 | $71.30\text{M}$ | $1{,}460$ ($343$) | $382.73\text{M}$ | $0.979890 \pm 0.000676$ | $0.978880 \pm 0.000677$ | $0.980392 \pm 0.000657$ | $1.121140$ / $0.353495$ / $0.785159$ | $+0.1426$ | $1096.0\text{s}$ / $17062\text{MB}$ |
| `PyrTarget-SplitAp-Medial4` ($2^6 3^2\to 2^2 4^8 6^2$) | 4 | $71.30\text{M}$ | $1{,}460$ ($328$) | $382.73\text{M}$ | $0.978926 \pm 0.001707$ | $0.977953 \pm 0.001819$ | $0.979450 \pm 0.001759$ | $1.119964$ / $0.353297$ / $0.794008$ | $+0.1151$ | $1108.9\text{s}$ / $17058\text{MB}$ |
| `PyrTarget-KeepAp-Medial4` ($2^6 6^2\to 2^2 4^8 6^2$) | 4 | $71.30\text{M}$ | $1{,}460$ ($375$) | $382.73\text{M}$ | $0.978697 \pm 0.000647$ | $0.977744 \pm 0.000602$ | **$0.979257 \pm 0.000576$** | $1.119788$ / $0.353345$ / $0.783655$ | $+0.1445$ | $1106.0\text{s}$ / $17070\text{MB}$ |
| `Static-16L-Unif2d` ($2^{16}$) | 4 | $75.50\text{M}$ | $1{,}294$ ($-$) | $339.21\text{M}$ | $0.988647 \pm 0.001297$ | $0.987799 \pm 0.001287$ | $0.989254 \pm 0.001271$ | $1.131080$ / $0.356924$ / $0.812456$ | $0.0000$ | $1131.6\text{s}$ / $17860\text{MB}$ |
| `Static-16L-Basal40` ($6^2,2^{14}$) | 4 | $79.69\text{M}$ | $1{,}208$ ($-$) | $316.67\text{M}$ | $0.991377 \pm 0.000668$ | $0.990533 \pm 0.000696$ | $0.991926 \pm 0.000642$ | $1.133720$ / $0.359893$ / $0.810332$ | $0.0000$ | $1118.5\text{s}$ / $18940\text{MB}$ |
| `Static-16L-Pyr40` ($2^{14},6^2$) | 4 | $79.69\text{M}$ | $1{,}208$ ($-$) | $316.67\text{M}$ | $0.987565 \pm 0.000750$ | $0.986664 \pm 0.000807$ | $0.988126 \pm 0.000833$ | $1.129301$ / $0.358648$ / $0.813951$ | $0.0000$ | $1122.0\text{s}$ / $18940\text{MB}$ |
| `Static-16L-Linear40` ($1^4,2^4,3^4,4^4$) | 4 | $79.69\text{M}$ | $1{,}208$ ($-$) | $316.67\text{M}$ | $0.987008 \pm 0.001136$ | $0.986120 \pm 0.001101$ | $0.987598 \pm 0.001086$ | $1.128686$ / $0.358568$ / $0.811452$ | $0.0000$ | $1124.7\text{s}$ / $18940\text{MB}$ |
| `Pyr16-2Stage-Copy-Medial4` ($2^6 6^2\to 2^{14} 6^2$) | 4 | $79.69\text{M}$ | $1{,}360$ ($353$) | $356.52\text{M}$ | $0.980981 \pm 0.000245$ | $0.979979 \pm 0.000214$ | $0.981488 \pm 0.000250$ | $1.122771$ / $0.352298$ / $0.782633$ | $+0.4246$ | $1106.5\text{s}$ / $19069\text{MB}$ |
| `Pyr16-3Stage-Copy-Medial4` ($2^6 6^2\to 2^{10} 6^2\to 2^{14} 6^2$) | 4 | $79.69\text{M}$ | $1{,}360$ ($216,490$) | $356.52\text{M}$ | $0.980401 \pm 0.000503$ | $0.979382 \pm 0.000480$ | **$0.980932 \pm 0.000512$** | $1.122064$ / $0.352393$ / $0.782879$ | $+0.0939$/$+0.0681$ | $1128.3\text{s}$ / $19385\text{MB}$ |
| `MG-V12-Pyr-Phase0` / `MG-V16-Pyr-Phase0` | 1 | $71.3$/$79.7\text{M}$ | $1600$ / $1626$ | $419.4$/$426.3\text{M}$ | $0.989659$ / $0.993613$ | $0.988883$ / $0.992912$ | $0.990401$ / $0.994328$ | $1.132300$ / $1.136725$ (nov) | $0.0000$ | $1143$/$1206\text{s}$ |

**Table A2: Uncheckpointed `Split8x32` ($524{,}288$ tok/step, $B=32$, $0\%$ Recomp) Summaries & `Slice B` Descriptive Token-Class Profiles (`paper5_tables_verified.txt` lines 90–144)**

- **`Split8x32` Arms (`Mean ± SD`, 4 seeds)**: `Static-12L-PyrTarget-Split8x32`: `Slice B` $0.997561 \pm 0.000473$, `confirm` $0.999504 \pm 0.000534$, `sh9A` $0.998061 \pm 0.000498$ (`1075.4s`, `16941MB`); `G_stack-PyrTarget-Medial-Split8x32`: `Slice B` $0.993496 \pm 0.001336$, `confirm` $0.995310 \pm 0.001339$, `sh9A` $0.993954 \pm 0.001336$, $\Delta\mathcal{L}=+0.0970$ (`1078.7s`, `17086MB`); `G_stack-PyrTarget-Medial4-Split8x32`: `Slice B` $0.991225 \pm 0.001613$, `confirm` $0.993085 \pm 0.001556$, `sh9A` $0.991676 \pm 0.001581$, $\Delta\mathcal{L}=+0.0899$ (`1084.3s`, `17086MB`). Paired `Split8x32` contrasts ($\text{Mean }\Delta \pm \text{SE}$): `U1_split = -6.336 ± 0.645 mBPB` ($p = 0.00224$), `U2_split = -2.271 ± 0.154 mBPB` ($p = 0.000673$), `U3_split = -4.065 ± 0.517 mBPB` ($p = 0.00429$). Remaining secondary contrasts at $262\text{k}$ tok/step: `S6` (`G_stack-Unif4d-Medial4` vs. `M2D-Unif4d-Medial4`): $-0.771 \pm 0.337\text{ mBPB}$ ($p=0.106$); `S10` (`KeepAp` vs. `SplitAp`): $-0.194 \pm 0.658\text{ mBPB}$ ($p=0.787$).
- **`Slice B` Descriptive Bigram (`bi_nov` $81.3\%$, `bi_s` $18.1\%$, `bi_l` $0.55\%$) & Unigram (`uni_nov` $47.1\%$, `uni_s` $51.4\%$, `uni_l` $1.53\%$) Profiles (`paper5_tables_verified.txt` lines 97–115)**: Across `I1`, `I1b`, `T1`, and `T2`, apical `Pyr40` and smooth-ramp `Linear40` primarily improve novel compositional prediction (`I1` `Slice A`: $\Delta\text{bi\_nov} = -4.419\text{ mBPB}$; `I1b` `Slice A`: $\Delta\text{bi\_nov} = -5.034\text{ mBPB}$; `T1` `Slice B`: $\Delta\text{bi\_nov} = -4.315\text{ mBPB}$; `T2` `Slice B`: $\Delta\text{bi\_nov} = -3.783\text{ mBPB}$). Under `True4x64` depth growth across 8 seeds (`paper5_tables_verified.txt` lines 97–107), `U0_true_8s` (`PyrTarget-Medial4` vs. `Static-8L-m953`) gives $\Delta\text{bi\_nov} = -4.470$, $\Delta\text{bi\_s} = -3.859$, $\Delta\text{bi\_l} = -10.855\text{ mBPB}$ (4-seed subset: $-4.259, -3.420, -21.502\text{ mBPB}$); `U6_true_8s` (`Unif4d-Medial4` vs. `Unif4d-Apical`) gives $\Delta\text{bi\_nov} = -1.842$, $\Delta\text{bi\_s} = -4.550$, $\Delta\text{bi\_l} = -11.234\text{ mBPB}$ (4-seed subset: $-1.978, -4.317, -10.600\text{ mBPB}$); `U7_true_8s` gives $\Delta\text{bi\_nov} = -3.065$, $\Delta\text{bi\_s} = -4.149$, $\Delta\text{bi\_l} = -13.236\text{ mBPB}$ (4-seed subset: $-3.046, -3.576, -22.571\text{ mBPB}$); and `U0_exec_true` (4 seeds, vs. `Static-8L-m1068`) gives $\Delta\text{bi\_nov} = +3.585$, $\Delta\text{bi\_s} = +0.160$, $\Delta\text{bi\_l} = -16.073\text{ mBPB}$.

---

## Appendix B: Complete Quarter-Budget (`s240`, $2.87 \times 10^{16}$ FLOPs) Screening & Phase-6 Single-Seed Tables

**Table B1: 31 Phase-5 `s240` Runs (`4x32` Split) & 3 Phase-7 Forked `s240` Runs (`2x64` Unsplit `B=64`, `seed = 42`)**

| Run ID | Steps ($g$) | `val_bpb` / `sh9A` BPB ($\Delta\mathcal{L}$) | Run ID | Steps ($g$) | `val_bpb` / `sh9A` BPB ($\Delta\mathcal{L}$) |
| :--- | :---: | :---: | :--- | :---: | :---: |
| `p5_s240_static8_unif4d_m240_s42` | $480$ ($-$) | $1.086468$ / $1.087076$ ($0.0000$) | `p5_s240_pyr_hg4_modalw_apical_m184_g95_s42` | $368$ ($95$) | $1.106153$ / $1.106837$ ($+0.0001$) |
| `p5_s240_static12_unif4d_m166_s42` | $332$ ($-$) | $1.108623$ / $1.109354$ ($0.0000$) | `p5_s240_pyr_hg4_modalw_medial_m184_g95_s42` | $368$ ($95$) | $1.104336$ / $1.105082$ ($+0.0045$) |
| `p5_s240_static12_pyrtarget_m166_s42` | $332$ ($-$) | $1.104939$ / $1.105832$ ($0.0000$) | `p5_s240_pyr_hg4_modalw_copy_apical_m184_g95_s42` | $368$ ($95$) | $1.099664$ / $1.100392$ ($+0.1078$) |
| `p5_s240_static16_unif2d_m163_s42` | $326$ ($-$) | $1.110256$ / $1.110991$ ($0.0000$) | `p5_s240_pyr_hg4_modalw_copy_medial_m184_g95_s42` | $368$ ($95$) | $1.098698$ / $1.099424$ ($+0.0761$) |
| `p5_s240_static16_pyr40_m152_s42` | $304$ ($-$) | $1.115051$ / $1.115797$ ($0.0000$) | `p5_s240_pyr_hg6_modalw_medial_m184_g87_s42` | $368$ ($87$) | $1.100356$ / $1.101139$ ($+0.0019$) |
| `p5_s240_gstack_medial_m184_g118_s42` | $368$ ($118$) | $1.098641$ / $1.099347$ ($+0.0705$) | `p5_s240_pyr_hg6_modalw_copy_apical_m184_g87_s42` | $368$ ($87$) | $1.096717$ / $1.097459$ ($+0.0958$) |
| `p5_s240_gstack_medial4_m184_g118_s42` | $368$ ($118$) | $1.095264$ / $1.095874$ ($+0.0679$) | `p5_s240_pyr_hg6_modalw_copy_medial_m184_g87_s42` | $368$ ($87$) | $1.096215$ / $1.096928$ ($+0.0706$) |
| `p5_s240_gstack_pyrtarget_medial4_m184_g118_s42` | $368$ ($118$) | **$1.092842$ / $1.093651$** ($+0.0636$) | `p5_s240_pyr_hg6_modalw_copy_medial4_m184_g87_s42` | $368$ ($87$) | $1.096367$ / $1.097090$ ($+0.0664$) |
| `p5_s240_m2d_modalw_medial_m184_g80_s42` | $368$ ($80$) | $1.100769$ / $1.101387$ ($+0.0022$) | `p5_s240_pyrtarget_splitap_modalw_medial4_m184_g83_s42` | $368$ ($83$) | $1.095189$ / $1.095888$ ($+0.0554$) |
| `p5_s240_m2d_modalw_copy_medial_m184_g80_s42` | $368$ ($80$) | $1.097087$ / $1.097751$ ($+0.0512$) | `p5_s240_pyrtarget_keepap_modalw_medial4_m184_g95_s42` | $368$ ($95$) | $1.094830$ / $1.095471$ ($+0.0568$) |
| `p5_s240_m2d_modalw_copy_medial4_m184_g80_s42` | $368$ ($80$) | $1.096210$ / $1.096894$ ($+0.0452$) | `p5_s240_pyr16_2stage_eqpart_medial4_m171_g88_s42` | $342$ ($88$) | $1.101026$ / $1.101660$ ($+0.0484$) |
| `p5_s240_pyr_apical2_modalw_copy_apical_m184_g87_s42` | $368$ ($87$) | $1.096835$ / $1.097446$ ($+0.0685$) | `p5_s240_pyr16_2stage_copy_medial4_m171_g88_s42` | $342$ ($88$) | $1.100037$ / $1.100741$ ($+0.1416$) |
| `p5_s240_pyr_apical2_modalw_copy_medial_m184_g87_s42` | $368$ ($87$) | $1.096949$ / $1.097621$ ($+0.0481$) | `p5_s240_pyr16_3stage_modaleq_medial4_m171_g54_g122_s42` | $342$ ($54,122$) | $1.105312$ / $1.105986$ ($+0.0075$/$-0.0004$) |
| `p5_s240_pyr_apical2_modalw_copy_medial4_m184_g87_s42` | $368$ ($87$) | $1.096278$ / $1.097012$ ($+0.0372$) | `p5_s240_pyr16_3stage_copy_medial4_m171_g54_g122_s42` | $342$ ($54,122$) | $1.101907$ / $1.102592$ ($+0.0276$/$+0.0542$) |
| `p5_s240_pyr_apical4_modalw_copy_apical_m184_g95_s42` | $368$ ($95$) | $1.098535$ / $1.099252$ ($+0.0586$) | `p7_s240_fork_pyrtarget_medial4_m184_g118_s42` (`Slice B` $1.093158$) | $368$ ($118$) | $1.092767$ / $1.093552$ ($+0.0742$) |
| `p5_s240_pyr_basal2_modalw_copy_apical_m184_g87_s42` | $368$ ($87$) | $1.098424$ / $1.099308$ ($+0.0568$) | `p7_s240_fork_pyrtarget_medial_pos_m184_g118_s42` (`Slice B` $1.094931$) | $368$ ($118$) | $1.094561$ / $1.095345$ ($+0.1080$) |
| `p5_s240_pyr_mid2_modalw_copy_medial_m184_g87_s42` | $368$ ($87$) | $1.098348$ / $1.099023$ ($+0.0651$) | `p7_s240_fork_pyrtarget_medial_winlock_m184_g118_s42` (`Slice B` $1.094780$) | $368$ ($118$) | $1.094397$ / $1.095246$ ($+0.1091$) |

**Table B2: Remaining Single-Seed Phase-6 `s240` Coarsening Diagnostics (`seed = 42`)**: `MG-V12-Pyr-Haar` ($[6..9]$, $404$ st): `sh9A` $1.105109$ (Odd$-$Even $-1.378\text{ mBPB}$); `MG-V12-Pyr-Phase0` ($[6..9]$, $404$ st): `sh9A` $1.103488$ ($+1.884\text{ mBPB}$); `MG-V16-Pyr-Haar` ($[6..13]$, $410$ st): `sh9A` $1.107446$ ($-1.531\text{ mBPB}$); `MG-V16-Pyr-Phase0` ($[6..13]$, $410$ st): `sh9A` $1.106293$ ($+3.357\text{ mBPB}$); `C2F-MG12-Base` ($[2..9]$, $368$ st, $g=103$): `sh9A` $1.108194$ ($\Delta\mathcal{L}=+3.3181$); `C2F-MG12-ScaleFold` ($[2..9]$, $368$ st, $g=103$): `sh9A` $1.109041$ ($\Delta\mathcal{L}=+2.8754$); `C2F-MG12-Decimate` ($[2..9]$, $368$ st, $g=103$, $\gamma_R=0$): `sh9A` $1.107121$ ($\Delta\mathcal{L}=+2.2078$).

---

## Appendix C: Portable Reproducibility & Supplement Verification

`supplement_p5.zip` (`< 2,000,000` bytes compressed and expanded, `<= 200` archive entries) includes all **242 `result.json` records** (`138` Phase-5 runs, `25` Phase-6 runs, `67` Phase-7 runs, and `12` Revision-6 smooth-ramp / Logit-Lens runs; single-seed screening and auxiliary runs are bundled in `experiments_p5/s240_screening_runs.json` and auto-unpacked by `ensure_s240_unpacked()`), `experiments_p5/learning_curves_summary.json`, all training, evaluation, and backend scripts (`experiments_p5/train_pyramidal.py`, `experiments_p5/prepare.py`, `experiments_p5/attention_backend.py`, `experiments_p6/train_multigrid.py`, `experiments_p7/train_loop_grow.py`, `experiments_revival/train_revival.py`), `experiments_p5/shard9_provenance.json` (automatically resolved by `train_pyramidal.py`, `train_multigrid.py`, and `train_loop_grow.py`), summary CSVs (`experiments_p5/results/all_runs_p{5,6,7,revival}_summary.csv`), and `supplement/figure-1.png`–`figure-4.png`. Set `PYTHONPATH=experiments_p5` when running from the repository root in any Python 3.10+ environment with `numpy`, `scipy`, and `matplotlib`:
```bash
PYTHONPATH=experiments_p5 python3 experiments_p5/verify_paper5_tables.py --strict
PYTHONPATH=experiments_p5 python3 experiments_p5/make_paper5_figures.py
# Optional if raw ClimbMix parquet shards are available under DATA_DIR:
PYTHONPATH=experiments_p5 python3 experiments_p5/verify_shard9_provenance.py
```
