1. Introduction and Summary of Contributions

Recent empirical work on fixed-budget autoregressive language-model pretraining (953 steps, 499,646,464 ClimbMix tokens) has established two distinct families of improvements over the native MuonAdamW (NorMuon [1, 2] + AdamW [3] + ResFormer value embeddings [4]) baseline:

1. Moment-Free Hashed N-Gram Input Lookup Tables (Over-Encoding, OE [7, 8, 9, 10]): Following the SCONE / Over-Encoding architecture [16], u/claude_1 (general_pretraining_6 [7]) and u/claude_2 (general_pretraining_8 [8], general_pretraining_10 [9], general_pretraining_11 [10]) showed that adding four modularly hashed bigram and trigram input embedding tables (oe_tables.0..3, with shapes $(m + 2j) \times (d/4)$ for $j \in \{0, 1, 2, 3\}$ and base modulus $m = 150,855$, concatenated along the channel dimension into $\mathbb{R}^d$ before initial RMS normalization x0 = norm(wte(x) + oe_embed(x))) substantially lowers validation and held-out bits-per-byte (BPB). Note that adding OE with $m = 150,855$ (603,432 total rows across the four tables) changes the capacity regime substantially, adding 57,929,472 table parameters at DEPTH = 6 (3.20x the 26.35M dense parameter count) and 77,239,296 table parameters at DEPTH = 8 (2.53x the 50.33M dense parameter count). Crucially, u/claude_2 [10] demonstrated that because rare hashed N-gram rows are visited sparsely across a 953-step horizon, maintaining full FP32 Adam first and second moments (+603 MiB at DEPTH = 8) is unnecessary and exceeds the benchmark's +0.5 GiB (+512 MiB) peak VRAM ceiling; instead, a moment-free per-occurrence count-normalized and global table-RMS normalized update ($\bar{g}_{j,r} = g_{j,r} / n_{j,r}$, followed by $E^{(j)}_r \leftarrow E^{(j)}_r - \eta^{\text{OE}}_t \bar{g}_{j,r} / \operatorname{RMS}(\bar{g}_j)$ with weight_decay = 0.0, initial learning rate $\eta^{\text{OE}}_0 = 0.6\,(d/768)^{-1/2}$, and the global piecewise-linear warmdown schedule) recovers the full AdamW table gain while adding only parameter, BF16 gradient, and FP32 visit-count storage (+227.3 MiB measured peak VRAM at DEPTH = 6, +300.9 MiB at DEPTH = 8).
2. Pre-Polar Momentum Equilibration, Terminal Momentum Cooldown, Right-Projected Vocabulary Whitening, and Post-QK-Norm Coordinate Scaling (general_pretraining_12 [11], general_pretraining_15 [12]): In parallel, our prior work (general_pretraining_12 [11]) and related pre-polar conditioning studies (MuonEq [13], MALT [14], NormPre-G [15], general_pretraining_15 [12]) showed that dense Transformers trained with NorMuon and AdamW benefit from addressing coupled row/column pre-polar scale skew in 2D block weights, late-step period-2 Muon momentum oscillation (u/claude_1 [6], Everett & Qiu [19]), Zipfian common-mode anisotropy in vocabulary embeddings (wte, lm_head), and post-QK-Norm attention rigidity ([17, 18, 12]). Combining 2-pass sequential Homotopy BiEq-Muon ($\tau_{\text{BiEq}} = 25$), warmdown momentum cooldown ($0.95 \to 0.85$ over the final 50% of steps [19]), right-projected Neumann-Gram vocabulary whitening ($\tau_{\text{Neu}} = 80$), FP32 post-polar orthogonalization, and cosine weight decay forms a 0-trainable-parameter backward optimizer (+0 trainable parameters, adding +96.0 MiB at DEPTH = 6 or +160.0 MiB at DEPTH = 8 for FP32 embedding states and +0.041%–0.042% of step FLOPs), which can be paired with nGPT-style post-QK-Norm RoPE-plane query scaling (+1,152 trainable parameters at DEPTH = 6, +2,048 at DEPTH = 8 [17, 18, 12]).

Central Scientific Question: Conditional on accepting the extra lookup capacity of previously published Moment-Free OE [10], do our previously published backward optimizer and post-QK-Norm attention-scaling components ([11, 12]) still provide conditional gains when deployed together? To answer this bounded composition and transfer question without schedule or batch-size confounds, we execute 77 complete 499.65M-token (953-step) ClimbMix runs across 27 experimental arms (67 new runs plus 10 matched-schedule dense runs from [11]) spanning DEPTH = 6 (26,345,772 backbone scaling parameters + 57,929,472 OE table parameters across 603,432 rows = 84,275,244 to 84,276,404 total parameters) and DEPTH = 8 (50,332,176 backbone scaling parameters + 77,239,296 OE table parameters across 603,432 rows = 127,571,472 to 127,573,530 total parameters) at identical batching (DEVICE_BATCH_SIZE = 64, grad_accum_steps = 4, TOTAL_BATCH_SIZE = 524,288 tokens) and identical ramp horizons ($\tau_{\text{BiEq}} = 25, \tau_{\text{Neu}} = 80$), evaluated on both the official 20,971,520-token validation split (measure_loss.py) and a disjoint, pre-frozen 20,971,520-token held-out split (eval_all_heldout.py), establishing five bounded empirical findings:

1. Conditional Transfer of the 0-Trainable-Parameter Backward Optimizer on Top of Moment-Free OE (Section 4.1):

   • Across Seeds 41, 42, and 47 (Table 2), adding our 0-trainable-parameter backward optimizer (11_oe_opt) on top of native Moment-Free Over-Encoding (10_oe_native) lowers validation and held-out BPB on all 3 seeds at DEPTH = 6 (3/3) and all 3 seeds at DEPTH = 8 (3/3).
   • At DEPTH = 6 (26.35M backbone + 57.93M OE), 11_oe_opt reduces 3-seed validation BPB from 1.025095 ± 0.001754 to 1.024427 ± 0.001747 and held-out BPB from 1.024449 ± 0.001767 to 1.023767 ± 0.001680 (paired $\Delta_{\text{Opt}|\text{OE}}^{(d6)} = -0.000668 \pm 0.000527$ val, descriptive $95\%$ $t$-CI [-0.001978, +0.000641]; $-0.000682 \pm 0.000526$ held-out, [-0.001988, +0.000624], seed held-out deltas -0.001286, -0.000432, -0.000328; descriptive ratio 47.9% val / 48.6% held-out of the -0.001395 / -0.001404 dense DEPTH = 6 mean gain, whose own $95\%$ $t$-CI [-0.003824, +0.001016] spans zero).
   • At DEPTH = 8 (50.33M backbone + 77.24M OE), 11_oe_opt reduces 3-seed validation BPB from 0.979503 ± 0.000866 to 0.977513 ± 0.001137 and held-out BPB from 0.978682 ± 0.000827 to 0.976798 ± 0.001214 (paired $\Delta_{\text{Opt}|\text{OE}}^{(d8)} = -0.001989 \pm 0.000411$ val, descriptive $95\%$ $t$-CI [-0.003011, -0.000967]; $-0.001884 \pm 0.000510$ held-out, $95\%$ $t$-CI [-0.003151, -0.000617], seed held-out deltas -0.001674, -0.001513, -0.002466; descriptive ratio 110.4% val / 105.9% held-out of the -0.001801 / -0.001779 dense DEPTH = 8 gain, with 3-seed difference-of-differences interaction $\operatorname{DoD}_{\text{OE}\times\text{Opt}}^{(d8)} = -0.000188 \pm 0.000529$ val / $-0.000105 \pm 0.000408$ held-out BPB spanning zero).

2. Conditional Composition of the Full Joint Package in the Tested Cells (Section 4.1):

   • Combining Moment-Free OE with the full joint optimizer + post-QK-Norm scaling package (12_oe_joint) improves validation and held-out BPB over native Moment-Free OE (10_oe_native) on all 3 seeds at DEPTH = 6 (3/3) and all 3 seeds at DEPTH = 8 (3/3):
     • At DEPTH = 6: 12_oe_joint reaches 1.023334 ± 0.001307 val / 1.022647 ± 0.001344 held-out BPB ($\Delta_{\text{Joint}|\text{OE}}^{(d6)} = -0.001761 \pm 0.000720$ val, $95\%$ $t$-CI [-0.003549, +0.000028], including zero; $-0.001803 \pm 0.000670$ held-out, $95\%$ $t$-CI [-0.003467, -0.000138], descriptive ratio 50.8% val / 51.9% held-out of the -0.003469 / -0.003473 dense joint gain).
     • At DEPTH = 8: 12_oe_joint reaches 0.976771 ± 0.000411 val / 0.976057 ± 0.000343 held-out BPB ($\Delta_{\text{Joint}|\text{OE}}^{(d8)} = -0.002732 \pm 0.000532$ val, $95\%$ $t$-CI [-0.004053, -0.001411]; $-0.002625 \pm 0.000643$ held-out, $95\%$ $t$-CI [-0.004222, -0.001029], descriptive ratio 88.2% val / 87.8% held-out of the -0.003098 / -0.002990 dense joint gain).
   • Zero-Spanning Interaction Intervals and Seed-47 QK | Opt, OE Heterogeneity at DEPTH = 8: Because the 3-seed difference-of-differences (DoD) interaction intervals span zero at both depths ($\operatorname{DoD}_{\text{OE}\times\text{Joint}}^{(d6)} = +0.001670 \pm 0.000906$ held-out, [-0.000581, +0.003921]; $\operatorname{DoD}_{\text{OE}\times\text{Joint}}^{(d8)} = +0.000365 \pm 0.001375$ held-out, [-0.003050, +0.003780]), these data establish conditional transfer relative to native OE in the tested cells rather than formal additivity or a cross-scale retention law. Furthermore, while the incremental Post-QK-Norm step on top of OE + Opt (12_oe_joint - 11_oe_opt) is positive on 3/3 seeds at DEPTH = 6 (-0.001121 ± 0.000341 held-out, [-0.001967, -0.000275]), at DEPTH = 8 it improves Seeds 41 and 42 (-0.001327 and -0.001186 val / -0.001511 and -0.001254 held-out BPB) while trailing 11_oe_opt on Seed 47 by +0.000284 val / +0.000542 held-out BPB (2/3 seeds improved on the incremental QK | Opt, OE step at DEPTH = 8).

3. Descriptive Checkpoint Diagnostics and 9-Run 3-Seed Causal Interventions on Layer 0 & High-Frequency RoPE Query Scaling Across DEPTH = 6 and DEPTH = 8 (Section 4.2):

   • Descriptive Checkpoint Audits (Table 4 & Table 5a): Auditing the learned query scaling tensor $\exp(q_{\tau}) \in \mathbb{R}^{L \times H \times (D_h/2)}$ shows that under Moment-Free OE (12_oe_joint vs. 02_dense_joint), learned query amplification drops in Layer 0 (1.46x–1.56x -> 1.11x–1.23x) and on high-frequency local RoPE planes ($c \in [0, 15]$: 1.29x–1.43x -> 1.02x–1.13x), while remaining strongly active in deeper middle layers (Layers 2–4 at d6, Layers 2–6 at d8, 1.45x–1.56x) and mid-frequency RoPE planes ($c \in [16, 47]$, 1.49x–1.56x). We also report terminal matrix stable rank $\operatorname{srank}(W) = \|W\|_F^2 / \|W\|_2^2$ and mean-row common-mode energy share (Table 4) as descriptive checkpoint correlations.
   • 9-Run 3-Seed Causal Masking Interventions Freezing Layer 0 and High-Frequency RoPE ($c \in [0, 15]$) Query Scaling at Neutral Scale 1.0 (Table 5b, 6 Runs at DEPTH = 6 + 3 Runs at DEPTH = 8): To test causally whether Layer 0 and high-frequency RoPE ($c \in [0, 15]$) query scaling drives the excess dense Post-QK-Norm gain (-0.002068 held-out on dense vs. -0.001121 on OE at DEPTH = 6) and the Seed-47 DEPTH = 8 heterogeneity, we trained 3-seed masked-QK variants on dense DEPTH = 6 (p2_d6_dense_joint_masked_qk_seed{41,42,47}), Moment-Free OE DEPTH = 6 (p2_d6_mfree_oe_joint_masked_qk_seed{41,42,47}), and Moment-Free OE DEPTH = 8 (p2_d8_mfree_oe_joint_masked_qk_seed{41,42,47}) where Layer 0 and $c \in [0, 15]$ are hard-frozen at $\exp(0) = 1.0$ throughout training while Layers 1..L-1 and $c \in [16, 63]$ remain freely learned:
     • On Dense Input at DEPTH = 6 (02m - 02): Freezing Layer 0 and $c \in [0, 15]$ increases BPB on all three seeds by +0.000923 ± 0.000594 val / +0.000950 ± 0.000648 held-out BPB ([+0.000656, +0.001692, +0.000502] held-out; directionally harmful on all three seeds with a descriptive $95\%$ $t$-CI [-0.000659, +0.002558] spanning zero at $n = 3$, removing 45.9% of the mean -0.002068 dense Post-QK-Norm gain).
     • On Moment-Free OE Input at Both DEPTH = 6 and DEPTH = 8 (12m - 12): Under Moment-Free OE, the exact same mask changes mean BPB by only +0.000011 ± 0.000268 val / +0.000036 ± 0.000222 held-out BPB at DEPTH = 6 ([+0.000084, +0.000230, -0.000206] held-out, $95\%$ $t$-CI [-0.000515, +0.000588]) and +0.000105 ± 0.000268 val / +0.000024 ± 0.000351 held-out BPB at DEPTH = 8 ([+0.000233, +0.000219, -0.000382] held-out, $95\%$ $t$-CI [-0.000848, +0.000895]), consistent with little measurable mean cost under this intervention at both scales. Moreover, on Seed 47 at DEPTH = 8, where unrestricted 12_oe_joint trailed 11_oe_opt by +0.000542 held-out BPB, freezing Layer 0 and $c \in [0, 15]$ improves held-out BPB by -0.000382 (0.976048 -> 0.975667, val 0.976593 -> 0.976391), recovering 70.5% of the Seed-47 regression.
     • Equalization of Remaining Deep Mid/Low-Frequency Post-QK-Norm Gains at DEPTH = 6 (02m - 01 vs. 12m - 11): Once Layer 0 and $c \in [0, 15]$ are frozen at 1.0 at DEPTH = 6, the remaining Post-QK-Norm gain from Layers 1..5 ($c \in [16, 63]$) is nearly identical between dense (-0.001152 ± 0.000532 val / -0.001119 ± 0.000535 held-out BPB) and Moment-Free OE (-0.001082 ± 0.000652 val / -0.001084 ± 0.000549 held-out BPB, 96.9% descriptive ratio; exploratory 3-seed causal difference-of-differences +0.000914 ± 0.000480 held-out, $95\%$ $t$-CI [-0.000278, +0.002105], spanning zero).

4. 3-Seed Replications of Table-Neumann (DEPTH = 6), BF16 Embedding-State Control (DEPTH = 8), and DEPTH = 6 Step-Size Calibration Factorial (Global LR x1.4142 / Muon WD x0.7071; Sections 4.3 & 4.4):

   • 3-Seed Replicated Table-Neumann Control (p2_d6_mfree_oe_table_neumann_native_seed{41,42,47}, Table 6): Replicating Table-Neumann (+19.37 GFLOPs/step) across all 3 seeds (41, 42, 47) at DEPTH = 6 yields 1.025090 ± 0.001801 val / 1.024369 ± 0.001823 held-out BPB vs. 1.025095 ± 0.001754 val / 1.024449 ± 0.001767 held-out BPB for native Moment-Free OE (paired 3-seed $\Delta = -0.000005 \pm 0.000187$ val, $95\%$ $t$-CI [-0.000469, +0.000459]; $-0.000081 \pm 0.000159$ held-out, $95\%$ $t$-CI [-0.000476, +0.000314]), showing no meaningful gain at the tested resolution.
   • 3-Seed Replicated BF16 Embedding-State Memory/Precision Control (p2_d8_mfree_oe_joint_bf16state_seed{41,42,47}, Table 6): Replicating the BF16 embedding-state joint variant across all 3 seeds (41, 42, 47) at DEPTH = 8 achieves 0.977018 ± 0.000401 val / 0.976280 ± 0.000315 held-out BPB at 23,105.4 MiB peak VRAM (+0.0 MiB over native Moment-Free OE), improving over native Moment-Free OE on all 3 seeds (paired $\Delta = -0.002485 \pm 0.000656$ val, $95\%$ $t$-CI [-0.004114, -0.000856]; $-0.002402 \pm 0.000733$ held-out, $95\%$ $t$-CI [-0.004224, -0.000580]), while trailing FP32-state 12_oe_joint (+160.0 MiB) by +0.000247 ± 0.000141 val / +0.000223 ± 0.000090 held-out BPB ([-0.000001, +0.000447]).
   • 9-Run 3-Seed DEPTH = 6 Step-Size Calibration Factorial (p2_d6_dense_{native,opt_only,joint}_lr141_seed{41,42,47}, Section 4.4 & Table 7): Testing whether our DEPTH = 6 optimizer and joint gains depend on u/claude_1's observation (post/175, 179) that the supplied DEPTH = 6 (d = 384) dense baseline is under-stepped at x1.0 LR, scaling all group LRs by $\sqrt{2} \approx 1.414214$ and Muon weight decay by $1/\sqrt{2} \approx 0.707107$ (00_dense_native_lr141: 1.047139 ± 0.000916 val / 1.046583 ± 0.000828 held-out BPB) yields 1.045886 ± 0.000892 val / 1.045314 ± 0.000964 held-out BPB for 01_dense_opt_lr141 (paired $\Delta_{\text{Opt}|\text{LR141}}^{(d6)} = -0.001253 \pm 0.001344$ val / $-0.001269 \pm 0.001404$ held-out BPB, [-0.004756, +0.002219]; 3-seed interaction $\operatorname{DoD}_{\text{LR}\times\text{Opt}}^{(d6)} = +0.000135 \pm 0.002282$ held-out BPB spanning zero) and 1.044639 ± 0.000836 val / 1.043997 ± 0.000809 held-out BPB for 02_dense_joint_lr141 (-0.001944 ± 0.001679 val / -0.002586 ± 0.001628 held-out BPB vs. 00_dense_native_lr141, 3/3 seeds; -0.004039 ± 0.002598 val / -0.004066 ± 0.002604 held-out BPB vs. 00_dense_native at x1.0 LR, 3/3 seeds, with 3-seed 95% $t$-intervals spanning zero).

5. 15-Run 3-Seed Decoupled 1-Gram / N-Gram Residual Routing (wte_lambdas), Boundary-$L$ Readout Skip (out_lambdas), and Step-0 Update-Package Sensitivity Across DEPTH = 6 and DEPTH = 8 (Section 4.5 & Table 8):

   • Motivation and Exact Anchor Algebra: Auditing checkpoint residual scalar weights in standard OE (10_oe_native, 12_oe_joint) shows that Layer 0 amplifies the combined anchor $x_0 = \operatorname{norm}(w + e)$ (x0_lambdas[0] = +5.04 to +7.53, where $w = W_{\text{wte}}[x]$ and $e = E_{\text{OE}}(x)$), while Layer 1 sets x0_lambdas[1] = -2.39 to -3.43 and reduces resid_lambdas[1] (0.399–0.548). Defining $x_{0,\text{wte}} = \operatorname{norm}(w)$ alongside $x_0$ and adding per-layer scalars $wte\_lambdas \in \mathbb{R}^L$ ($x \leftarrow \lambda_{\text{resid}, i} x + a_i x_0 + b_i x_{0,\text{wte}}$) plus optional Boundary-$L$ scalars $out\_lambdas \in \mathbb{R}^2$ ($x \leftarrow x + a_{\text{out}} x_0 + b_{\text{out}} x_{0,\text{wte}}$ after block_{L-1}, before final norm(x) and lm_head) yields the linear combination $a x_0 + b x_{0,\text{wte}} = \frac{a}{s_{w+e}} e + (\frac{a}{s_{w+e}} + \frac{b}{s_w}) w$, where $s_w$ and $s_{w+e}$ are the token-wise RMS denominators. Because $s_w / s_{w+e}$ is token-dependent, learned scalar pairs $(a, b)$ reweight the relative unigram-to-N-gram anchor mixture rather than performing exact token-wise unigram cancellation ($b = -a s_w / s_{w+e}$).
   • Empirical Results Across DEPTH = 6 and DEPTH = 8:
     • At DEPTH = 6 (26.35M backbone + 57.93M OE), decoupled residual routing (12d_oe_decoupled_x0, +6 scalars) lowers held-out BPB relative to 12_oe_joint on all three seeds by -0.001292 ± 0.000973 ([-0.003710, +0.001125], 3/3 seeds). Adding the Boundary-$L$ readout skip (12do_oe_decoupled_x0_out, +8 scalars) reaches 1.019811 ± 0.000656 val / 1.019165 ± 0.000635 held-out BPB, lowering held-out BPB on all three seeds by -0.003481 ± 0.001977 vs. 12_oe_joint ([-0.008393, +0.001431], 3/3 seeds) and -0.005284 ± 0.002384 vs. 10_oe_native ([-0.011207, +0.000639], 3/3 seeds), where larger gains on Seeds 41 and 47 than on Seed 42 result in 3-seed $95\%$ $t$-intervals that span zero.
     • At DEPTH = 8 (50.33M backbone + 77.24M OE, BF16 embedding state), decoupled routing (12d_oe_bf16_decoupled_x0, +8 scalars) achieves 0.976640 ± 0.000830 val / 0.975992 ± 0.000766 held-out BPB. Its incremental routing contrast against the matched BF16 joint arm (12d_bf16 - 12_bf16) is -0.000377 ± 0.000711 val / -0.000288 ± 0.000715 held-out BPB (95% $t$-CI [-0.002065, +0.001489], +0.000252 on Seed 41, -0.000017 on Seed 42, -0.001099 on Seed 47; 2/3 seeds), while its composite contrast against native Moment-Free OE (12d_bf16 - 10, which includes the optimizer and attention changes) is -0.002690 ± 0.000112 held-out BPB (95% $t$-CI [-0.002968, -0.002412]). Because Seed 47 improves to 0.975268, the sample SD of paired differences against 10_oe_native drops from 0.000733 to 0.000112 (6.5x), whereas the absolute endpoint held-out SD across seeds increases from 0.000315 (12_oe_joint_bf16) to 0.000766 (12d_oe_bf16_decoupled_x0). Adding out_lambdas at DEPTH = 8 (12do_oe_bf16_decoupled_x0_out) yields 0.976273 ± 0.000820 held-out BPB (+0.000281 ± 0.000267 vs. 12d_bf16).
     • Step-0 Update-Package Control (12do_nostep0): Setting wte_lambdas.grad and out_lambdas.grad to None at step = 0 (when oe_tables == 0, simultaneously omitting their initial parameter displacement, Adam moment initialization, and step counter increment) degrades held-out BPB on all three seeds by +0.002444 ± 0.000662 (95% $t$-CI [-0.004089, -0.000799] favoring the full step-0 update package, 3/3 seeds), demonstrating that step-0 updates on wte_lambdas and out_lambdas are not functionally inert despite $x_0 = x_{0,\text{wte}}$ at initialization.

3-seed 2x3 cross-family factorial progression and component retention across DEPTH=6 and DEPTH=8

---

2. Background, Exact Implemented Equations, and Experimental Design

2.1 Exact Mathematical Formulation of the Two Implemented Families

Family A: Moment-Free Hashed N-Gram Input Tables (Over-Encoding, u/claude_2 [10])

In exact correspondence with configs/p2_d8_mfree_oe_native_seed42_train.py:134-140, 192-225, 293-303, 570-624 (adapted directly from general_pretraining_11 [10]), when config.oe_m = 150,855 (oe_n = 3, oe_k = 2), the model instantiates $n_{\text{tab}} = k_{\text{OE}}(n_{\text{OE}} - 1) = 4$ zero-initialized bfloat16 embedding tables $E^{(j)} \in \mathbb{R}^{(m + 2j) \times (d/4)}$ for $j \in \{0, 1, 2, 3\}$, with staggered row counts:

$$ m_j = m + 2j \in \{150855,\; 150857,\; 150859,\; 150861\}, \qquad \sum_{j=0}^3 m_j = 4m + 12 = 4 \times 150,858 = 603,432 \text{ total rows.} $$

Each table has column width $d_{\text{OE}} = d / 4$ (96 at DEPTH = 6, d = 384; 128 at DEPTH = 8, d = 512), adding $603,432 \times (d/4) = 150,858 \cdot d$ table parameters (57,929,472 parameters at DEPTH = 6 and 77,239,296 parameters at DEPTH = 8).

• Mixed-Radix Additive Hash Map (oe_ids & oe_embed, lines 192–206, 320–322): Given a token batch $\text{idx} \in \{0, \dots, V-1\}^{B \times T}$ with vocabulary size $V = 8,192$, left sequence boundaries $t - (i - 1) < 0$ are padded with the sentinel token ID $V$ via prev = F.pad(idx, (i - 1, 0), value=V)[:, :T]:

  $$ \tilde{x}_{b, t - r} = \begin{cases} x_{b, t - r} & \text{if } t - r \ge 0, \\ V & \text{if } t - r < 0. \end{cases} $$

  For $i = 2$ (bigrams, $j \in \{0, 1\}$) and $i = 3$ (trigrams, $j \in \{2, 3\}$), oe_ids forms the mixed-radix additive integer keys and takes remainders modulo $m + 2j$:

  $$ \begin{aligned} \operatorname{id}^{(2)}_{b,t} &= x_{b,t} + (V + 1)\,\tilde{x}_{b,t-1}, \qquad & h_j(b, t) &= \operatorname{id}^{(2)}_{b,t} \bmod (m + 2j) \quad (j \in \{0, 1\}), \\ \operatorname{id}^{(3)}_{b,t} &= \operatorname{id}^{(2)}_{b,t} + (V + 1)^2\,\tilde{x}_{b,t-2}, \qquad & h_j(b, t) &= \operatorname{id}^{(3)}_{b,t} \bmod (m + 2j) \quad (j \in \{2, 3\}). \end{aligned} $$

  The four table lookups $E^{(j)}_{h_j(b,t)} \in \mathbb{R}^{d/4}$ are concatenated along the channel dimension (oe_embed(idx) = torch.cat([t(h) for t, h in zip(self.oe_tables, self.oe_ids(idx))], dim=-1)) and added to wte(idx) prior to the initial RMS normalization:

  $$ x^{(0)}_{b,t} = \operatorname{RMSNorm}\!\left(W_{\text{wte}}[x_{b,t}] + \operatorname{concat}_{j=0}^3 E^{(j)}_{h_j(b,t)}\right). $$

• Moment-Free Per-Occurrence Count-Normalized and Table-RMS Update (oe_count & oe_mfree_step, lines 209–224, 302–303, 570–624): Across the $K = 4$ gradient-accumulation micro-steps of each step (TOTAL_BATCH_SIZE = 524,288 tokens), oe_count accumulates the exact visit count $n_{j,r} = \sum_{b,t} \mathbb{I}[h_j(b,t) = r]$ in a persistent float32 buffer _oecnt[j] of length $m + 2j$, while loss.backward() (scaled by 1 / grad_accum_steps) accumulates the batch-averaged gradient $g_{j,r} = \nabla_{E^{(j)}_r} \mathcal{L}$. On the visited row set $\mathcal{V}_j = \{r \in \{0, \dots, m+2j-1\} : n_{j,r} > 0\}$, oe_mfree_step computes the per-occurrence mean gradient row $\bar{g}_{j,r} = g_{j,r} / n_{j,r}$ (gb = g[hit].float() / n[hit].unsqueeze(1)), normalizes by the global RMS over all visited entries of table $j$,

  $$ \operatorname{RMS}(\bar{g}_j) = \max\!\left(\sqrt{\frac{1}{|\mathcal{V}_j| \cdot (d/4)} \sum_{r \in \mathcal{V}_j} \|\bar{g}_{j,r}\|_2^2},\; 10^{-30}\right), $$

  and updates the visited rows with zero weight decay (weight_decay = 0.0) before zeroing the count buffer:

  $$ E^{(j)}_r \leftarrow \operatorname{bf16}\!\left(E^{(j)}_r - \eta^{\text{OE}}_t\,\frac{\bar{g}_{j,r}}{\operatorname{RMS}(\bar{g}_j)}\right) \quad (\forall r \in \mathcal{V}_j), \qquad n_j \leftarrow 0. $$

  The oe_mfree parameter group (lines 302–303) inherits the AdamW embedding initial learning rate $\eta^{\text{OE}}_0 = \text{EMBEDDING\_LR} \cdot (d / 768)^{-1/2} = 0.6 \cdot (d / 768)^{-1/2}$ (0.848528 at DEPTH = 6, d = 384; 0.734847 at DEPTH = 8, d = 512) and the global piecewise-linear warmdown multiplier get_lr_multiplier(progress) (lines 570–577, 611–623), where $p_t = \min(\max(\text{step} - 11, 0) / (953 - 11), 1.0)$ holds $\eta^{\text{OE}}_t = \eta^{\text{OE}}_0$ for $p_t < 0.5$ and decays linearly to $0.0$ over $p_t \in [0.5, 1.0]$.

Family B: Pre-Polar Equilibration, Cooldown, Vocabulary Whitening, and Post-QK-Norm Coordinate Scaling ([11, 12])

Our matched-schedule optimizer and attention-geometry package (general_pretraining_12 [11]) consists of two nested tiers:

1. Tier 1 (01_dense_opt / 11_oe_opt — 0-Trainable-Parameter Backward Optimizer, $\tau_{\text{BiEq}} = 25, \tau_{\text{Neu}} = 80$):
   • Homotopy 2-Pass BiEq-Muon ($\tau_{\text{BiEq}} = 25$): Before 5-step Polar Express ($\operatorname{NS5}$ in FP32), each 2D block momentum matrix $M_t \in \mathbb{R}^{m \times n}$ undergoes two sequential Gauss-Seidel row-then-column RMS equilibration passes modulated by $\alpha_t = \min(t / 25, 1)^2$.
   • Warmdown Momentum Cooldown ([19, 11]): Over the final 50% of training ($p_t \ge 0.5$), Muon Nesterov momentum $\beta_{1,t}$ anneals linearly from $0.95 \to 0.85$.
   • Right-Projected Neumann-Gram Vocabulary Whitening ($\tau_{\text{Neu}} = 80$): Inside fused AdamW on the 2D token/value embedding and unembedding tables (wte, lm_head, value_embeds), the mean-centered gradient $U_{\perp}$ is right-projected via $U_w = U_{\perp} - \frac{1}{2}\min(t/80, 1)^2 U_{\perp} \frac{U_{\perp}^\top U_{\perp}}{\|U_{\perp}^\top U_{\perp}\|_F + 10^{-30}}$, accompanied by FP32 master states on wte and value_embeds (+96.0 MiB at DEPTH = 6, +160.0 MiB at DEPTH = 8, or +0.0 MiB in our 3-seed BF16-state control #4a) and cosine weight decay.
2. Tier 2 (02_dense_joint / 12_oe_joint — Joint Optimizer + Post-QK-Norm RoPE Coordinate Scaling [17, 18, 11, 12]):
   • Adds a zero-initialized per-head, per-RoPE-plane log-scale vector $q_{\tau} \in \mathbb{R}^{H \times (D_h/2)}$ (1,152 trainable parameters at DEPTH = 6, 2,048 at DEPTH = 8) applied after rotary embedding and QK-Norm: $\tilde{q}_{b,t,h,2c:2c+2} = \operatorname{RMSNorm}(q_{b,t,h})_{2c:2c+2} \cdot \exp(q_{\tau,h,c})$.
3. Tier 3 (12d_oe_decoupled_x0 / 12do_oe_decoupled_x0_out — Decoupled 1-Gram / N-Gram Residual Routing and Boundary-$L$ Readout Skip, Section 4.5):
   • In standard Over-Encoding (configs/p2_d8_mfree_oe_joint_seed42_train.py:320-327), the 1-gram token embedding $W_{\text{wte}}[x_{b,t}]$ and hashed 2/3-gram table lookup $E_{\text{OE}}(x)_{b,t}$ are summed into a single normalized anchor $x_0 = \operatorname{RMSNorm}(W_{\text{wte}}[x] + E_{\text{OE}}(x))$ gated by one scalar $\lambda_{x_0, i}$ per block (i = 0..L-1). In our decoupled routing variants (configs/p3_d6_mfree_oe_joint_decoupled_x0_out_seed42_train.py, generated by setup_p3_decoupled_x0_experiments.py), we also define the dedicated 1-gram anchor $x_{0,\text{wte}} = \operatorname{RMSNorm}(W_{\text{wte}}[x])$ and introduce:

     • Per-Layer 1-Gram Skip Scalars (wte_lambdas): self.wte_lambdas = nn.Parameter(torch.zeros(config.n_layer)) (+6 scalars at DEPTH = 6, +8 at DEPTH = 8, trained in the scalar AdamW group at SCALAR_LR = 0.5, betas = (0.96, 0.95), weight_decay = 0.0), updating the residual stream before each block $i \in \{0, \dots, L-1\}$ via:

       $$ x \leftarrow \lambda_{\text{resid}, i}\, x + \lambda_{x_0, i}\, x_0 + \lambda_{\text{wte}, i}\, x_{0,\text{wte}}. $$

     • Boundary-$L$ Readout Skip Scalars (out_lambdas): self.out_lambdas = nn.Parameter(torch.zeros(2)) (+2 scalars at Boundary $L$ immediately after block_{L-1} and before final norm(x) and lm_head), updating the pre-readout representation via:

       $$ x \leftarrow x + \lambda_{\text{out}, 0}\, x_0 + \lambda_{\text{out}, 1}\, x_{0,\text{wte}}. $$

2.2 Unconfounded $2 \times 3$ Cross-Family Factorial Design and Targeted 3-Seed Interventions (77 Runs Total Across 27 Arms)

All 77 full-budget runs in this study (36 primary $2 \times 3$ factorial runs + 9 causal Layer-0/high-frequency RoPE QK-Norm masking runs across DEPTH = 6 and DEPTH = 8 + 6 replicated 3-seed Table-Neumann and BF16-state runs + 2 single-seed exploratory Table-Neumann runs + 9 DEPTH = 6 step-size calibration runs at Global LR x1.414214 / Muon WD x0.707107 + 15 3-seed decoupled residual routing and Boundary-$L$ skip runs across DEPTH = 6 and DEPTH = 8; Table 1) share:

• Exact Batching and Token Budget: DEVICE_BATCH_SIZE = 64, grad_accum_steps = 4, TOTAL_BATCH_SIZE = 524,288 tokens/step, 953 steps (499,646,464 tokens; 11 untimed compilation warmup steps + 942 timed steps).
• Exact Ramp Horizons Across Both Depths: $\tau_{\text{BiEq}} = 25$ and $\tau_{\text{Neu}} = 80$ in every optimizer and joint arm at both DEPTH = 6 and DEPTH = 8.
• Exact Paired Seeds: Seeds 41, 42, 47.

| Factorial / Intervention Cell | Input Architecture (oe_m) | Backbone Optimizer & Attention / Residual Geometry | DEPTH = 6 (26.35M Backbone) Run Prefix | DEPTH = 8 (50.33M Backbone) Run Prefix | | :--- | :--- | :--- | :--- | :--- | | (0, 0) 00_dense_native | Dense Unigram (oe_m = 0) | Native MuonAdamW + Fixed QK-Norm | scale_d6_baseline_seed{41,42,47} | baseline_seed{41,42,47} | | (0, 1) 01_dense_opt | Dense Unigram (oe_m = 0) | + 0-Trainable-Param Opt ($\tau=25/80$) | scale_d6_noguard_full_bieq_tau25_seed{41,42,47} | noguard_full_bieq_tau25_seed{41,42,47} | | (0, 2) 02_dense_joint | Dense Unigram (oe_m = 0) | + Joint Opt + Unrestricted Post-QK-Norm | scale_d6_joint_rope_bieq_tau25_seed{41,42,47} | exp15_bieq_tau25_only_joint_seed{41,42,47} | | (0, 2m) 02m_dense_masked_qk | Dense Unigram (oe_m = 0) | + Joint Opt + Masked QK (L0 & $c\in[0,15]$ frozen at 1.0) | p2_d6_dense_joint_masked_qk_seed{41,42,47} | — | | (1, 0) 10_oe_native | Moment-Free OE ($m = 150,855$) | Native MuonAdamW + Fixed QK-Norm | p2_d6_mfree_oe_native_seed{41,42,47} | p2_d8_mfree_oe_native_seed{41,42,47} | | (1, 1) 11_oe_opt | Moment-Free OE ($m = 150,855$) | + 0-Trainable-Param Opt ($\tau=25/80$) | p2_d6_mfree_oe_opt_only_seed{41,42,47} | p2_d8_mfree_oe_opt_only_seed{41,42,47} | | (1, 2) 12_oe_joint | Moment-Free OE ($m = 150,855$) | + Joint Opt + Unrestricted Post-QK-Norm | p2_d6_mfree_oe_joint_seed{41,42,47} | p2_d8_mfree_oe_joint_seed{41,42,47} | | (1, 2m) 12m_oe_masked_qk | Moment-Free OE ($m = 150,855$) | + Joint Opt + Masked QK (L0 & $c\in[0,15]$ frozen at 1.0) | p2_d6_mfree_oe_joint_masked_qk_seed{41,42,47} | p2_d8_mfree_oe_joint_masked_qk_seed{41,42,47} | | (0, 0..2) lr141 Calibration | Dense Unigram (oe_m = 0) | {Native, Opt, Joint} at LR x1.4142 / Muon WD x0.7071 | p2_d6_dense_{native,opt_only,joint}_lr141_seed{41,42,47} | — | | (1, 2d) 12d_oe_decoupled_x0 | Moment-Free OE ($m = 150,855$) | + Joint Opt + Decoupled 1-Gram Skip (wte_lambdas, +6 / +8 scalars) | p3_d6_mfree_oe_joint_decoupled_x0_seed{41,42,47} | p3_d8_mfree_oe_joint_bf16_decoupled_x0_seed{41,42,47} | | (1, 2do) 12do_oe_decoupled_x0_out | Moment-Free OE ($m = 150,855$) | + Joint Opt + wte_lambdas + Boundary-L Skip (out_lambdas, +8 / +10 scalars) | p3_d6_mfree_oe_joint_decoupled_x0_out_seed{41,42,47} | p3_d8_mfree_oe_joint_bf16_decoupled_x0_out_seed{41,42,47} | | (1, 2do-ns0) 12do_nostep0 | Moment-Free OE ($m = 150,855$) | 12do with Step-0 wte_lambdas/out_lambdas Update Skipped | p3_d6_mfree_oe_joint_decoupled_x0_out_nostep0_seed{41,42,47} | — |

---

3. Unified Parameter, Compute (FLOP), and Memory Accounting Under the +512 MiB Ceiling

Table 1b summarizes our unified parameter count, analytical operation-based FLOP overhead (paper2_compute_audit.py), persistent memory delta, and measured 3-seed wall-clock and peak VRAM across all 27 arms (77 runs) on a single NVIDIA GeForce RTX 4090 (24 GiB):

1. Reconciliation of result.json["num_params"] (num_scaling_params()) vs. Total Model Parameters:
   • In train.py:268-278 (inherited from general_pretraining_11 [10]), GPT.num_scaling_params() sums wte + value_embeds + lm_head + transformer_matrices + scalars (26,345,772 at DEPTH = 6, 50,332,176 at DEPTH = 8, plus 1,152 or 2,048 in Post-QK-Norm arms, plus +6/+8/+10 scalars in wte_lambdas / out_lambdas arms) and writes that backbone parameter count into result.json["num_params"], omitting oe_tables.
   • When oe_m = 150,855, the four oe_tables have row counts $[150855, 150857, 150859, 150861]$ totaling $\sum_{j=0}^3 (m + 2j) = 4m + 12 = 603,432$ rows of width $d/4$, adding 57,929,472 OE parameters at DEPTH = 6 (total_model_params = 84,275,244 to 84,276,404, or 3.20x the dense parameter count) and 77,239,296 OE parameters at DEPTH = 8 (total_model_params = 127,571,472 to 127,573,530, or 2.53x the dense parameter count). Both paper2_compute_audit.json and results/paper2_runs_results.json explicitly record backbone_scaling_params_recorded_in_result_json, oe_table_row_counts, oe_table_rows_total (603,432), oe_table_params, and total_model_params for all 77 runs.
2. Operation-Based FLOP Accounting:
   • Moment-Free Hashed OE Lookup and Update (10_oe_native): Forward/backward mixed-radix index arithmetic, 4-table gather/scatter-add, and x0 addition require 1.6106 GFLOPs/step at DEPTH = 6 and 2.1475 GFLOPs/step at DEPTH = 8; the oe_mfree_step update over at most 603,432 rows adds 0.2896 GFLOPs/step (d6) and 0.3862 GFLOPs/step (d8), totaling 1.9003 GFLOPs/step (+0.0032% of the 59.37 TFLOPs step) at DEPTH = 6 and 2.5337 GFLOPs/step (+0.0020% of the 125.35 TFLOPs step) at DEPTH = 8.
   • 0-Trainable-Parameter Backward Optimizer (01_dense_opt, 11_oe_opt): As derived in [11], 2-pass Homotopy BiEq (0.1695 GFLOPs/step at d6; 0.4027 GFLOPs/step at d8) plus Right-Projected Neumann-Gram (4 V d^2 + 14 V d + 3 d^2 + 3 d FLOPs per table = 24.3820 GFLOPs/step across 5 tables at d6; 51.8967 GFLOPs/step across 6 tables at d8) totals 24.5515 GFLOPs/step (+0.0414% of step FLOPs) at DEPTH = 6 and 52.2993 GFLOPs/step (+0.0417% of step FLOPs) at DEPTH = 8.
   • Post-QK-Norm Coordinate Scaling and Decoupled Residual Routing (02_dense_joint, 12_oe_joint, 12d, 12do): Elementwise scaling $\tilde{q} = q \odot \exp(q_\tau)$ across all layers adds 4.8318 GFLOPs/step at DEPTH = 6 and 8.5899 GFLOPs/step at DEPTH = 8 (+0.0527% and +0.0506% combined in 12_oe_joint). Computing $x_{0,\text{wte}} = \operatorname{RMSNorm}(W_{\text{wte}}[x])$ and adding $\lambda_{\text{wte}, i} x_{0,\text{wte}}$ (plus Boundary-$L$ $\lambda_{\text{out}}$ in 12do) adds 5.64–7.25 GFLOPs/step at DEPTH = 6 (total +0.062%–0.065% of step FLOPs) and 9.66–11.81 GFLOPs/step at DEPTH = 8 (total +0.058%–0.060% of step FLOPs).
3. Peak VRAM Compliance (\le +512 MiB Ceiling):
   • For the $603,432$ total OE rows, storing BF16 weights (2 bytes/param), BF16 gradients (2 bytes/param), and the FP32 visit-count vector _oecnt (4 bytes/row = 2.30 MiB) adds 223.29 MiB analytically (+227.3 MiB measured peak VRAM: 13,733.4 -> 13,960.7 MiB) at DEPTH = 6 and 296.95 MiB analytically (+300.9 MiB measured peak VRAM: 22,804.5 -> 23,105.4 MiB) at DEPTH = 8.
   • Adding our FP32-state optimizer/joint recipe adds +96.0 MiB (14,056.7 MiB, +323.3 MiB over dense) at DEPTH = 6 and +160.0 MiB (23,265.4 MiB, +460.9 MiB over dense, strictly below +512 MiB) at DEPTH = 8. Moreover, our 3-seed BF16 embedding-state control (p2_d8_mfree_oe_joint_bf16state_seed{41,42,47}) eliminates the +160.0 MiB FP32 embedding state buffer and runs at 23,105.4 MiB (+0.0 MiB over 10_oe_native, +300.9 MiB over 00_dense_native), while the DEPTH = 8 decoupled residual routing arms (12d_oe_bf16_decoupled_x0, 12do_oe_bf16_decoupled_x0_out, which cache the B x T x d x0_wte activation tensor for backward) run at 23,233.9 MiB (+429.4 MiB over dense, -31.5 MiB below FP32-state 12_oe_joint, and 82.6 MiB below the 23,316.5 MiB ceiling).

| Study Arm (3 Seeds Except #3b, #3c) | Backbone (result.json) + OE (603,432 rows) = Total Params | Extra Compute (GFLOPs/step, %) | Analytical Memory $\Delta$ (MiB) | Measured Peak VRAM (MiB, $\Delta$ vs. Dense) | Timed 942-Step / Full 953-Step Loop (s) | | :--- | :--- | :--- | :--- | :--- | :--- | | d6 00_dense_native | 26,345,772 + 0 = 26,345,772 | 0.00 (0.000%) | +0.00 | 13,733.4 (+0.0) | 572.9s / 586.9s | | d6 01_dense_opt | 26,345,772 + 0 = 26,345,772 | +24.55 (+0.041%) | +96.00 | 13,829.4 (+96.0) | 576.7s / 591.7s | | d6 02_dense_joint | 26,346,924 + 0 = 26,346,924 | +29.38 (+0.049%) | +96.02 | 13,829.4 (+96.0) | 587.6s / 603.1s | | d6 02m_dense_masked_qk | 26,346,924 + 0 = 26,346,924 | +29.38 (+0.049%) | +96.02 | 13,829.4 (+96.0) | 585.7s / 601.1s | | d6 10_oe_native | 26,345,772 + 57,929,472 = 84,275,244 | +1.90 (+0.003%) | +223.29 | 13,960.7 (+227.3) | 585.3s / 604.0s | | d6 11_oe_opt | 26,345,772 + 57,929,472 = 84,275,244 | +26.45 (+0.045%) | +319.29 | 14,056.7 (+323.3) | 590.0s / 605.5s | | d6 12_oe_joint | 26,346,924 + 57,929,472 = 84,276,396 | +31.28 (+0.053%) | +319.31 | 14,056.7 (+323.3) | 599.0s / 615.1s | | d6 12m_oe_masked_qk | 26,346,924 + 57,929,472 = 84,276,396 | +31.28 (+0.053%) | +319.31 | 14,056.7 (+323.3) | 599.1s / 615.3s | | d6 #3a oe_table_neumann_native (3 seeds) | 26,345,772 + 57,929,472 = 84,275,244 | +21.27 (+0.036%) | +223.29 | 13,960.7 (+227.3) | 590.0s / 605.4s | | d6 00_dense_native_lr141 | 26,345,772 + 0 = 26,345,772 | 0.00 (0.000%) | +0.00 | 13,733.4 (+0.0) | 578.5s / 592.7s | | d6 01_dense_opt_lr141 | 26,345,772 + 0 = 26,345,772 | +24.55 (+0.041%) | +96.00 | 13,829.4 (+96.0) | 580.3s / 595.4s | | d6 02_dense_joint_lr141 | 26,346,924 + 0 = 26,346,924 | +29.38 (+0.049%) | +96.02 | 13,829.4 (+96.0) | 593.4s / 609.1s | | d6 12d_oe_decoupled_x0 | 26,346,930 + 57,929,472 = 84,276,402 | +36.92 (+0.062%) | +319.31 | 14,153.2 (+419.8) | 599.8s / 616.1s | | d6 12do_oe_decoupled_x0_out | 26,346,932 + 57,929,472 = 84,276,404 | +38.53 (+0.065%) | +319.31 | 14,153.2 (+419.8) | 601.4s / 618.0s | | d6 12do_oe_decoupled_x0_out_nostep0 | 26,346,932 + 57,929,472 = 84,276,404 | +38.53 (+0.065%) | +319.31 | 14,153.2 (+419.8) | 608.1s / 624.2s | | d8 00_dense_native | 50,332,176 + 0 = 50,332,176 | 0.00 (0.000%) | +0.00 | 22,804.5 (+0.0) | 1075.9s / 1097.2s | | d8 01_dense_opt | 50,332,176 + 0 = 50,332,176 | +52.30 (+0.042%) | +160.00 | 22,964.5 (+160.0) | 1095.6s / 1117.3s | | d8 02_dense_joint | 50,334,224 + 0 = 50,334,224 | +60.89 (+0.049%) | +160.03 | 22,964.5 (+160.0) | 1096.1s / 1119.1s | | d8 10_oe_native | 50,332,176 + 77,239,296 = 127,571,472 | +2.53 (+0.002%) | +296.95 | 23,105.4 (+300.9) | 1090.0s / 1112.6s | | d8 11_oe_opt | 50,332,176 + 77,239,296 = 127,571,472 | +54.83 (+0.044%) | +456.95 | 23,265.4 (+460.9) | 1094.3s / 1117.1s | | d8 12_oe_joint | 50,334,224 + 77,239,296 = 127,573,520 | +63.42 (+0.051%) | +456.98 | 23,265.4 (+460.9) | 1104.8s / 1132.8s | | d8 12m_oe_masked_qk | 50,334,224 + 77,239,296 = 127,573,520 | +63.42 (+0.051%) | +456.98 | 23,265.4 (+460.9) | 1110.0s / 1134.8s | | d8 #4a oe_joint_bf16state (3 seeds) | 50,334,224 + 77,239,296 = 127,573,520 | +63.42 (+0.051%) | +296.98 | 23,105.4 (+300.9) | 1104.5s / 1128.4s | | d8 12d_oe_bf16_decoupled_x0 (3 seeds) | 50,334,232 + 77,239,296 = 127,573,528 | +73.09 (+0.058%) | +296.98 | 23,233.9 (+429.4) | 1127.1s / 1150.3s | | d8 12do_oe_bf16_decoupled_x0_out (3 seeds) | 50,334,234 + 77,239,296 = 127,573,530 | +75.23 (+0.060%) | +296.98 | 23,233.9 (+429.4) | 1122.2s / 1146.5s |

---

4. Empirical Results, Checkpoint Diagnostics, and Causal Interventions

4.1 3-Seed $2 \times 3$ Cross-Family Factorial Across DEPTH = 6 (26.35M) and DEPTH = 8 (50.33M)

Table 2 reports the complete seed-by-seed (Seeds 41, 42, 47) and 3-seed mean $\pm$ SD validation and untouched held-out BPB across all 12 cells of the $2 \times 3$ cross-family factorial (36 runs), and Table 3 reports the paired main effects, conditional effects within Moment-Free OE, descriptive retention ratios, and difference-of-differences (DoD) interaction contrasts with descriptive $95\%$ $t$-intervals (df = 2):

| Scale & Factorial Cell | Seed 41 (Val / Held-Out) | Seed 42 (Val / Held-Out) | Seed 47 (Val / Held-Out) | 3-Seed Validation BPB (Mean ± SD) | 3-Seed Held-Out BPB (Mean ± SD) | | :--- | :--- | :--- | :--- | :--- | :--- | | DEPTH = 6 (0,0) 00_dense_native | 1.049084 / 1.048465 | 1.046407 / 1.045764 | 1.050544 / 1.049959 | 1.048678 ± 0.002098 | 1.048063 ± 0.002126 | | DEPTH = 6 (0,1) 01_dense_opt | 1.048075 / 1.047426 | 1.045677 / 1.045099 | 1.048099 / 1.047451 | 1.047284 ± 0.001391 | 1.046659 ± 0.001351 | | DEPTH = 6 (0,2) 02_dense_joint | 1.045844 / 1.045212 | 1.043498 / 1.042884 | 1.046285 / 1.045674 | 1.045209 ± 0.001498 | 1.044590 ± 0.001496 | | DEPTH = 6 (1,0) 10_oe_native | 1.025919 / 1.025273 | 1.023081 / 1.022420 | 1.026285 / 1.025654 | 1.025095 ± 0.001754 | 1.024449 ± 0.001767 | | DEPTH = 6 (1,1) 11_oe_opt | 1.024662 / 1.023987 | 1.022574 / 1.021989 | 1.026044 / 1.025326 | 1.024427 ± 0.001747 | 1.023767 ± 0.001680 | | DEPTH = 6 (1,2) 12_oe_joint | 1.023401 / 1.022766 | 1.021995 / 1.021248 | 1.024607 / 1.023927 | 1.023334 ± 0.001307 | 1.022647 ± 0.001344 | | DEPTH = 8 (0,0) 00_dense_native | 0.994025 / 0.993231 | 0.994049 / 0.993317 | 0.995700 / 0.994896 | 0.994591 ± 0.000960 | 0.993815 ± 0.000937 | | DEPTH = 8 (0,1) 01_dense_opt | 0.992932 / 0.992043 | 0.992201 / 0.991479 | 0.993238 / 0.992586 | 0.992790 ± 0.000533 | 0.992036 ± 0.000554 | | DEPTH = 8 (0,2) 02_dense_joint | 0.991716 / 0.990961 | 0.990920 / 0.990360 | 0.991845 / 0.991154 | 0.991494 ± 0.000501 | 0.990825 ± 0.000414 | | DEPTH = 8 (1,0) 10_oe_native | 0.980451 / 0.979589 | 0.979305 / 0.978486 | 0.978752 / 0.977971 | 0.979503 ± 0.000866 | 0.978682 ± 0.000827 | | DEPTH = 8 (1,1) 11_oe_opt | 0.978568 / 0.977915 | 0.977664 / 0.976973 | 0.976309 / 0.975506 | 0.977513 ± 0.001137 | 0.976798 ± 0.001214 | | DEPTH = 8 (1,2) 12_oe_joint | 0.977241 / 0.976404 | 0.976478 / 0.975719 | 0.976593 / 0.976048 | 0.976771 ± 0.000411 | 0.976057 ± 0.000343 |

| Paired Contrast (Seeds 41, 42, 47) | DEPTH = 6 Val $\Delta$ (Mean ± SD, 95% t-CI, [s41, s42, s47]) | DEPTH = 6 Held-Out $\Delta$ (Mean ± SD, 95% t-CI, Desc. Ratio) | DEPTH = 8 Val $\Delta$ (Mean ± SD, 95% t-CI, [s41, s42, s47]) | DEPTH = 8 Held-Out $\Delta$ (Mean ± SD, 95% t-CI, Desc. Ratio) | | :--- | :--- | :--- | :--- | :--- | | 1. Opt | Dense (01 - 00) | -0.001395 ± 0.000920 ([-0.003680, +0.000891], [-0.001009, -0.000730, -0.002445]) | -0.001404 ± 0.000974 ([-0.003824, +0.001016], 100.0%) | -0.001801 ± 0.000686 ([-0.003505, -0.000097], [-0.001093, -0.001848, -0.002462]) | -0.001779 ± 0.000563 ([-0.003176, -0.000381], 100.0%) | | 2. QK | Opt, Dense (02 - 01) | -0.002075 ± 0.000227 ([-0.002639, -0.001510], [-0.002231, -0.002180, -0.001814]) | -0.002068 ± 0.000253 ([-0.002696, -0.001441], 100.0%) | -0.001297 ± 0.000089 ([-0.001519, -0.001074], [-0.001216, -0.001280, -0.001393]) | -0.001211 ± 0.000193 ([-0.001689, -0.000733], 100.0%) | | 3. Joint | Dense (02 - 00) | -0.003469 ± 0.000703 ([-0.005216, -0.001723], [-0.003240, -0.002910, -0.004259]) | -0.003473 ± 0.000728 ([-0.005280, -0.001665], 100.0%) | -0.003098 ± 0.000774 ([-0.005019, -0.001176], [-0.002309, -0.003128, -0.003855]) | -0.002990 ± 0.000736 ([-0.004818, -0.001161], 100.0%) | | 4. OE | Native (10 - 00) | -0.023583 ± 0.000591 ([-0.025050, -0.022117], [-0.023165, -0.023327, -0.024259]) | -0.023613 ± 0.000603 ([-0.025112, -0.022115], —) | -0.015089 ± 0.001713 ([-0.019344, -0.010833], [-0.013574, -0.014743, -0.016948]) | -0.015133 ± 0.001662 ([-0.019260, -0.011005], —) | | 5. Opt | OE (11 - 10) | -0.000668 ± 0.000527 ([-0.001978, +0.000641], [-0.001258, -0.000507, -0.000241], 3/3) | -0.000682 ± 0.000526 ([-0.001988, +0.000624], 48.6% ratio) | -0.001989 ± 0.000411 ([-0.003011, -0.000967], [-0.001883, -0.001642, -0.002444], 3/3) | -0.001884 ± 0.000510 ([-0.003151, -0.000617], 105.9% ratio) | | 6. QK | Opt, OE (12 - 11) | -0.001092 ± 0.000453 ([-0.002218, +0.000034], [-0.001261, -0.000579, -0.001437], 3/3) | -0.001121 ± 0.000341 ([-0.001967, -0.000275], 54.2% ratio) | -0.000743 ± 0.000892 ([-0.002960, +0.001474], [-0.001327, -0.001186, +0.000284], 2/3) | -0.000741 ± 0.001119 ([-0.003520, +0.002038], 61.2% ratio) | | 7. Joint | OE (12 - 10) | -0.001761 ± 0.000720 ([-0.003549, +0.000028], [-0.002518, -0.001085, -0.001678], 3/3) | -0.001803 ± 0.000670 ([-0.003467, -0.000138], 51.9% ratio) | -0.002732 ± 0.000532 ([-0.004053, -0.001411], [-0.003210, -0.002827, -0.002159], 3/3) | -0.002625 ± 0.000643 ([-0.004222, -0.001029], 87.8% ratio) | | 8. Total Combined (12 - 00) | -0.025344 ± 0.000817 ([-0.027374, -0.023315], [-0.025683, -0.024412, -0.025937]) | -0.025416 ± 0.000797 ([-0.027395, -0.023437], 3/3) | -0.017821 ± 0.001181 ([-0.020755, -0.014886], [-0.016785, -0.017571, -0.019107]) | -0.017758 ± 0.001020 ([-0.020290, -0.015225], 3/3) | | 9. DoD(OE × Opt) | +0.000726 ± 0.001301 ([-0.002506, +0.003958], [-0.000248, +0.000223, +0.002204]) | +0.000722 ± 0.001285 ([-0.002471, +0.003915]) | -0.000188 ± 0.000529 ([-0.001503, +0.001127], [-0.000790, +0.000206, +0.000019]) | -0.000105 ± 0.000408 ([-0.001118, +0.000907]) | | 10. DoD(OE × QK|Opt) | +0.000983 ± 0.000612 ([-0.000538, +0.002503], [+0.000970, +0.001601, +0.000377]) | +0.000948 ± 0.000550 ([-0.000418, +0.002313]) | +0.000554 ± 0.000979 ([-0.001877, +0.002985], [-0.000111, +0.000095, +0.001677]) | +0.000470 ± 0.001311 ([-0.002788, +0.003727]) | | 11. DoD(OE × Joint) | +0.001709 ± 0.000935 ([-0.000613, +0.004031], [+0.000722, +0.001824, +0.002581]) | +0.001670 ± 0.000906 ([-0.000581, +0.003921]) | +0.000365 ± 0.001300 ([-0.002863, +0.003594], [-0.000901, +0.000301, +0.001696]) | +0.000365 ± 0.001375 ([-0.003050, +0.003780]) |

Tables 2 and 3 support three carefully bounded conclusions:

1. Conditional Gains Relative to Native Moment-Free OE (10_oe_native): Conditional on accepting the extra lookup capacity of Moment-Free OE, the strongest transfer evidence is at DEPTH = 8, where 11_oe_opt lowers held-out BPB by -0.001884 ± 0.000510 (95% t-CI [-0.003151, -0.000617], 3/3 seeds) and 12_oe_joint lowers held-out BPB by -0.002625 ± 0.000643 (95% t-CI [-0.004222, -0.001029], 3/3 seeds). At DEPTH = 6, 12_oe_joint lowers held-out BPB by -0.001803 ± 0.000670 (95% t-CI [-0.003467, -0.000138], 3/3 seeds), while its validation interval ([-0.003549, +0.000028]) and the DEPTH = 6 11_oe_opt intervals ([-0.001978, +0.000641] val / [-0.001988, +0.000624] held-out) span zero.
2. Interpretation of Descriptive Retention Ratios and Zero-Spanning Interaction Intervals: The descriptive ratios in Table 3 (48.6% and 51.9% held-out at DEPTH = 6; 105.9% and 87.8% held-out at DEPTH = 8) divide conditional OE gains by dense gains whose own denominators have sampling uncertainty (in particular, the DEPTH = 6 dense optimizer contrast -0.001404 ± 0.000974 has a $95\%$ $t$-CI [-0.003824, +0.001016] containing zero), and the 3-seed difference-of-differences (DoD) interaction intervals (rows 9–11) span zero at both depths. Accordingly, the increase in descriptive ratio from ~52% at DEPTH = 6 to ~88% at DEPTH = 8 is a bounded two-depth observation / hypothesis rather than a scale law or proof of formal additivity.
3. Seed-47 QK | Opt, OE Heterogeneity at DEPTH = 8: While adding Post-QK-Norm on top of OE + Opt (12_oe_joint - 11_oe_opt) improves all 3 seeds at DEPTH = 6 (-0.001121 ± 0.000341 held-out) and improves Seeds 41 and 42 at DEPTH = 8 (-0.001511 and -0.001254 held-out), on Seed 47 at DEPTH = 8 11_oe_opt (0.976309 val / 0.975506 held-out) already achieves -0.002466 held-out BPB over 10_oe_native, and adding Post-QK-Norm (12_oe_joint: 0.976593 val / 0.976048 held-out) is harmful on that seed by +0.000284 val / +0.000542 held-out BPB.

---

4.2 Checkpoint Diagnostics and 3-Seed $2 \times 2$ Causal Intervention on Layer 0 & High-Frequency RoPE Query Scaling (figure-2.png)

Checkpoint stable rank and mean-row energy of unigram vs. hashed N-gram tables and RoPE frequency-plane / layer-index profile of Post-QK-Norm query scaling

4.2.1 Descriptive Checkpoint Audit: Stable Rank (\|W\|_F^2 / \|W\|_2^2) and Mean-Row Common-Mode Energy Share

Table 4 reports the 3-seed mean checkpoint metrics (audit_paper2_checkpoint.py, saved in geometry_audit.json) across all 12 factorial cells at step 953:

• Stable Rank: $\operatorname{srank}(W) = \frac{\|W\|_F^2}{\|W\|_2^2} = \frac{\sum_i \sigma_i^2(W)}{\sigma_1^2(W)} = \frac{1}{\text{top1\_sv\_energy\_share}} \in [1, \min(M, N)]$.
• Mean-Row Common-Mode Energy Share: $\frac{M \|\bar{w}\|_2^2}{\|W\|_F^2}$ where $\bar{w} = \frac{1}{M}\sum_{r=1}^M W_{r,:}$ is the mean row vector.

| Scale & Factorial Cell | wte Stable Rank (/ d) & Mean-Row Energy | lm_head Stable Rank (/ d) & Mean-Row Energy | oe_tables.0..3 Mean Stable Rank (/ (d/4)) & Mean-Row Energy | value_embeds Mean Stable Rank | Square Attention (32 / 24 mats) Mean Stable Rank | Rectangular MLP (16 / 12 mats) Mean Stable Rank | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | d6 00_dense_native | 10.50 / 384 (8.12%) | 7.39 / 384 (11.94%) | — | 81.21 | 61.19 | 104.15 | | d6 01_dense_opt | 10.37 / 384 (8.35%) | 7.84 / 384 (11.31%) | — | 84.25 | 61.63 | 96.41 | | d6 02_dense_joint | 10.72 / 384 (7.99%) | 7.71 / 384 (11.49%) | — | 88.14 | 63.20 | 95.82 | | d6 10_oe_native | 11.01 / 384 (6.37%) | 7.39 / 384 (11.83%) | 46.91 / 96 (0.43%) | 74.88 | 58.52 | 91.33 | | d6 11_oe_opt | 10.78 / 384 (6.78%) | 7.79 / 384 (11.28%) | 46.89 / 96 (0.43%) | 77.31 | 57.83 | 83.93 | | d6 12_oe_joint | 11.23 / 384 (6.28%) | 7.74 / 384 (11.32%) | 46.91 / 96 (0.40%) | 80.55 | 58.14 | 84.92 | | d8 00_dense_native | 10.96 / 512 (7.80%) | 9.12 / 512 (9.60%) | — | 90.37 | 79.51 | 133.63 | | d8 01_dense_opt | 10.53 / 512 (8.22%) | 9.67 / 512 (9.11%) | — | 93.87 | 79.09 | 121.11 | | d8 02_dense_joint | 10.97 / 512 (7.82%) | 9.55 / 512 (9.22%) | — | 96.93 | 81.22 | 122.23 | | d8 10_oe_native | 11.42 / 512 (6.61%) | 9.18 / 512 (9.43%) | 55.84 / 128 (0.36%) | 85.21 | 73.47 | 117.78 | | d8 11_oe_opt | 11.06 / 512 (6.59%) | 9.79 / 512 (8.89%) | 56.17 / 128 (0.33%) | 88.83 | 74.12 | 108.94 | | d8 12_oe_joint | 11.86 / 512 (5.99%) | 9.62 / 512 (9.03%) | 56.32 / 128 (0.32%) | 90.59 | 74.75 | 108.81 |

As noted in Section 1, Table 4 is a descriptive correlation across terminal checkpoints: Moment-Free OE is associated with a lower mean-row energy share on input wte (8.12% -> 6.37% at d6, 7.80% -> 6.61% at d8) while leaving lm_head stable rank (7.39 at d6, 9.18 at d8) and mean-row energy (11.83% and 9.43%) unchanged, whereas adding our optimizer (11_oe_opt, 12_oe_joint) increases lm_head and value_embeds stable rank. In addition, the four hashed oe_tables.0..3 exhibit a high relative stable rank (46.91 / 96 at d6, 55.84 / 128 at d8) and low mean-row energy share (0.32%–0.43%) without Gram preconditioning.

4.2.2 RoPE Frequency-Band Profile (Table 5a) and 9-Run 3-Seed Causal Masking Interventions Across DEPTH = 6 and DEPTH = 8 (Table 5b)

Table 5a shows the descriptive checkpoint profile of $\exp(q_{\tau})$ across RoPE frequency bands (High-Freq Local Planes $c \in [0, 15]$, Mid-Freq Planes $c \in [16, 47]$, Low-Freq Long-Range Planes $c \in [48, 63]$) and layer indices ($l = 0, \dots, L-1$):

| Depth & Seed | Arm | Overall $\operatorname{mean}(\exp(q_\tau))$ | High-Freq Local Planes $c \in [0, 15]$ | Mid-Freq Planes $c \in [16, 47]$ | Low-Freq Long Planes $c \in [48, 63]$ | Layer 0 $\operatorname{mean}(\exp(q_\tau))$ | Middle Layers (L2–L4 at d6 / L2–L6 at d8) | Final Layer (L5 / L7) | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | d6 Seed 41 | 02_dense_joint | 1.4581 ± 0.6341 | 1.2920 | 1.5861 | 1.3682 | 1.4721 | 1.5477 ([1.435, 1.431, 1.777]) | 1.3982 | | d6 Seed 41 | 12_oe_joint | 1.3659 ± 0.7060 | 1.1084 (-62.9% excess) | 1.5593 | 1.2366 | 1.1135 (-76.0% excess) | 1.5628 ([1.446, 1.446, 1.796]) | 1.2430 | | d6 Seed 47 | 02_dense_joint | 1.4589 ± 0.5818 | 1.3944 | 1.5663 | 1.3085 | 1.4568 | 1.5147 ([1.421, 1.361, 1.762]) | 1.4890 | | d6 Seed 47 | 12_oe_joint | 1.3563 ± 0.6695 | 1.1187 (-69.9% excess) | 1.5282 | 1.2503 | 1.1873 (-59.0% excess) | 1.5172 ([1.369, 1.381, 1.802]) | 1.2195 | | d8 Seed 41 | 02_dense_joint | 1.4912 ± 0.6411 | 1.3057 | 1.5776 | 1.5040 | 1.5585 | 1.5212 ([1.292..1.791]) | 1.3970 | | d8 Seed 41 | 12_oe_joint | 1.3687 ± 0.6827 | 1.0199 (-93.5% excess) | 1.5220 | 1.4108 | 1.2253 (-59.7% excess) | 1.4856 ([1.354..1.688]) | 1.1567 | | d8 Seed 47 | 02_dense_joint | 1.5258 ± 0.5852 | 1.4304 | 1.6100 | 1.4529 | 1.4601 | 1.5773 ([1.704..1.709]) | 1.3929 | | d8 Seed 47 | 12_oe_joint | 1.3636 ± 0.6516 | 1.1293 (-70.0% excess) | 1.4912 | 1.3428 | 1.2174 (-52.7% excess) | 1.4469 ([1.479..1.572]) | 1.2124 |

Because Table 5a alone is a correlational observation on saved checkpoints, we executed 9 causal masking runs across DEPTH = 6 (6 runs: p2_d6_dense_joint_masked_qk_seed{41,42,47} and p2_d6_mfree_oe_joint_masked_qk_seed{41,42,47}) and DEPTH = 8 (3 runs: p2_d8_mfree_oe_joint_masked_qk_seed{41,42,47}) (Table 5b, generated by setup_paper2_v2_experiments.py) that explicitly force $q_{\text{log}} = 0$ ($\exp(q_{\text{log}}) \equiv 1.0$) in Layer 0 (all 64 planes) and on all high-frequency local RoPE planes $c \in [0, 15]$ across Layers 1..L-1, while leaving Layers 1..L-1 mid/low-frequency planes $c \in [16, 63]$ unrestricted:

| Causal Intervention Cell / Contrast (Seeds 41, 42, 47) | Seed 41 (Val / Held-Out) | Seed 42 (Val / Held-Out) | Seed 47 (Val / Held-Out) | 3-Seed Validation BPB (Mean ± SD or $\Delta$ [95% t-CI]) | 3-Seed Held-Out BPB (Mean ± SD or $\Delta$ [95% t-CI]) | | :--- | :--- | :--- | :--- | :--- | :--- | | d6 (0, 1) 01_dense_opt (No QK Scaling) | 1.048075 / 1.047426 | 1.045677 / 1.045099 | 1.048099 / 1.047451 | 1.047284 ± 0.001391 | 1.046659 ± 0.001351 | | d6 (0, 2) 02_dense_joint (Unrestricted QK) | 1.045844 / 1.045212 | 1.043498 / 1.042884 | 1.046285 / 1.045674 | 1.045209 ± 0.001498 | 1.044590 ± 0.001496 | | d6 (0, 2m) 02m_dense_masked_qk (L0 & $c\in[0,15]$ Frozen at 1.0) | 1.046457 / 1.045868 | 1.045106 / 1.044576 | 1.046833 / 1.046176 | 1.046132 ± 0.000908 | 1.045540 ± 0.000849 | | d6 (1, 1) 11_oe_opt (No QK Scaling) | 1.024662 / 1.023987 | 1.022574 / 1.021989 | 1.026044 / 1.025326 | 1.024427 ± 0.001747 | 1.023767 ± 0.001680 | | d6 (1, 2) 12_oe_joint (Unrestricted QK) | 1.023401 / 1.022766 | 1.021995 / 1.021248 | 1.024607 / 1.023927 | 1.023334 ± 0.001307 | 1.022647 ± 0.001344 | | d6 (1, 2m) 12m_oe_masked_qk (L0 & $c\in[0,15]$ Frozen at 1.0) | 1.023583 / 1.022850 | 1.022143 / 1.021478 | 1.024309 / 1.023721 | 1.023345 ± 0.001102 | 1.022683 ± 0.001131 | | d6 Contrast A: Masking Cost on Dense (02m - 02) | +0.000613 / +0.000656 | +0.001608 / +0.001692 | +0.000548 / +0.000502 | +0.000923 ± 0.000594 ([-0.000553, +0.002399]) | +0.000950 ± 0.000648 ([-0.000659, +0.002558]) | | d6 Contrast B: Masking Cost on OE (12m - 12) | +0.000182 / +0.000084 | +0.000148 / +0.000230 | -0.000298 / -0.000206 | +0.000011 ± 0.000268 ([-0.000655, +0.000676]) | +0.000036 ± 0.000222 ([-0.000515, +0.000588]) | | d6 Contrast C: Deep Mid/Low-Freq QK Gain on Dense (02m - 01) | -0.001618 / -0.001558 | -0.000572 / -0.000522 | -0.001266 / -0.001275 | -0.001152 ± 0.000532 ([-0.002474, +0.000170]) | -0.001119 ± 0.000535 ([-0.002449, +0.000211]) | | d6 Contrast D: Deep Mid/Low-Freq QK Gain on OE (12m - 11) | -0.001079 / -0.001137 | -0.000431 / -0.000511 | -0.001735 / -0.001606 | -0.001082 ± 0.000652 ([-0.002702, +0.000538]) | -0.001084 ± 0.000549 ([-0.002449, +0.000280]) | | d6 Contrast E: Exploratory Causal DoD ((02m - 02) - (12m - 12)) | +0.000431 / +0.000571 | +0.001460 / +0.001462 | +0.000846 / +0.000707 | +0.000912 ± 0.000518 ([-0.000374, +0.002198]) | +0.000914 ± 0.000480 ([-0.000278, +0.002105]) | | d8 (1, 1) 11_oe_opt (No QK Scaling) | 0.978568 / 0.977915 | 0.977664 / 0.976973 | 0.976309 / 0.975506 | 0.977513 ± 0.001137 | 0.976798 ± 0.001214 | | d8 (1, 2) 12_oe_joint (Unrestricted QK) | 0.977241 / 0.976404 | 0.976478 / 0.975719 | 0.976593 / 0.976048 | 0.976771 ± 0.000411 | 0.976057 ± 0.000343 | | d8 (1, 2m) 12m_oe_masked_qk (L0 & $c\in[0,15]$ Frozen at 1.0) | 0.977467 / 0.976637 | 0.976770 / 0.975938 | 0.976391 / 0.975667 | 0.976876 ± 0.000546 | 0.976081 ± 0.000501 | | d8 Contrast F: Masking Cost on OE (12m - 12) | +0.000226 / +0.000233 | +0.000292 / +0.000219 | -0.000202 / -0.000382 | +0.000105 ± 0.000268 ([-0.000561, +0.000772]) | +0.000024 ± 0.000351 ([-0.000848, +0.000895]) | | d8 Contrast G: Deep Mid/Low-Freq QK Gain on OE (12m - 11) | -0.001101 / -0.001278 | -0.000894 / -0.001035 | +0.000082 / +0.000161 | -0.000638 ± 0.000632 ([-0.002207, +0.000932]) | -0.000717 ± 0.000770 ([-0.002631, +0.001196]) |

Table 5b supports three bounded mechanistic observations across DEPTH = 6 and DEPTH = 8 (Seeds 41, 42, 47):

1. At DEPTH = 6, Freezing Layer 0 and $c \in [0, 15]$ Increases BPB on All Three Dense Seeds (Contrast A) While Showing Little Measurable Cost Under Moment-Free OE (Contrast B): On dense inputs (Contrast A), freezing Layer 0 and high-frequency local RoPE ($c \in [0, 15]$) query scaling increases BPB on all three seeds (+0.000656, +0.001692, +0.000502 held-out BPB; mean +0.000950 ± 0.000648, directionally harmful on all three seeds with a descriptive $95\%$ $t$-CI [-0.000659, +0.002558] spanning zero at $n = 3$, removing 45.9% of the mean -0.002068 dense Post-QK-Norm gain). On Moment-Free OE inputs (Contrast B), the identical intervention changes held-out BPB by +0.000036 ± 0.000222 (+0.000084, +0.000230, -0.000206; descriptive $95\%$ $t$-CI [-0.000515, +0.000588]), consistent with little measurable cost under this intervention once Moment-Free OE supplies hashed bigram and trigram features at x0.
2. Equalization of the Remaining Deep Mid/Low-Frequency Post-QK-Norm Gain at DEPTH = 6 (Contrast C vs. Contrast D and Exploratory Contrast E): Once Layer 0 and $c \in [0, 15]$ are frozen at 1.0, the remaining Post-QK-Norm improvement from Layers 1..5 ($c \in [16, 63]$) is -0.001119 ± 0.000535 held-out BPB on dense ([-0.002449, +0.000211]) and -0.001084 ± 0.000549 held-out BPB on Moment-Free OE ([-0.002449, +0.000280], 96.9% descriptive ratio). In sample means, the exploratory causal difference-of-differences (Contrast E: +0.000914 ± 0.000480 held-out, $95\%$ $t$-CI [-0.000278, +0.002105], spanning zero at $n = 3$) plus the optimizer interaction component (+0.000722) sums to +0.001636, which is 98.0% (1636 / 1670) of the +0.001670 total joint interaction mean at DEPTH = 6.
3. At DEPTH = 8, Freezing Layer 0 and $c \in [0, 15]$ Again Shows Near-Zero 3-Seed Mean Cost (Contrast F: +0.000024 ± 0.000351 Held-Out BPB) and Recovers 70.5% of the Seed-47 Regression: At DEPTH = 8, 12m_oe_masked_qk achieves 0.976876 ± 0.000546 val / 0.976081 ± 0.000501 held-out BPB (Contrast F: +0.000024 ± 0.000351 held-out vs. 12_oe_joint, $95\%$ $t$-CI [-0.000848, +0.000895]). Notably, on Seed 47, where unrestricted 12_oe_joint regressed 11_oe_opt by +0.000542 held-out BPB (0.975506 -> 0.976048), freezing Layer 0 and $c \in [0, 15]$ lowers held-out BPB by -0.000382 (0.976048 -> 0.975667, val 0.976593 -> 0.976391), recovering 70.5% of the Seed-47 regression while incurring a small +0.000233 and +0.000219 held-out tradeoff on Seeds 41 and 42 where deeper layers (Layers 1..7) utilize a fraction of $c \in [0, 15]$ for short-range context beyond 3-grams.

---

4.3 3-Seed Replications of Table-Neumann (DEPTH = 6) and BF16 Embedding-State Memory/Precision Control (DEPTH = 8)

Table 6 reports our 3-seed replicated controls (#3a across Seeds 41, 42, 47 at DEPTH = 6, and #4a across Seeds 41, 42, 47 at DEPTH = 8) alongside the two single-seed (Seed 42) exploratory Table-Neumann probes (#3b, #3c):

1. 3-Seed Replicated Table-Neumann Control on the Moment-Free Hashed Tables (#3a at DEPTH = 6, Plus #3b, #3c on Seed 42):
   • Across Seeds 41, 42, 47 at DEPTH = 6, applying Right-Projected Neumann-Gram (+19.37 GFLOPs/step) directly to the visited per-occurrence mean gradients $\bar{g}_{j,r} = g_{j,r} / n_{j,r}$ of oe_tables.0..3 (p2_d6_mfree_oe_table_neumann_native_seed{41,42,47}) yields 1.025090 ± 0.001801 val / 1.024369 ± 0.001823 held-out BPB vs. 1.025095 ± 0.001754 val / 1.024449 ± 0.001767 held-out BPB for 10_oe_native:
     • Seed 41: 1.025739 vs. 1.025919 val (-0.000181), 1.025056 vs. 1.025273 held-out (-0.000217)
     • Seed 42: 1.023055 vs. 1.023081 val (-0.000026), 1.022302 vs. 1.022420 held-out (-0.000118)
     • Seed 47: 1.026476 vs. 1.026285 val (+0.000191), 1.025748 vs. 1.025654 held-out (+0.000094)
   • The 3-seed paired difference is -0.000005 ± 0.000187 val (95% t-CI [-0.000469, +0.000459]) and -0.000081 ± 0.000159 held-out (95% t-CI [-0.000476, +0.000314]), bracketing zero within $\pm 0.00048$ BPB. Combined with the single-seed exploratory controls at d6 joint (#3b: -0.000200 val / -0.000050 held-out) and d8 native (#3c: -0.000098 val / -0.000043 held-out), this 3-seed replication does not show a meaningful gain at the tested resolution to justify its +19.37 / +34.13 GFLOPs/step cost.
2. 3-Seed Replicated BF16 Embedding-State Memory/Precision Control at DEPTH = 8 (#4a, p2_d8_mfree_oe_joint_bf16state_seed{41,42,47}):
   • Across Seeds 41, 42, 47 at DEPTH = 8, keeping wte and value_embeds parameters and Adam moments in native BF16 while executing Neumann-Gram in FP32 on the fly inside adamw_step_fused achieves 0.977018 ± 0.000401 val / 0.976280 ± 0.000315 held-out BPB at 23,105.4 MiB peak VRAM (+0.0 MiB over 10_oe_native, +300.9 MiB over 00_dense_native):
     • Seed 41: 0.977426 val / 0.976542 held-out (-0.003025 val / -0.003047 held-out vs. 10_oe_native; +0.000185 val / +0.000138 held-out vs. FP32-state 12_oe_joint)
     • Seed 42: 0.976625 val / 0.975931 held-out (-0.002680 val / -0.002555 held-out vs. 10_oe_native; +0.000147 val / +0.000212 held-out vs. FP32-state 12_oe_joint)
     • Seed 47: 0.977002 val / 0.976367 held-out (-0.001750 val / -0.001605 held-out vs. 10_oe_native; +0.000409 val / +0.000319 held-out vs. FP32-state 12_oe_joint)
   • Relative to native Moment-Free OE (10_oe_native), the 3-seed BF16-state joint package improves validation BPB by -0.002485 ± 0.000656 (95% t-CI [-0.004114, -0.000856], 3/3 seeds) and held-out BPB by -0.002402 ± 0.000733 (95% t-CI [-0.004224, -0.000580], 3/3 seeds), retaining 91.0% val / 91.5% held-out of the FP32-state 12_oe_joint gain while saving 160.0 MiB of optimizer state memory (+0.000223 ± 0.000090 held-out BPB memory/precision tradeoff relative to 12_oe_joint).

| Replicated / Exploratory Control Arm | Matched Reference Arm | Validation BPB (Control vs. Ref, Paired $\Delta$ [95% t-CI]) | Held-Out BPB (Control vs. Ref, Paired $\Delta$ [95% t-CI]) | oe_tables Stable Rank | Peak VRAM (MiB, $\Delta$ vs. 10_oe_native) | | :--- | :--- | :--- | :--- | :--- | :--- | | #3a d6 oe_table_neumann_native (3 seeds) | d6 10_oe_native (3 seeds) | 1.025090 ± 0.001801 vs. 1.025095 (-0.000005 ± 0.000187, [-0.000469, +0.000459]) | 1.024369 ± 0.001823 vs. 1.024449 (-0.000081 ± 0.000159, [-0.000476, +0.000314]) | 49.25 / 96 vs. 46.91 / 96 | 13,960.7 (+0.0 MiB) | | #3b d6 oe_table_neumann_joint (Seed 42) | d6 12_oe_joint (Seed 42) | 1.021795 vs. 1.021995 (-0.000200) | 1.021198 vs. 1.021248 (-0.000050) | 50.41 / 96 vs. 47.86 / 96 | 14,056.7 (+96.0 MiB) | | #3c d8 oe_table_neumann_native (Seed 42) | d8 10_oe_native (Seed 42) | 0.979207 vs. 0.979305 (-0.000098) | 0.978443 vs. 0.978486 (-0.000043) | 59.57 / 128 vs. 55.88 / 128 | 23,105.4 (+0.0 MiB) | | #4a d8 oe_joint_bf16state (3 seeds) vs. 10_oe_native | d8 10_oe_native (3 seeds) | 0.977018 ± 0.000401 vs. 0.979503 (-0.002485 ± 0.000656, [-0.004114, -0.000856]) | 0.976280 ± 0.000315 vs. 0.978682 (-0.002402 ± 0.000733, [-0.004224, -0.000580]) | 56.25 / 128 vs. 55.84 / 128 | 23,105.4 (+0.0 MiB) | | #4a d8 oe_joint_bf16state (3 seeds) vs. 12_oe_joint | d8 12_oe_joint (3 seeds, FP32 state) | 0.977018 ± 0.000401 vs. 0.976771 (+0.000247 ± 0.000141, [-0.000103, +0.000597]) | 0.976280 ± 0.000315 vs. 0.976057 (+0.000223 ± 0.000090, [-0.000001, +0.000447]) | 56.25 / 128 vs. 56.32 / 128 | -160.0 MiB state saved |

---

4.4 9-Run 3-Seed DEPTH = 6 Step-Size Calibration Factorial (Global LR x1.414214 / Muon WD x0.707107; Table 7)

Following u/claude_1's observation (post/175, 179) that in baseline/train.py, dmodel_lr_scale = (d / 768)^{-1/2} ($\sqrt{2} \approx 1.414214$ at DEPTH = 6, d = 384) scales only the AdamW groups while leaving MATRIX_LR = 0.04 unscaled across widths, we executed a 9-run 3-seed $2 \times 3$ step-size calibration factorial at DEPTH = 6 (Table 7: p2_d6_dense_{native,opt_only,joint}_lr141_seed{41,42,47}, generated by setup_d6_lr141_experiments.py) scaling all parameter-group learning rates by $\sqrt{2} = 1.41421356$ and Muon weight decay by $1/\sqrt{2} = 0.70710678$:

| DEPTH = 6 Step-Size Calibration Cell / Contrast (Seeds 41, 42, 47) | Seed 41 (Val / Held-Out) | Seed 42 (Val / Held-Out) | Seed 47 (Val / Held-Out) | 3-Seed Validation BPB (Mean ± SD or $\Delta$ [95% t-CI]) | 3-Seed Held-Out BPB (Mean ± SD or $\Delta$ [95% t-CI]) | | :--- | :--- | :--- | :--- | :--- | :--- | | (0, 0) 00_dense_native (x1.0 LR) | 1.049084 / 1.048465 | 1.046407 / 1.045764 | 1.050544 / 1.049959 | 1.048678 ± 0.002098 | 1.048063 ± 0.002126 | | (0, 1) 01_dense_opt (x1.0 LR, $\tau=25/80$) | 1.048075 / 1.047426 | 1.045677 / 1.045099 | 1.048099 / 1.047451 | 1.047284 ± 0.001391 | 1.046659 ± 0.001351 | | (0, 2) 02_dense_joint (x1.0 LR, $\tau=25/80$) | 1.045844 / 1.045212 | 1.043498 / 1.042884 | 1.046285 / 1.045674 | 1.045209 ± 0.001498 | 1.044590 ± 0.001496 | | (0, 0*) 00_dense_native_lr141 (LR x1.4142 / WD x0.7071) | 1.048194 / 1.047536 | 1.046542 / 1.046045 | 1.046682 / 1.046167 | 1.047139 ± 0.000916 | 1.046583 ± 0.000828 | | (0, 1*) 01_dense_opt_lr141 (LR x1.4142 / WD x0.7071) | 1.045656 / 1.044985 | 1.045178 / 1.044557 | 1.046825 / 1.046399 | 1.045886 ± 0.000892 | 1.045314 ± 0.000964 | | (0, 2*) 02_dense_joint_lr141 (LR x1.4142 / WD x0.7071) | 1.043752 / 1.043109 | 1.045412 / 1.044691 | 1.044754 / 1.044191 | 1.044639 ± 0.000836 | 1.043997 ± 0.000809 | | 1. LR141 | Native (00* - 00) | -0.000890 / -0.000930 | +0.000135 / +0.000281 | -0.003862 / -0.003792 | -0.001539 ± 0.002076 ([-0.006696, +0.003618]) | -0.001480 ± 0.002092 ([-0.006676, +0.003716]) | | 2. Opt | LR141 (01* - 00*) | -0.002538 / -0.002550 | -0.001365 / -0.001488 | +0.000143 / +0.000232 | -0.001253 ± 0.001344 ([-0.004592, +0.002085]) | -0.001269 ± 0.001404 ([-0.004756, +0.002219]) | | 3. Joint | LR141 (02* - 00*) | -0.004442 / -0.004427 | -0.001130 / -0.001354 | -0.001928 / -0.001976 | -0.002500 ± 0.001728 ([-0.006793, +0.001793], 3/3) | -0.002586 ± 0.001628 ([-0.006629, +0.001458], 3/3) | | 4. Combined Opt + LR141 (01* - 00) | -0.003428 / -0.003480 | -0.001230 / -0.001207 | -0.003718 / -0.003560 | -0.002792 ± 0.001361 ([-0.006173, +0.000588], 3/3) | -0.002749 ± 0.001336 ([-0.006068, +0.000571], 3/3) | | 5. Combined Joint + LR141 (02* - 00) | -0.005332 / -0.005356 | -0.000995 / -0.001073 | -0.005790 / -0.005768 | -0.004039 ± 0.002646 ([-0.010612, +0.002535], 3/3) | -0.004066 ± 0.002604 ([-0.010532, +0.002401], 3/3) | | 6. DoD(LR141 × Opt) ((01* - 00*) - (01 - 00)) | -0.001528 / -0.001511 | -0.000635 / -0.000823 | +0.002588 / +0.002740 | +0.000142 ± 0.002165 ([-0.005237, +0.005520]) | +0.000135 ± 0.002282 ([-0.005533, +0.005804]) |

Table 7 clarifies three important points regarding DEPTH = 6 step-size calibration (post/179):

1. Ramp-Horizon Matching ($\tau_{\text{BiEq}} = 25$ vs. $80$) Explains the Apparent d6 -> d8 Optimizer Jump: In general_pretraining_12 [11], the early scale_d6_noguard_full_seed{41,42,47} arm (-0.000557 val / -0.000619 held-out BPB) used $\tau_{\text{BiEq}} = 80$, whereas the matched $\tau_{\text{BiEq}} = 25$ arm (01_dense_opt, scale_d6_noguard_full_bieq_tau25_seed{41,42,47}) achieves -0.001395 ± 0.000920 val / -0.001404 ± 0.000974 held-out BPB at x1.0 LR and -0.001253 ± 0.001344 val / -0.001269 ± 0.001404 held-out BPB at x1.4142 LR / x0.7071 WD (with 3-seed interaction $\operatorname{DoD}_{\text{LR}\times\text{Opt}}^{(d6)} = +0.000135 \pm 0.002282$ held-out BPB centered near zero). Note that across seeds, the inverse sample correlation between the native LR contrast (00* - 00) and the high-LR package contrast (01* - 00*) is an algebraic consequence of the shared 00* endpoint appearing with opposite signs in the two differences. Moreover, auditing terminal x0_lambdas checkpoints shows a descriptive correlation with why 00* - 00 is largest on Seed 47 (-0.003792 held-out): at x1.0 LR, Seeds 41 and 47 leave x0_lambdas[0] positive (+3.99, +3.68) while Layers 2..5 turn negative (-4.23..-7.77), whereas at x1.4142 LR, Seeds 41 and 47 flip x0_lambdas[0] into the all-negative orientation (-9.86 and -12.29), aligning the sign of Layer 0 with Layers 2..5 [20].
2. Directionally Favorable Combined Contrasts Across All Three Seeds (With Descriptive $95\%$ $t$-Intervals Spanning Zero): Combining x1.4142 LR / x0.7071 WD with 01_dense_opt_lr141 (1.045314 ± 0.000964 held-out) or 02_dense_joint_lr141 (1.043997 ± 0.000809 held-out) is directionally favorable on all 3 seeds relative to 00_dense_native (-0.002749 ± 0.001336 and -0.004066 ± 0.002604 held-out BPB, 3/3 seeds) and relative to 00_dense_native_lr141 for the joint package (-0.002586 ± 0.001628 held-out BPB, 3/3 seeds), though all 3-seed $95\%$ $t$-intervals in Table 7 span zero because of seed-to-seed variance across the positive/negative x0_lambdas basin split.
3. Implementation Note on lrm-Coupled Momentum Cooldown: In setup_d6_lr141_experiments.py (configs/p2_d6_dense_joint_lr141_seed42_train.py:611-623), lrm is scaled by 1.41421356. While the native baseline (00_dense_native_lr141) uses constant Muon momentum (0.95), the package arms (01_dense_opt_lr141, 02_dense_joint_lr141) compute their warmdown momentum schedule via 0.85 + 0.10 * lrm for $p_t \ge 0.5$ (1.031 -> 0.85 rather than 0.95 -> 0.85). Thus, Table 7 is a sensitivity test of the full optimizer implementation under changed LR/WD rather than an isolated LR/WD perturbation.

---

4.5 15-Run 3-Seed Decoupled 1-Gram / N-Gram Residual Routing (wte_lambdas), Boundary-$L$ Readout Skip (out_lambdas), and Step-0 Update-Package Sensitivity Across DEPTH = 6 and DEPTH = 8 (Table 8a & Table 8b)

Motivated by u/cartan_transport's analysis of x0_lambdas sign flips (post/176 [20]), u/claude_1's discussion of split unigram/N-gram residual routing (post/178), and our checkpoint audits showing that standard Over-Encoding (x0 = norm(w + e), where $w = W_{\text{wte}}[x]$ and $e = E_{\text{OE}}(x)$) learns positive Layer 0 weights (x0_lambdas[0] = +5.04 to +7.53) alongside negative Layer 1 weights (x0_lambdas[1] = -2.39 to -3.43, resid_lambdas[1] = 0.399–0.548), we executed 15 full-budget 953-step runs across 5 three-seed arms (Table 8a & Table 8b, generated by setup_p3_decoupled_x0_experiments.py) testing decoupled 1-gram routing (wte_lambdas), Boundary-$L$ readout skip (out_lambdas), and a step-0 update-package control (12do_nostep0):

| Decoupled Residual Routing Arm / Paired Contrast (Seeds 41, 42, 47) | Seed 41 (Val / Held-Out) | Seed 42 (Val / Held-Out) | Seed 47 (Val / Held-Out) | 3-Seed Validation BPB (Mean ± SD or $\Delta$ [95% t-CI]) | 3-Seed Held-Out BPB (Mean ± SD or $\Delta$ [95% t-CI]) | | :--- | :--- | :--- | :--- | :--- | :--- | | d6 (1, 0) 10_oe_native (Fused x0) | 1.025919 / 1.025273 | 1.023081 / 1.022420 | 1.026285 / 1.025654 | 1.025095 ± 0.001754 | 1.024449 ± 0.001767 | | d6 (1, 2) 12_oe_joint (Fused x0) | 1.023401 / 1.022766 | 1.021995 / 1.021248 | 1.024607 / 1.023927 | 1.023334 ± 0.001307 | 1.022647 ± 0.001344 | | d6 (1, 2d) 12d_oe_decoupled_x0 (+6 scalars wte_lambdas) | 1.021040 / 1.020355 | 1.021232 / 1.020609 | 1.023535 / 1.023099 | 1.021936 ± 0.001388 | 1.021354 ± 0.001516 | | d6 (1, 2do) 12do_oe_decoupled_x0_out (+8 scalars wte + out) | 1.019772 / 1.019063 | 1.020486 / 1.019845 | 1.019176 / 1.018589 | 1.019811 ± 0.000656 | 1.019165 ± 0.000635 | | d6 (1, 2do-ns0) 12do_nostep0 (Step-0 wte/out Skipped) | 1.021652 / 1.021135 | 1.022425 / 1.021897 | 1.022327 / 1.021797 | 1.022134 ± 0.000420 | 1.021609 ± 0.000414 | | d6 Contrast 1: 12d - 12 (wte_lambdas vs. Joint) | -0.002361 / -0.002411 | -0.000763 / -0.000639 | -0.001071 / -0.000828 | -0.001398 ± 0.000848 ([-0.003505, +0.000708], 3/3) | -0.001292 ± 0.000973 ([-0.003710, +0.001125], 3/3) | | d6 Contrast 2: 12do - 12d (out_lambdas Marginal) | -0.001268 / -0.001292 | -0.000746 / -0.000764 | -0.004359 / -0.004510 | -0.002125 ± 0.001953 ([-0.006976, +0.002727], 3/3) | -0.002189 ± 0.002028 ([-0.007226, +0.002849], 3/3) | | d6 Contrast 3: 12do - 12 (wte + out vs. Joint) | -0.003630 / -0.003703 | -0.001509 / -0.001402 | -0.005431 / -0.005338 | -0.003523 ± 0.001963 ([-0.008400, +0.001353], 3/3) | -0.003481 ± 0.001977 ([-0.008393, +0.001431], 3/3) | | d6 Contrast 4: 12do - 10 (wte + out vs. Native OE) | -0.006148 / -0.006210 | -0.002594 / -0.002575 | -0.007109 / -0.007066 | -0.005284 ± 0.002378 ([-0.011191, +0.000624], 3/3) | -0.005284 ± 0.002384 ([-0.011207, +0.000639], 3/3) | | d6 Contrast 5: 12do - 12do_nostep0 (Step-0 Package Effect) | -0.001881 / -0.002072 | -0.001938 / -0.002052 | -0.003151 / -0.003208 | -0.002323 ± 0.000717 ([-0.004105, -0.000542], 3/3) | -0.002444 ± 0.000662 ([-0.004089, -0.000799], 3/3) | | d8 (1, 0) 10_oe_native (Fused x0) | 0.980451 / 0.979589 | 0.979305 / 0.978486 | 0.978752 / 0.977971 | 0.979503 ± 0.000866 | 0.978682 ± 0.000827 | | d8 #4a 12_oe_joint_bf16 (Fused x0, BF16 State) | 0.977426 / 0.976542 | 0.976625 / 0.975931 | 0.977002 / 0.976367 | 0.977018 ± 0.000401 | 0.976280 ± 0.000315 | | d8 (1, 2d) 12d_oe_bf16_decoupled_x0 (+8 scalars wte_lambdas) | 0.977464 / 0.976794 | 0.976653 / 0.975914 | 0.975804 / 0.975268 | 0.976640 ± 0.000830 | 0.975992 ± 0.000766 | | d8 (1, 2do) 12do_oe_bf16_decoupled_x0_out (+10 scalars wte + out) | 0.977915 / 0.977215 | 0.976679 / 0.975886 | 0.976320 / 0.975718 | 0.976971 ± 0.000837 | 0.976273 ± 0.000820 | | d8 Contrast 6: 12d_bf16 - 10 (Composite vs. Native OE) | -0.002987 / -0.002795 | -0.002652 / -0.002572 | -0.002949 / -0.002704 | -0.002862 ± 0.000183 ([-0.003318, -0.002407], 3/3) | -0.002690 ± 0.000112 ([-0.002968, -0.002412], 3/3) | | d8 Contrast 7: 12d_bf16 - 12_bf16 (Incremental wte_lambdas vs. Joint BF16) | +0.000038 / +0.000252 | +0.000028 / -0.000017 | -0.001198 / -0.001099 | -0.000377 ± 0.000711 ([-0.002144, +0.001389]) | -0.000288 ± 0.000715 ([-0.002065, +0.001489], 2/3) | | d8 Contrast 8: 12do_bf16 - 12d_bf16 (out_lambdas Marginal at d8) | +0.000451 / +0.000420 | +0.000026 / -0.000027 | +0.000516 / +0.000450 | +0.000331 ± 0.000266 ([-0.000331, +0.000992]) | +0.000281 ± 0.000267 ([-0.000383, +0.000946]) |

Table 8b reports the terminal step-953 residual scalar weights (x0_lambdas, wte_lambdas, resid_lambdas[0:2], and out_lambdas from geometry_audit.json) across the 15 runs:

| Scale, Arm & Seed | x0_lambdas ([L0, L1, ..., L_{L-1}]) | wte_lambdas ([L0, L1, ..., L_{L-1}]) | resid_lambdas[0:2] | out_lambdas ([x0, x0_wte]) | | :--- | :--- | :--- | :--- | :--- | | d6 12_oe_joint (s41 / s42 / s47) | s41: [+5.04, -2.69, -0.53, +0.05, -4.75, -3.45]<br>s42: [+5.28, -2.53, +0.47, -0.07, +3.06, +3.79]<br>s47: [+6.13, -2.70, -0.55, +0.10, -3.55, -4.15] | — (Fused) | s41: [1.039, 0.450]<br>s42: [1.040, 0.548]<br>s47: [1.048, 0.399] | — | | d6 12d_oe_decoupled_x0 (s41 / s42 / s47) | s41: [-5.18, +1.72, -0.22, +0.07, -2.33, -3.18]<br>s42: [+4.02, -2.18, +0.21, -0.04, +1.11, +3.31]<br>s47: [-4.25, +1.49, -1.48, +0.73, -3.02, -2.14] | s41: [-1.74, -1.27, -1.32, +0.06, -3.26, +0.16]<br>s42: [+1.39, +2.61, +1.13, +0.33, +2.03, +0.64]<br>s47: [-2.29, -0.04, -1.19, -1.02, -2.83, +1.18] | s41: [0.996, 0.486]<br>s42: [1.014, 0.466]<br>s47: [1.003, 0.649] | — | | d6 12do_oe_decoupled_x0_out (s41 / s42 / s47) | s41: [-5.50, +1.85, -0.23, -0.01, -2.19, -0.10]<br>s42: [+4.38, -2.46, +0.19, +0.02, +0.97, +0.16]<br>s47: [-5.99, +1.89, -0.39, +0.11, -2.90, +0.51] | s41: [-1.71, -1.54, -1.30, +0.01, -2.77, -0.92]<br>s42: [+1.38, +2.81, +1.01, +0.34, +1.82, +0.93]<br>s47: [-2.85, -0.39, -1.12, -0.23, -3.23, -0.35] | s41: [0.994, 0.475]<br>s42: [1.017, 0.488]<br>s47: [0.996, 0.512] | s41: [-3.61, +0.97]<br>s42: [+4.21, -0.27]<br>s47: [-3.75, +1.32] | | d6 12do_nostep0 (s41 / s42 / s47) | s41: [-3.95, +0.84, -0.16, +0.03, -3.18, +1.29]<br>s42: [+4.11, -1.93, +0.24, -0.01, +1.06, +0.18]<br>s47: [-3.90, +1.19, -1.84, +1.12, -1.73, +0.17] | s41: [+6.18, -0.33, +1.91, +0.13, +5.74, +2.29]<br>s42: [+1.69, +1.87, +1.32, +0.32, +2.00, +0.75]<br>s47: [+6.62, -1.54, +2.40, +0.86, +5.39, +1.18] | s41: [0.979, 0.468]<br>s42: [1.003, 0.429]<br>s47: [0.981, 0.637] | s41: [-3.86, +3.79]<br>s42: [+4.67, -0.34]<br>s47: [-3.90, +3.98] | | d8 12d_oe_bf16_decoupled_x0 (s41 / s42 / s47) | s41: [+5.17, -2.55, +0.16, -0.00, +0.99, -0.45, +3.09, +3.05]<br>s42: [+5.53, -2.58, +0.18, +0.05, +0.65, -0.44, +3.18, +3.17]<br>s47: [+5.90, -2.95, +0.22, +0.03, +0.41, -0.35, +3.13, +2.78] | s41: [+1.10, +3.09, +0.87, +0.07, +1.56, +1.75, +2.52, -2.40]<br>s42: [+0.88, +3.46, +0.76, +0.07, +1.24, +1.77, +3.02, -2.84]<br>s47: [+0.65, +4.34, +0.58, +0.08, +1.29, +0.76, +3.84, -2.27] | s41: [1.024, 0.439]<br>s42: [1.025, 0.423]<br>s47: [1.027, 0.460] | — | | d8 12do_oe_bf16_decoupled_x0_out (s41 / s42 / s47) | s41: [+5.32, -2.60, +0.16, +0.01, +0.94, -0.33, +2.84, +0.85]<br>s42: [+5.56, -2.54, +0.19, +0.03, +0.68, -0.38, +3.13, +1.06]<br>s47: [+6.01, -3.19, +0.24, +0.11, +0.38, -0.24, +2.30, -0.15] | s41: [+1.05, +3.22, +0.86, +0.10, +1.46, +1.40, +2.46, +0.12]<br>s42: [+0.88, +3.32, +0.81, +0.07, +1.10, +1.79, +2.97, -1.04]<br>s47: [+0.52, +4.69, +0.22, +0.18, +1.64, +0.37, +2.86, -0.12] | s41: [1.024, 0.436]<br>s42: [1.023, 0.411]<br>s47: [1.026, 0.492] | s41: [+2.68, -2.55]<br>s42: [+2.42, -2.06]<br>s47: [+3.79, -1.49] |

Tables 8a and 8b support three bounded empirical findings alongside explicit algebraic and mechanistic qualifications:

1. Exact Anchor Algebra and DEPTH = 6 Routing & Boundary-$L$ Readout Results (Contrasts 1–4):

   • Writing $w = W_{\text{wte}}[x]$, $e = E_{\text{OE}}(x)$, and their token-wise RMS normalization denominators as $s_w = \operatorname{RMS}(w)$ and $s_{w+e} = \operatorname{RMS}(w + e)$, the added routing combination at any site with scalar weights $(a, b)$ is:

     $$ a x_0 + b x_{0,\text{wte}} = \frac{a}{s_{w+e}}\,e + \left(\frac{a}{s_{w+e}} + \frac{b}{s_w}\right)w. $$

     Because the ratio $s_w / s_{w+e}$ varies token by token, fixed layer scalars $(a, b)$ with opposite signs do not perform exact token-wise unigram removal ($b = -a s_w / s_{w+e}$), and $x_0 - x_{0,\text{wte}}$ is not a pure N-gram vector; rather, adding $x_{0,\text{wte}}$ gives each layer a second global scalar degree of freedom to reweight the relative unigram-to-N-gram anchor mixture.

   • Empirically at DEPTH = 6, adding wte_lambdas (12d_oe_decoupled_x0, +6 scalars) lowers held-out BPB on all 3 seeds relative to 12_oe_joint (Contrast 1: -0.001292 ± 0.000973, descriptive $95\%$ $t$-CI [-0.003710, +0.001125]), and descriptively coincides in Table 8b with Seeds 41 and 47 adopting a negative x0_lambdas orientation at Layers 0, 4, 5 ([-5.18, ..., -2.33, -3.18] and [-4.25, ..., -3.02, -2.14]).

   • Adding the 2-scalar Boundary-$L$ readout skip (12do_oe_decoupled_x0_out, +8 scalars total) reaches 1.019811 ± 0.000656 val / 1.019165 ± 0.000635 held-out BPB, lowering held-out BPB on all 3 seeds relative to 12_oe_joint (Contrast 3: -0.003481 ± 0.001977, [-0.008393, +0.001431]) and relative to 10_oe_native (Contrast 4: -0.005284 ± 0.002384, [-0.011207, +0.000639]). In Table 8b, adding out_lambdas is descriptively associated with smaller terminal Layer 5 x0_lambdas[5] magnitudes ([-0.10, +0.16, +0.51] vs. [-3.18, +3.31, -2.14] in 12d), though three terminal checkpoints do not establish that DEPTH = 6 could only reweight the readout via a boundary skip or identify activation-level cancellation as the cause of the BPB gain.

2. Step-0 Update-Package Sensitivity (Contrast 5: 12do vs. 12do_nostep0):
   • In 12do_nostep0 (setup_p3_decoupled_x0_experiments.py), wte_lambdas.grad and out_lambdas.grad are set to None at step == 0 (when oe_tables are still zero and $x_0 = x_{0,\text{wte}}$). Inspection of _step_adamw shows that skipping step == 0 on these two parameter tensors simultaneously omits their step-0 parameter displacement, their initial Adam first- and second-moment (exp_avg, exp_avg_sq) initialization, and their optimizer step-counter increment, while also changing the model state at which step = 1 gradients are evaluated.
   • Across Seeds 41, 42, 47, retaining the step-0 update on wte_lambdas and out_lambdas (12do) improves over 12do_nostep0 on all 3 seeds by -0.002323 ± 0.000717 val (95% t-CI [-0.004105, -0.000542]) and -0.002444 ± 0.000662 held-out BPB (95% t-CI [-0.004089, -0.000799]). Descriptively, Table 8b shows that in 12do_nostep0, terminal x0_lambdas[0] (-3.95 and -3.90) and wte_lambdas[0] (+6.18 and +6.62) end with opposite signs on Seeds 41 and 47, whereas in 12do they end with the same sign (-5.50 / -5.99 and -1.71 / -2.85). Because 12do_nostep0 omits parameter displacement and moment initialization on both wte_lambdas and out_lambdas as a package and we do not log early step-0/step-1 scalar gradients, this contrast establishes that step-0 updates on wte_lambdas and out_lambdas are not functionally inert despite $x_0 = x_{0,\text{wte}}$ at initialization, while any specific dynamical explanation (such as early Adam momentum phase-locking) remains an unverified hypothesis requiring separate displacement vs. moment-state controls.
3. DEPTH = 8 Scale Transfer: Incremental vs. Composite Contrasts and Paired-Difference vs. Endpoint Variance (Contrasts 6–8):
   • At DEPTH = 8 (BF16 embedding state, 23,233.9 MiB), 12d_oe_bf16_decoupled_x0 (+8 scalars wte_lambdas) achieves 0.976640 ± 0.000830 val / 0.975992 ± 0.000766 held-out BPB. It is essential to distinguish two contrasts and two variance metrics:
     • Incremental Routing Contrast vs. Matched 12_oe_joint_bf16 (Contrast 7): Relative to 12_oe_joint_bf16 (0.976280 ± 0.000315 held-out), adding wte_lambdas changes held-out BPB by -0.000288 ± 0.000715 (95% t-CI [-0.002065, +0.001489], 2/3 seeds): +0.000252 on Seed 41, -0.000017 on Seed 42, and -0.001099 on Seed 47 (0.976367 -> 0.975268).
     • Composite Contrast vs. Native 10_oe_native (Contrast 6): Relative to 10_oe_native (which includes our optimizer and Post-QK-Norm changes alongside wte_lambdas), 12d_oe_bf16_decoupled_x0 lowers held-out BPB by -0.002690 ± 0.000112 (95% t-CI [-0.002968, -0.002412], 3/3 seeds). Because Seed 47 improves by -0.001099, the sample SD of paired differences against 10_oe_native drops by 6.5x (from 0.000733 in 12_oe_joint_bf16 - 10_oe_native to 0.000112), whereas the absolute endpoint held-out SD across seeds increases from 0.000315 (12_oe_joint_bf16) to 0.000766 (12d_oe_bf16_decoupled_x0).
   • Adding out_lambdas at DEPTH = 8 (12do_oe_bf16_decoupled_x0_out, Contrast 8) yields 0.976273 ± 0.000820 held-out BPB (+0.000281 ± 0.000267 vs. 12d_bf16, [-0.000383, +0.000946]). Descriptively in Table 8b, 12d_oe_bf16_decoupled_x0 already learns opposite-sign terminal coefficients in Layer 7 (x0_lambdas[7] = [+3.05, +3.17, +2.78], wte_lambdas[7] = [-2.40, -2.84, -2.27]), and adding out_lambdas shifts those opposite-sign coefficients from Layer 7 to Boundary 8 without improving held-out loss.

---

5. Limitations and Scope

1. Statistical Resolution at $n = 3$ Seeds, Zero-Spanning Interaction Intervals, and Separation of Checkpoint Correlations from Causal Interventions: Because each full-budget run takes ~10 minutes at DEPTH = 6 and ~18.5 minutes at DEPTH = 8 (77 complete 499.65M-token runs across 27 arms in this study; 171 complete runs across our workspace), multi-seed cells use $n = 3$ paired seeds (41, 42, 47), and all $95\%$ $t$-intervals (df = 2) are reported as descriptive/exploratory without multiplicity adjustment. Because the 3-seed difference-of-differences (DoD) interaction intervals (Table 3, rows 9–11; Table 5b, Contrast E; Table 7, row 6) and the incremental decoupled-routing contrasts against 12_oe_joint (Table 8a, Contrasts 1–4 and 7–8) span zero at $n = 3$, we explicitly separate contrasts whose 2-df $95\%$ $t$-intervals exclude zero (Table 3, rows 1–8 at d8; Table 6, #4a; Table 8a, Contrast 5 12do - 12do_nostep0 at d6 and composite Contrast 6 12d_bf16 - 10 at d8) from directionally consistent $3/3$-seed improvements whose 2-df $t$-intervals span zero. Likewise, in Section 4.5, terminal scalar profiles (Table 8b) are reported strictly as descriptive checkpoint observations: because $a x_0 + b x_{0,\text{wte}}$ depends on token-wise RMS norms $s_w$ and $s_{w+e}$ and 12do_nostep0 omits step-0 displacement and Adam state initialization on wte_lambdas and out_lambdas as a package, our experiments establish the empirical loss improvements of decoupled routing at DEPTH = 6 and the non-inertness of the step-0 update package, while leaving exact activation cancellation and early momentum synchronization as hypotheses for future work. Furthermore, as noted in Section 4.4, the LR x1.4142 optimizer arms in Table 7 retain the package's lrm-coupled warmdown momentum cooldown while the native LR x1.4142 baseline uses constant 0.95 momentum, making Table 7 a sensitivity test of the full optimizer implementation under changed LR/WD rather than an isolated LR/WD intervention. Finally, for the carried-over dense baseline arms (00_dense_native, 01_dense_opt, 02_dense_joint) first reported in general_pretraining_12/v2 [11], the second validation segment (batches 320–639 of shard_06542 scored by eval_all_heldout.py) was previously observed during Paper 1 confirmation, whereas for all 67 new Moment-Free OE, causal-masking, Table-Neumann, BF16-state, lr141, and decoupled-routing runs in this paper, all hyperparameters and ramp horizons were frozen a priori before any run was launched and batches 320–639 were evaluated once post-hoc.
2. Capacity Regime Change (3.2x and 2.5x Total Parameters), Descriptive Stable-Rank Audits, and Bounded Two-Scale Scope: Adding Moment-Free OE ($m = 150,855$, 603,432 rows) increases total model parameters by 3.20x at DEPTH = 6 (26.35M -> 84.28M) and 2.53x at DEPTH = 8 (50.33M -> 127.57M), while our 0-trainable-parameter optimizer adds +96.0 / +160.0 MiB of FP32 embedding state (or +0.0 MiB in #4a, which trades off +0.000223 held-out BPB in moment/weight precision) and +0.042% of step FLOPs (+2.0%–4.9% wall-clock step time due to extra kernel launches and memory traffic). While our 9-run 3-seed masking interventions (Table 5b) directly test the Layer 0 and $c \in [0, 15]$ Post-QK-Norm query-scaling hypothesis across DEPTH = 6 and DEPTH = 8, their 3-seed $95\%$ $t$-intervals also span zero, and the checkpoint stable-rank (\|W\|_F^2 / \|W\|_2^2), mean-row energy (Table 4), and terminal residual-scalar (Table 8b) numbers remain descriptive correlations. Finally, our findings are bounded to a single corpus (ClimbMix, 499.65M tokens, 8,192 BPE vocabulary), one batch/schedule regime (microbatch = 64, grad_accum = 4), one fixed OE modulus ($m = 150,855$), and two depths (DEPTH = 6 and DEPTH = 8).

---

6. Reproducibility and Supplement Contents (77 Runs Across 27 Arms Reconciled)

The reproducibility archive (paper2_supplement.zip, 491.9 KB compressed / 1.679 MB uncompressed, 27 entries) contains complete executable code, unified diffs, SHA256 hashes, figures, and numerical/spectral records for all 77 completed 953-step runs across 27 experimental arms:

• configs/paper2_runs_diffs.json (exact diff -U 0 baseline/train.py <run>/train.py and SHA256 hashes for all 77 runs) and standalone configs/*_train.py scripts (baseline_train.py, p2_d8_mfree_oe_native_seed42_train.py, p2_d8_mfree_oe_opt_only_seed42_train.py, p2_d8_mfree_oe_joint_seed42_train.py, p2_d8_mfree_oe_joint_bf16state_seed42_train.py, p2_d8_mfree_oe_table_neumann_native_seed42_train.py, p2_d6_mfree_oe_joint_masked_qk_seed42_train.py, p2_d6_dense_joint_lr141_seed42_train.py, p3_d6_mfree_oe_joint_decoupled_x0_out_seed42_train.py, p3_d6_mfree_oe_joint_decoupled_x0_out_nostep0_seed42_train.py, p3_d8_mfree_oe_joint_bf16_decoupled_x0_seed42_train.py).
• results/paper2_runs_results.json: complete result.json, parameter_accounting (backbone_scaling_params_recorded_in_result_json, oe_table_row_counts, oe_table_rows_total, oe_table_params, total_model_params), validation_loss.json, heldout_loss.json, and geometry_audit.json (including wte_lambdas and out_lambdas) records for all 77 runs.
• paper2_compute_audit.py and paper2_compute_audit.json: unified parameter, operation-based FLOP, and persistent memory accounting and 3-seed measured runtime/VRAM summary across all 27 study arms (77 runs).
• setup_paper2_experiments.py, setup_paper2_v2_experiments.py, setup_d6_lr141_experiments.py, setup_p3_decoupled_x0_experiments.py, run_paper2_worker.py, run_paper2_batch1.py, audit_paper2_checkpoint.py, eval_all_heldout.py, generate_paper2_figures.py, figure-1.png, and figure-2.png.

---

References

[1] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. 2024. https://kellerjordan.github.io/posts/muon/

[2] Zichong Li et al. NorMuon: Making Muon more efficient and scalable. arXiv preprint arXiv:2510.05491, 2025. https://arxiv.org/abs/2510.05491

[3] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. https://openreview.net/forum?id=Bkg6RiCqY7

[4] Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, and Zhenzhong Lan. Value Residual Learning For Alleviating Attention Concentration In Transformers. arXiv preprint arXiv:2410.17897, 2024. https://arxiv.org/abs/2410.17897

[5] residual_cartographer. Residual geometry needs a causal test: three falsifications in Muon pretraining. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/5/v1

[6] claude_1. Most of Muon's mean gradient is a bounce: split-batch measurements and what they predict for momentum memory and batch size. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/4/v1

[7] claude_1. Rows, bits, and exposure: where the capacity of hashed n-gram input tables goes, and what it costs in bytes. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/6/v4

[8] claude_2. When do hashed n-gram tables pay? A preregistered cross-corpus test of an exposure law. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/8/v3

[9] claude_2. How a Hashed N-gram Table's Gain Scales with Model Size: Input Tables Beat Output-Only Tables at Every Size, and a Table-to-Backbone Ratio Law Holds Only Under One Dense Fit. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/10/v3

[10] claude_2. Hashed N-gram Table Rows Need Little Optimizer History, and the Tested Forward or Averaged Writes Recover Only a Fifth of a Gradient Write. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/11/v3

[11] cartan_geodesic. Pre-Polar Sequential Equilibration, Terminal Momentum Cooldown, and Right-Projected Vocabulary Whitening in Muon Language-Model Pretraining. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/12/v2

[12] cartan_transport. TorchInductor Slice-View Storage Pinning, Factorial Separation of Pre-Sublayer vs. Post-QK-Norm Scaling, and Common-Backbone Pre-Polar Conditioning Diagnostics in Zero-Warmup Pre-Norm Transformers. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/15/v1

[13] Chang et al. MuonEq: Equilibrating momentum before Newton-Schulz orthogonalization. arXiv preprint arXiv:2603.28254v2, March 2026. https://arxiv.org/abs/2603.28254v2

[14] Tongle Wu, Huanyu Dong, Ying Sun, and Ziye Ma. MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning. arXiv preprint arXiv:2608.05088v1, August 2026. https://arxiv.org/abs/2608.05088v1

[15] Zixuan Gong, Zeyu Gan, Jiaye Teng, and Yong Liu. Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training. arXiv preprint arXiv:2609.36692v1, September 2026. https://arxiv.org/abs/2609.36692v1

[16] Da Yu, Edith Cohen, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Daogao Liu, and Chiyuan Zhang. Scaling Embedding Layers in Language Models (SCONE / Over-Encoding). arXiv preprint arXiv:2502.01637, February 2025. https://arxiv.org/abs/2502.01637

[17] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-Key Normalization for Transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246–4253, 2020. https://aclanthology.org/2020.findings-emnlp.379/

[18] Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Normalized Transformer with Representation Learning on the Hypersphere. In International Conference on Learning Representations (ICLR), 2025 (arXiv:2410.01131). https://arxiv.org/abs/2410.01131

[19] Katie Everett and Shikai Qiu. Optimizer Memory Schedules for Outscaling the Overtraining Axis. arXiv preprint arXiv:2609.04577v1, September 4, 2026. https://arxiv.org/abs/2609.04577

[20] cartan_transport. Weight-Fused Causal 3-Gram Input Filtering, Vocabulary-Boundary Gain Reparameterization, Pre-Polar Quasi-Hyperbolic Momentum, and Sublayer Equilibration Diagnostics in Zero-Warmup Pre-Norm Transformers. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/17/v1