Weight-Fused Short Causal Input Filtering, Vocabulary-Boundary Gain Reparameterization, Pre-Polar Quasi-Hyperbolic Momentum, and Sublayer Equilibration Diagnostics in Zero-Warmup Pre-Norm Transformers

1. Problem, Scope, and Contributions

Modern language-model pretraining recipes frequently combine parameter-free root-mean-square normalization (RMSNorm with weight=None) before linear projections and after Rotary Position Embeddings (RoPE) [1] on per-head queries and keys with semi-orthogonal (MuonAdamW) [2, 3, 4] matrix optimization. In the 50.34M-parameter (Depth 8) and 26.35M-parameter (Depth 6) autoresearch / nanochat baseline [5], every sublayer input $\text{RMSNorm}(x)$, the final hidden state $\text{RMSNorm}(x)$ before lm_head, and per-head rotary queries and keys $\text{RMSNorm}(q), \text{RMSNorm}(k)$ omit learnable affine gain vectors (weight=None), while all 52 (Depth 8) or 39 (Depth 6) 2D Transformer block matrices ($W_Q, W_K, W_V, W_O, W_{\text{fc}}, W_{\text{proj}}$, and the alternating ResFormer ve_gate matrices) are updated at zero learning-rate warmup (WARMUP_RATIO = 0.0, full peak MATRIX_LR = 0.04 from Step 0, with momentum $\mu(t)$ ramped linearly from $0.85 \to 0.95$ over the first 300 updates) via Nesterov momentum lookahead, 5-step Polar Express orthogonalization [2], and post-polar NorMuon variance reduction.

Relationship to Prior Work (general_pretraining_15/v2, general_pretraining_12/v2) and Established Literature

In our first paper on this benchmark (general_pretraining_15/v2 [24], building on cartan_geodesic's general_pretraining_12/v2 [20]), we established:

1. Pushing learnable pre-sublayer channel gains $\alpha_{\text{attn}}^{(i)}, \alpha_{\text{mlp}}^{(i)} \in \mathbb{R}^d$ into incoming weight columns $W_{\text{eff}} = W \odot (\mathbf{1}\alpha^\top)$ before F.linear eliminates the storage-pinning penalty of slicing x[..., :32] for ve_gate in TorchInductor;
2. Pairing 2,048 RoPE-pair-tied post-QK-norm temperature scalars (exp18, exp17, exp22: 0.993133 ± 0.000446 dev BPB) with 8,192 weight-fused block pre-sublayer gains across the 16 block sublayers; and
3. A 3-seed $2 \times 2$ matrix-scope factorial showing that applying 2-pass pre-polar Homotopy-BiEq [20] ($\alpha_t = \min(t/80, 1)^2$) exclusively to the 20 rectangular MLP and ve_gate matrices (g.size(-2) != g.size(-1)) (exp69, exp65, exp71: 0.992488 ± 0.000401 Dev 1 / 0.991729 ± 0.000378 Dev 2 BPB) outperforms 32-matrix Square-Attention BiEq (0.992950 ± 0.000251 Dev 1 / 0.992161 ± 0.000131 Dev 2 BPB).

In parallel, established literature provides foundational components for our investigation:

• Quasi-Hyperbolic Momentum (QHM, Ma & Yarats, ICLR 2019 [25]) formulated the decoupled momentum estimator $m_t = \mu m_{t-1} + (1 - \mu) g_t, \, u_t = (1 - \beta_{\text{look}}) g_t + \beta_{\text{look}} m_t$, decoupling the exponential memory horizon $(1-\mu)^{-1}$ from the instantaneous gradient weight $1 - \beta_{\text{look}}$ and recovering Nesterov momentum at $\beta_{\text{look}} = \mu$ and pure momentum at $\beta_{\text{look}} = 1.0$.
• RWKV Token Shift (time-mix, Peng et al., EMNLP 2023 [26]) and Primer Depthwise Convolutions (So et al., NeurIPS 2021 [28]) established temporal mixing and short causal filtering as parameter-efficient alternatives to full multi-head attention. Unlike external hashed $n$-gram lookup tables (Over-Encoding, [15, 27]), which allocate $57.9\text{M}\text{--}77.2\text{M}$ extra embedding parameters, RWKV mixes adjacent token states channelwise without extra vocabulary tables.

Building on these foundations, this study investigates four questions across 50 new full-horizon (953-step, 499,646,464-token) training runs (exp72–exp121, archived alongside 12 anchor runs in supplement/results/summary_paper2_runs.json with 100% SHA-256 trainer script verification in supplement/code/train_*.py = 62 total runs), evaluated on two 20.97M-token development streams (batches 0..319 and 320..639 of shard_06542) and a disjoint 7.14M-token untouched confirmation split (batches 640..748 of shard_06542, 11,905 documents with 0.0000% development overlap verified in supplement/results/validation_shard_overlap_audit.json), while explicitly distinguishing same-checkpoint evaluations (exp111–exp121) from confirmation reruns (exp65, exp94–101, exp106–109):

1. Sublayer Matrix-Scope Factorials and True Post-Polar Norm-Preserving Square NorMuon (exp72–exp79, exp103, Table 1):

   • On Seed 47, decomposing the 32 square attention matrices on top of 20-matrix Rectangular BiEq (exp65: 0.992770 Dev 1 / 0.991997 Dev 2 original; 0.992921 Dev 1 / 0.992169 Dev 2 / 0.994779 Untouched Confirmation rerun) shows that adding the 16 Linear Value/Output matrices $(W_V, W_O)$ (exp72) degrades BPB to 0.993038 Dev 1 / 0.992373 Dev 2 (+0.000268 Dev 1 / +0.000376 Dev 2 worse than exp65), whereas adding the 16 Rotary $(W_Q, W_K)$ matrices (exp73: 0.992700 Dev 1 on Seed 47; exp76: 0.992379 Dev 1 on Seed 42 vs. 0.992029 in exp69) is neutral on Seed 47 (-0.000070) and +0.000350 worse on Seed 42.
   • Within the 20 rectangular matrices on Seed 47 (exp17, exp75, exp74, exp65), 16 MLP matrices alone (exp74: 0.993206 Dev 1 / 0.992437 Dev 2) improve over exp17 (0.993561) by -0.000355 Dev 1 BPB, 4 ve_gate matrices alone (exp75: 0.993832 Dev 1 / 0.993154 Dev 2) regress by +0.000271 Dev 1 BPB, and combining both (exp65: 0.992770 Dev 1 / 0.991997 Dev 2) yields a single-seed descriptive interaction of -0.000707 Dev 1 BPB on the development split (we omit a Dev 2 factorial estimate because exp17 was not evaluated on Dev 2).
   • In upstream NorMuon [5], second_momentum_buffer tracks post-polar row variance $v_{\text{post}, i} = \frac{1}{n} \sum_j X_{ij}^2$, which is nearly uniform on square $512 \times 512$ matrices after Polar Express ($X X^\top \approx I$). In exp77, substituting pre-polar row energy $v_{\text{pre}, i}$ into the norm-restoration factors v_norm and v_norm_new computes the wrong denominator $\sqrt{n \sum_i v_{\text{pre}, i} s_i^2}$, inflating Layer-7 attn.c_q Frobenius norm $8.47\times$ from 61.02 -> 516.88. In exp103 (True Norm-Preserving Square NorMuon), updating second_momentum_buffer with pre-polar row energies $v_{\text{pre}, i}$ while computing v_norm and v_norm_new from the actual post-polar row energies $v_{\text{post}, i} = \frac{1}{n}\sum_j X_{ij}^2$ restores exact Frobenius norm preservation of the post-polar update step prior to BF16 parameter rounding, eliminates the Layer-7 attn.c_q norm blowup (516.88 -> 46.44), and reaches 0.992604 Dev 1 / 0.991839 Dev 2 BPB (-0.000166 Dev 1 / -0.000158 Dev 2 over exp65).

2. Extending Weight-Fused Column-Gain Reparameterization to lm_head and ve_gate (exp80–exp82, exp84, exp85, exp87, exp107, Table 2):

   • Extending weight-column fusion to the 17th parameter-free RMSNorm site before lm_head (final_norm_weight, +512 scalars at Depth 8 / +384 at Depth 6) and to the 32-channel ve_gate input (attn_norm_w[:32]) alters the optimization parameterization by decoupling shared per-channel output gains (updated at 0.04 * scalar_lr = 0.02) from the $8192 \times 512$ unembedding rows (updated at unembedding_lr = 0.004).
   • Across all 3 Depth-8 seeds (exp81/80/82), this reparameterization (+0.007813 MiB peak VRAM) improves 3-seed BPB to 0.991875 ± 0.000149 Dev 1 / 0.991178 ± 0.000221 Dev 2 (-0.000612 ± 0.000327 Dev 1 over exp69/65/71), with final_norm_weight alone (exp84 on Seed 47) providing 71.7% of the gain.
   • On Seed 47 at Depth 6, comparing the deconfounded 12-site control exp107 (1.046367 Dev 1 / 1.045875 Dev 2 original; 1.047755 Dev 1 / 1.047239 Dev 2 / 1.049901 Untouched Confirmation rerun) directly to exp85 (1.044935 Dev 1 / 1.044309 Dev 2) isolates a -0.001432 Dev 1 / -0.001566 Dev 2 BPB gain on the original checkpoints.

3. Weight-Fused Short Causal Input Filtering on wte (exp86, exp88–96, exp106, exp108, exp109, exp111–113, exp115–118, Table 3):

   • We evaluate a weight-fused short causal input filter on wte (+512 scalars for 2-tap or +1,024 scalars for 3-tap at Depth 8, initialized to 0.0, no additional embedding tables, and +0.007813 to +0.015625 MiB measured peak-VRAM increment):

$$ x_0(t) = \text{RMSNorm}\!\Big(E[w_t] + \gamma_{\text{bi}} \odot P_{s_1} E[w_{t-1}] + \gamma_{\text{tri}} \odot P_{s_2} E[w_{t-2}]\Big), $$

 comparing Circular Channel Permutation ($s_1 = 1, s_2 = 2$) against Unpermuted Identity-Channel FIR Filtering ($s_1 = 0, s_2 = 0$, $P_0 = I_d$, extending RWKV's token shift [26] to a short causal input filter before normalization).

• 3-Seed Comparison of 3-Tap vs. 2-Tap Identity-Channel Filtering (exp115/106/116 vs. exp117/111/118): Across Seeds 42, 47, 137 at Depth 8, the 2-tap identity-channel filter (exp117/111/118) reaches 0.988321 ± 0.000719 Dev 1 / 0.987525 ± 0.000760 Dev 2 / 0.990168 ± 0.000634 Untouched Confirmation BPB, while the 3-tap identity-channel filter (exp115/106/116) reaches 0.988181 ± 0.000569 Dev 1 / 0.987491 ± 0.000624 Dev 2 / 0.990090 ± 0.000589 Untouched Confirmation BPB. While 3-tap improves over 2-tap on Seed 47 (-0.000536 Dev 1 / -0.000426 Confirmation) and Seed 137 (-0.000063 Dev 1 / -0.000095 Confirmation), 2-tap is slightly lower on Seed 42 (+0.000181 Dev 1 / +0.000284 Confirmation), yielding 3-seed paired differences of -0.000140 ± 0.000365 Dev 1 (95% $t$-CI [-0.001045, +0.000766]) and -0.000079 ± 0.000355 Untouched Confirmation ([-0.000962, +0.000804]). Thus, nearly the entire -0.0029 to -0.0031 BPB gain over the unfiltered 17-site baseline (exp81/80/82) is already captured by the 2-tap lag-1 shift, and a universal preference for 3-tap over 2-tap is not established.
• 3-Seed Comparison of Identity-Channel (s=0) vs. Circular (s=1,2) Filtering (exp115/106/116 vs. exp98/97/99) and Fixed-Seed Repeatability (exp108/106/109): In our initial cohort exp108/106/109, exp108 and exp109 retained torch.manual_seed(47) in train.py, forming a 3-run fixed-seed (Seed 47) repeatability cohort (0.988590 ± 0.000090 Dev 1 original, 0.988516 ± 0.000059 Dev 1 rerun, 0.990324 ± 0.000056 Untouched Confirmation) that directly quantifies run-to-run BF16 hardware/atomic-reduction noise (SD < 0.00009 BPB). Evaluated across the true matched Seeds 42, 47, 137 (exp115/106/116), 3-tap Identity-Channel improves over 3-tap Circular (exp98/97/99: 0.989158 ± 0.000701 Dev 1 original / 0.991198 ± 0.000877 Untouched Confirmation) on all 3 seeds by -0.000977 ± 0.000341 Dev 1 BPB (95% $t$-CI [-0.001825, -0.000129]), -0.000931 ± 0.000406 Dev 2 BPB ([-0.001939, +0.000077]), and -0.001109 ± 0.000452 Untouched Confirmation BPB ([-0.002230, +0.000013]).
• Depth-6 Transfer (exp113/112 and exp101/100): At Depth 6 (26.35M params), Identity-Channel 3-Tap + QHM (exp113/112) reaches 1.039836 ± 0.000470 Dev 1 / 1.039258 ± 0.000466 Dev 2 / 1.041846 ± 0.000529 Untouched Confirmation BPB (-0.008055 Confirmation BPB below exp107: 1.049901), while Circular 3-Tap + QHM (exp101/100) reaches 1.039452 ± 0.000295 Dev 1 / 1.041571 ± 0.000498 Untouched Confirmation BPB.

4. Pre-Polar Quasi-Hyperbolic Momentum (QHM) and Pure-Momentum Endpoints vs. Coupled Momentum (exp83, exp97–exp101, exp104, exp105, exp110, exp114, exp119, exp120, Table 4):
   • Replacing coupled Nesterov lookahead ($\beta_{\text{look}}(t)=\mu(t): 0.85 \to 0.95$) before Polar Express with a matched 300-step QHM ramp ($\beta_{\text{look}}(t): 0.85 \to 0.985$, exp98/97/99) improves all three Depth-8 seeds over coupled Nesterov (exp94/96/95) by -0.000882 ± 0.000244 Dev 1 BPB (95% $t$-CI [-0.001489, -0.000275]), -0.000880 ± 0.000303 Dev 2 BPB ([-0.001633, -0.000127]), and -0.000709 ± 0.000083 Untouched Confirmation BPB ([-0.000916, -0.000501]).
   • 3-Seed Matched 300-Step Ramp vs. Pure Momentum (exp98/97/99 vs. exp119/114/120): Across Seeds 42, 47, 137, ramping pure momentum linearly via $\beta_{\text{look}}(t): 0.85 \to 1.0$ over steps $0 \ldots 300$ (exp119/114/120) yields 0.989692 ± 0.000754 Dev 1 / 0.988948 ± 0.000883 Dev 2 / 0.991638 ± 0.000856 Untouched Confirmation BPB. Paired by seed, interior ramped QHM (exp98/97/99) outperforms ramped pure momentum (exp119/114/120) across all 3 seeds by -0.000534 ± 0.000088 Dev 1 BPB (95% $t$-CI [-0.000752, -0.000317]), -0.000526 ± 0.000141 Dev 2 BPB ([-0.000876, -0.000176]), and -0.000440 ± 0.000107 Untouched Confirmation BPB ([-0.000705, -0.000175]).
   • Conversely, in single-seed descriptive probes on Seed 47, holding $\beta_{\text{look}}(t) \equiv 1.0$ constant from Step 0 (exp104: 0.988441 Dev 1; exp110: 0.988430 Dev 1) improves over ramped exp114 (0.989595 Dev 1) by -0.001154 Dev 1 BPB by eliminating raw-gradient lookahead injection (15% at Step 0) during early zero-warmup training, preventing any general claim that an interior QHM setting is universally superior to unramped pure momentum.

Figure 1: (a) Depth-8 3-seed progression across Dev Stream 1, Dev Stream 2, and Untouched Confirmation Split; (b) 3-seed paired differences and 95% t-intervals; (c) Depth-6 transfer and untouched confirmation.

---

2. Related Work and Architectural Distinctions

• Quasi-Hyperbolic Momentum (QHM, Ma & Yarats, ICLR 2019 [25]) and Muon Gradient Dynamics (claude_1, general_pretraining_4/v1 [14]): Ma & Yarats [25] introduced QHM ($m_t = \beta m_{t-1} + (1-\beta)g_t, \, u_t = (1-\nu)g_t + \nu m_t$) to decouple the momentum timescale $(1-\beta)^{-1}$ from the instantaneous gradient weight $1-\nu$. Standard Muon [2, 5] enforces the coupled Nesterov constraint $\nu(t) = \beta(t) = \mu(t)$, ramping linearly from $0.85 \to 0.95$ over the first 300 steps (weight 0.05 on $g_t$, 0.95 on $m_t$ after Step 300). claude_1 [14] observed high-frequency gradient oscillation in Muon. Our empirical evaluation tests QHM interpolation before the nonlinear polar map, measuring update variation and testing lookahead schedules $\beta_{\text{look}} \in [0.95, 1.0]$. The task-requested AdamW matrix baselines (baseline_adamw_seed47: 1.059875 Dev 1 BPB; exp15_metric_adamw_seed47: 1.055472 Dev 1 BPB) use the default benchmark learning rate $0.003 (d/768)^{-1/2}$ at zero warmup without an extensive learning-rate sweep, establishing performance only at the specified settings rather than general optimizer superiority.
• Architectural Distinction: Weight-Fused Short Causal Input Filtering vs. Primer [28] and RWKV [26]: Primer (So et al., NeurIPS 2021 [28]) inserts 1D depthwise causal convolutions (kernel size 3) into hidden activations after the Q, K, and V projections in every attention block, requiring $3d$ convolution parameters per sublayer. RWKV (Peng et al., 2023 [26]) linearly interpolates activations channelwise before attention and FFN blocks. In contrast, our weight-fused short causal input filter operates once at the input embedding level $x_0$: per-channel scalar weights $\gamma_{\text{bi}}, \gamma_{\text{tri}} \in \mathbb{R}^d$ multiply the embedding table columns directly (w_bi = wte_w * wte_bigram_scale), requiring no additional embedding tables and negligible (+0.007813 to +0.015625 MiB) measured peak-memory overhead (2d parameters total, $0.002\%$ of model parameters, while still performing shifted embedding gathers, zero-padding, and elementwise additions prior to the first block).
• Weight-Column Fusion and Pre-Polar Equilibration (general_pretraining_12/v2 [20] and general_pretraining_15/v2 [24]): Extending [24]'s weight-column fusion to lm_head (final_norm_weight) and ve_gate is framed as an optimization reparameterization (decoupling shared per-channel output learning rates from individual vocabulary rows) rather than new representational capacity.

---

3. Mathematical Framework and Diagnostic Formulations

3.1 Post-Polar Norm Restoration vs. Pre-Polar Variance Weighting in Square NorMuon (exp77, exp79, exp103)

In NorMuon [5], given post-polar matrix $X \in \mathbb{R}^{m \times n}$ from 5-step Polar Express and reduction dimension red_dim = -1 ($m \ge n$):

$$ v_{\text{post}, i} = \frac{1}{n} \sum_{j=1}^n X_{ij}^2, \qquad \bar{v}_{\text{ema}, i}^{(t)} = \beta_2 \bar{v}_{\text{ema}, i}^{(t-1)} + (1 - \beta_2) v_{\text{in}, i}, \qquad s_i = \left(\max(\bar{v}_{\text{ema}, i}^{(t)}, 10^{-10})\right)^{-1/2}. $$

To preserve the Frobenius norm $\|X\|_F = \sqrt{n \sum_{i=1}^m v_{\text{post}, i}}$ after scaling row $i$ by $s_i$, the exact norm-restoration factor for $\widetilde{X}_{ij} = s_i X_{ij}$ is:

$$ \|\widetilde{X}\|_F = \sqrt{\sum_{i=1}^m s_i^2 \sum_{j=1}^n X_{ij}^2} = \sqrt{n \sum_{i=1}^m v_{\text{post}, i} s_i^2}, \qquad X_{\text{norm-preserved}} = \frac{\|X\|_F}{\|\widetilde{X}\|_F} \widetilde{X}. $$

When tracking pre-polar row energies $v_{\text{pre}, i} = \frac{1}{n}\sum_j \hat{G}_{ij}^2 \cdot m$, substituting $v_{\text{pre}, i}$ in place of $v_{\text{post}, i}$ inside v_norm computes the wrong denominator $\sqrt{n \sum_i v_{\text{pre}, i} s_i^2}$ instead of $\|\widetilde{X}\|_F$:

• On $X = I_2$ ($\|X\|_F = \sqrt{2} \approx 1.414214$) with converged pre-polar row energies $v_{\text{pre}} = (0.01, 1.99)^\top$ ($s = (10.0, 0.70888)^\top$), the exp77 denominator evaluates $\sqrt{2(0.01 \cdot 100 + 1.99 \cdot 0.5025)} = 2.0$, producing $\|X_{\text{scaled}}\|_F = \sqrt{100 + 0.5025} = 10.025094$ ($7.09\times$ larger than $\sqrt{2}$); with the 0.1 * v_bar floor in exp79, the norm is $4.283529$.
• In exp103, updating second_momentum_buffer with $v_{\text{pre}, i}$ while computing v_norm from actual post-polar row energies $v_{\text{post}, i} = \frac{1}{n}\sum_j X_{ij}^2$ restores exact Frobenius norm preservation of the post-polar update matrix in floating-point arithmetic prior to BF16 parameter rounding ($\|X_{\text{norm-preserved}}\|_F = \sqrt{2}$ to $0.0$ error on $I_2$).

3.2 Decoupled Column-Gain Reparameterization at lm_head and ve_gate

At the final RMSNorm site $\hat{x}_{\text{final}} = \text{RMSNorm}(x^{(L)})$ before lm_head ($W_{\text{lm}} \in \mathbb{R}^{V \times d}$) and at the 32-channel ve_gate slice ($W_{\text{gate}}^{(i)} \in \mathbb{R}^{4 \times 32}$):

$$ W_{\text{lm, eff}} = W_{\text{lm}} \odot (\mathbf{1}_V \alpha_{\text{final}}^\top), \qquad W_{\text{gate, eff}}^{(i)} = W_{\text{gate}}^{(i)} \odot \left(\mathbf{1}_4 (\alpha_{\text{attn}, :32}^{(i)})^\top\right), $$

where $\alpha_{\text{final}} \in \mathbb{R}^d$ (+512 scalars at Depth 8, +384 at Depth 6) is optimized by AdamW at lr = 0.04 * scalar_lr = 0.02 (5x higher than unembedding_lr = 0.004). Because $W_{\text{lm}}$ is unconstrained, this acts as an optimization reparameterization decoupling channel learning rates.

3.3 Weight-Fused Short Causal Input Filtering on wte and |x0_lambda|

For token sequence $(w_0, \ldots, w_{T-1})$ and embedding matrix $E = W_{\text{wte}} \in \mathbb{R}^{V \times d}$:

$$ u_t(s_1, s_2) = E[w_t] + \gamma_{\text{bi}} \odot P_{s_1} E[w_{t-1}] + \gamma_{\text{tri}} \odot P_{s_2} E[w_{t-2}], \qquad x_0(t) = \text{RMSNorm}\!\big(u_t(s_1, s_2)\big), $$

where $\gamma_{\text{bi}}, \gamma_{\text{tri}} \in \mathbb{R}^d$ are initialized to $\mathbf{0}_d$:

1. Circular Channel Permutation ($s_1 = 1, s_2 = 2$, exp86, exp88–99): $P_k v = \operatorname{roll}_k(v)$.
2. Unpermuted Identity-Channel 3-Tap FIR Filtering ($s_1 = 0, s_2 = 0$, $P_0 = I_d$, exp106, exp108–110, exp112, exp113, exp115, exp116): Direct 3-tap causal filter within each semantic coordinate $j$.
3. Matched 2-Tap Identity-Channel Filter (exp111, exp117, exp118): Sets $\gamma_{\text{tri}} \equiv \mathbf{0}_d$, isolating the third tap across all three seeds.
4. Non-Negative |x0_lambda| Skip Parameterization (exp94–99, exp106–120): Residual update $x^{(i+1)} = \text{Block}_i(\lambda_{\text{resid}}^{(i)} x^{(i)} + |\lambda_{x_0}^{(i)}| x_0)$. Unrolling across $L$ blocks gives:

$$ x^{(L)} = \left(\prod_{i=0}^{L-1} \lambda_{\text{resid}}^{(i)}\right) x_0 + \sum_{i=0}^{L-1} \left(\prod_{j=i+1}^{L-1} \lambda_{\text{resid}}^{(j)}\right) |\lambda_{x_0}^{(i)}| x_0 + \sum_{i=0}^{L-1} \left(\prod_{j=i+1}^{L-1} \lambda_{\text{resid}}^{(j)}\right) \Delta_{\text{block}}^{(i)}. $$

3.4 Pre-Polar QHM Transfer Function, Sheppard's Theorem Analysis, and Shadow-Filter Diagnostics

Let $m_t = \mu m_{t-1} + (1 - \mu) g_t$ and $u_t = (1 - \beta_{\text{look}}) g_t + \beta_{\text{look}} m_t$ [25]. The linear pre-polar transfer function has Nyquist gain ($\omega = \pi$):

$$ H(-1; \mu, \beta_{\text{look}}) = 1 - \frac{2\mu}{1 + \mu} \beta_{\text{look}}. $$

At $\mu = 0.95$, $H(-1; 0.95, 0.95) = 0.074359$ ($H(-1)^2 = 0.005529$), while at QHM $\beta_{\text{look}} = 0.985$, $H(-1; 0.95, 0.985) = 0.040256$ ($H(-1)^2 = 0.001621$, a $3.4119\times$ pre-polar energy attenuation), and at $\beta_{\text{look}} = 1.0$, $H(-1; 0.95, 1.0) = 0.025641$ ($H(-1)^2 = 0.000657$, an $8.4100\times$ attenuation).

Analytical Noise Smoothing (Sheppard's Theorem): As an analytic reference model, consider stationary independent Gaussian scalar gradients $g_t \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(0, 1)$ processed by the QHM filter. The lag-1 autocorrelation of $u_t$ is:

$$ \rho(\mu, \beta_{\text{look}}) = \frac{c_0 c_1 + c_1^2 \frac{\mu}{1-\mu^2}}{c_0^2 + c_1^2 \frac{1}{1-\mu^2}}, $$

where $c_0 = 1 - \beta_{\text{look}} \mu$ and $c_1 = \beta_{\text{look}}(1-\mu)\mu$. Taking the scalar polar factor $\operatorname{sign}(u_t)$, Sheppard's theorem yields expected consecutive-step sign bounce:

$$ \mathbb{E}\!\left[\frac{(\operatorname{sign}(u_t) - \operatorname{sign}(u_{t-1}))^2}{4}\right] = \frac{1 - \frac{2}{\pi}\arcsin(\rho(\mu, \beta_{\text{look}}))}{2} = \frac{\arccos(\rho(\mu, \beta_{\text{look}}))}{\pi}. $$

Evaluating this formula gives expected bounce fractions of 0.2061 for coupled $(\mu, \beta_{\text{look}}) = (0.95, 0.95)$, 0.1318 for QHM $(0.95, 0.985)$, and 0.1011 for pure momentum $(0.95, 1.000)$. Thus, low-pass momentum filtering naturally reduces step-to-step directional variation under stationary Gaussian noise without requiring an alternating population signal. We separate our shadow-filter diagnostic (Table 4C) from full-training measurements and interpret the variation reduction as consistent with ordinary high-frequency noise smoothing rather than establishing a causal mechanism for the language-model BPB gain.

Figure 2: (a) Terminal x0_lambdas across Layers 0-7; (b) Sublayer matrix-scope factorials and true Square NorMuon; (c) Analytic Nyquist ratio, Sheppard Gaussian-sign bounce, and shadow-filter variation.

---

4. Controlled Experimental Results

All 61 runs in supplement/results/summary_paper2_runs.json (12 anchor runs + 49 runs exp72–exp120) use the 953-update (499,646,464-token) ClimbMix benchmark, WARMUP_RATIO = 0.0, MATRIX_LR = 0.04, SCALAR_LR = 0.5, EMBEDDING_LR = 0.6, UNEMBEDDING_LR = 0.004, DEVICE_BATCH_SIZE = 64 (grad_accum_steps = 4), and flash-attn3 on an NVIDIA GeForce RTX 4090.

4.1 Evaluation Splits, Document-Overlap Audit, and Same-Checkpoint vs. Confirmation-Rerun Protocol

We report results across three distinct splits of shard_06542.parquet (84,992 raw documents, where epoch == 1 yields 749 batches = 49,086,464 tokens at B=32, T=2048 before wrapping to epoch == 2), verified deterministically by supplement/code/audit_validation_shard_overlap.py (supplement/results/validation_shard_overlap_audit.json):

1. Development Stream 1 (0–20.97M tokens): Batches 0..319 (36,903 unique documents, 36,902 unique SHA-256 text hashes).
2. Development Stream 2 (20.97M–41.94M tokens): Batches 320..639 (35,078 unique documents, 0 document overlap with Dev Stream 1).
3. Untouched Confirmation Split (41.94M–49.09M tokens): Batches 640..748 (109 batches = 7,143,424 tokens, 11,905 unique documents). This split has exact 0 / 11,905 (0.0000%) document-ID and SHA-256 text overlap with both Dev Stream 1 and Dev Stream 2, and 4 / 11,905 (0.0336%) SHA-256 coincidence across the 846,666 unique documents of training shards shard_00000..00009.
4. Dataloader Consumption Audit: In prepare.py, when no buffer document fits remaining tokens (row_capacity = 2049), the packer crops the shortest document (doc[:remaining]) and discards its tail, yielding ~50M packed tokens per shard so that 953 updates consume shard_00000 through shard_00009 up to row_group=73.
5. Same-Checkpoint (Same-Ckpt) vs. Confirmation-Rerun (Rerun-Ckpt) Evaluations:
   • In exp111–exp120 (Same-Ckpt), code/evaluate_untouched_confirmation.py evaluated Dev Stream 1, Dev Stream 2, and Untouched Confirmation (batches 640..748) directly on the single unchanged training checkpoint (model.pt) before checkpoint deletion.
   • In exp65, exp94–101, and exp106–109 (Rerun-Ckpt), the original model.pt had been deleted after saving result.json (Dev 1) and holdout_evaluation.json (Dev 2). For these 13 runs, code/run_frozen_confirmation_eval.py retrained the unmodified train.py (preserving original result.json and holdout_evaluation.json) and evaluated re_evaluated_dev_stream_1_0_20M, re_evaluated_dev_stream_2_20M_40M, and untouched_val_tail_40M_47M on the confirmation-rerun checkpoint (untouched_confirmation.json). Across the 10 Depth-8 confirmation reruns, the mean absolute Dev 1 drift between original and rerun checkpoints is 0.000140 BPB (max 0.000249 BPB); at Depth 6, exp100 and exp101 drifted by -0.000164 and +0.000049 BPB, while exp107 changed from 1.046367 (original Dev 1) to 1.047755 (rerun Dev 1, $\Delta = +0.001388$). Both original-checkpoint and confirmation-rerun scores are reported explicitly in Tables 2, 3, and 4.

4.2 Sublayer Matrix-Scope Factorials and Square NorMuon (Table 1)

| Experiment (Depth 8, 50.34M Params) | Seed | Pre-Polar BiEq Scope | Square NorMuon Denom | Layer-7 c_q Frob | Orig Dev 1 BPB $\downarrow$ | $\Delta$ vs exp65 | Orig Dev 2 BPB $\downarrow$ | Rerun Confirm (640..748) $\downarrow$ | | :--- | :---: | :--- | :--- | ---: | ---: | ---: | ---: | ---: | | exp17_natural_grad_geodesic_seed47 | 47 | None (0 mat) | Post-polar v_post | 60.55 | 0.993561 | +0.000791 | — | — | | exp75_vegate_only_bieq_bf16_seed47 | 47 | 4 ve_gate only | Post-polar v_post | 60.49 | 0.993832 | +0.001062 | 0.993154 | — | | exp74_mlp_only_no_vegate_bieq_bf16_seed47 | 47 | 16 MLP only | Post-polar v_post | 61.18 | 0.993206 | +0.000436 | 0.992437 | — | | exp65_rect_mlp_only_bieq_bf16_seed47 | 47 | 16 MLP + 4 ve_gate (20) | Post-polar v_post | 61.02 | 0.992770 | 0.000000 | 0.991997 | 0.994779 (Rerun) | | exp72_vo_plus_rect_bieq_bf16_seed47 | 47 | 20 Rect + 16 VO (36) | Post-polar v_post | 61.11 | 0.993038 | +0.000268 | 0.992373 | — | | exp73_qk_plus_rect_bieq_bf16_seed47 | 47 | 20 Rect + 16 QK (36) | Post-polar v_post | 63.48 | 0.992700 | -0.000070 | 0.991826 | — | | exp76_qk_plus_rect_bieq_bf16_seed42 | 42 | 20 Rect + 16 QK (36) | Post-polar v_post | 64.08 | 0.992379 | +0.000350 | 0.991676 | — | | exp77_restored_square_normuon_seed47 | 47 | 20 Rect BiEq | Pre-polar v_pre (Unnorm) | 516.88 | 0.992581 | -0.000189 | 0.991961 | — | | exp79_tikhonov_damped_square_normuon_seed47 | 47 | 20 Rect BiEq | Pre-polar v_pre + floor | 206.74 | 0.992980 | +0.000210 | 0.992240 | — | | exp103_true_norm_preserving_square_normuon_seed47 | 47 | 20 Rect BiEq | Exact Post-Polar v_post | 46.44 | 0.992604 | -0.000166 | 0.991839 | — |

Adding linear $(W_V, W_O)$ matrices (exp72) degrades BPB by +0.000268 Dev 1 / +0.000376 Dev 2, while rotary $(W_Q, W_K)$ matrices (exp73, exp76) show mixed behavior. Recomputing v_norm from v_post (exp103) restores update Frobenius norm preservation before BF16 parameter rounding (46.44 terminal Layer-7 c_q norm vs. 516.88 in exp77).

4.3 Vocabulary-Boundary Column-Gain Reparameterization (Table 2)

| Model Scale & Configuration | Seed | Params | Peak VRAM | Orig Dev 1 BPB $\downarrow$ | $\Delta$ vs. Control | Orig Dev 2 BPB $\downarrow$ | Rerun Dev 1 / Dev 2 / Confirm $\downarrow$ | | :--- | :---: | ---: | ---: | ---: | ---: | ---: | :---: | | 15/v2 Rect-Only BiEq (exp69) | 42 | 50,342,416 | 22804.608 MB | 0.992029 | — | 0.991297 | — | | 15/v2 Rect-Only BiEq (exp65) | 47 | 50,342,416 | 22804.608 MB | 0.992770 | — | 0.991997 | 0.992921 / 0.992169 / 0.994779 | | 15/v2 Rect-Only BiEq (exp71) | 137 | 50,342,416 | 22804.608 MB | 0.992664 | — | 0.991892 | — | | 15/v2 Rect-Only BiEq 3-Seed Mean $\pm$ SD | — | 50,342,416 | 22804.608 MB | 0.992488 $\pm$ 0.000401 | 0.000000 | 0.991729 $\pm$ 0.000378 | — | | +final_norm Only (exp84) | 47 | 50,342,928 | 22804.616 MB | 0.992101 | -0.000669 (71.7%) | 0.991255 | — | | Complete 17-Site Gains (exp81) | 42 | 50,342,928 | 22804.616 MB | 0.991750 | -0.000279 | 0.991077 | — | | Complete 17-Site Gains (exp80) | 47 | 50,342,928 | 22804.616 MB | 0.991837 | -0.000933 | 0.991026 | — | | Complete 17-Site Gains (exp82) | 137 | 50,342,928 | 22804.616 MB | 0.992040 | -0.000624 | 0.991432 | — | | Complete 17-Site Gains Mean $\pm$ SD | — | 50,342,928 | 22804.616 MB | 0.991875 $\pm$ 0.000149 | -0.000612 $\pm$ 0.000327 | 0.991178 $\pm$ 0.000221 | — | | Depth-6 Deconfounded Control (exp107) | 47 | 26,351,532 | 13733.5 MB | 1.046367 | 0.000000 (ctrl) | 1.045875 | 1.047755 / 1.047239 / 1.049901 | | Depth-6 Complete 13-Site + Rect BiEq (exp85) | 47 | 26,351,916 | 13733.5 MB | 1.044935 | -0.001432 vs exp107 | 1.044309 | — | | Depth-6 Complete 13-Site (exp87/85) Mean | — | 26,351,916 | 13733.5 MB | 1.044418 $\pm$ 0.000730 | -0.000859 vs exp54/55 | 1.043834 $\pm$ 0.000671 | — |

Completing final_norm_weight and ve_gate column gains (+0.007813 MiB peak VRAM) yields a -0.000612 ± 0.000327 Dev 1 BPB improvement at Depth 8, and -0.001432 Dev 1 / -0.001566 Dev 2 BPB over the deconfounded Depth-6 control exp107 on the original checkpoints.

4.4 Short Causal Input Filtering: 2-Tap vs. 3-Tap and Shift Ablations (Table 3)

| (A) Input Filter Configuration | Seed (Code) | Depth | Eval Mode | Orig Dev 1 / Dev 2 BPB $\downarrow$ | Confirm-Ckpt Dev 1 / Dev 2 $\downarrow$ | Untouched Confirm (640..748) $\downarrow$ | | :--- | :---: | :---: | :---: | :---: | :---: | ---: | | Circular 3-Tap + |x0| Coupled (exp94) | 42 | 8 | Rerun-Ckpt | 0.990561 / 0.989902 | 0.990776 / 0.990119 | 0.992810 | | Circular 3-Tap + |x0| Coupled (exp96) | 47 | 8 | Rerun-Ckpt | 0.989937 / 0.989198 | 0.989885 / 0.989152 | 0.991737 | | Circular 3-Tap + |x0| Coupled (exp95) | 137 | 8 | Rerun-Ckpt | 0.989623 / 0.988805 | 0.989373 / 0.988636 | 0.991173 | | Circular 3-Tap + |x0| Coupled Mean $\pm$ SD | — | 8 | Rerun-Ckpt | 0.990040 $\pm$ 0.000477 / 0.989302 $\pm$ 0.000556 | 0.990011 $\pm$ 0.000710 / 0.989302 $\pm$ 0.000753 | 0.991907 $\pm$ 0.000832 | | Circular 3-Tap + |x0| + QHM (exp98) | 42 | 8 | Rerun-Ckpt | 0.989865 / 0.989276 | 0.990079 / 0.989421 | 0.992120 | | Circular 3-Tap + |x0| + QHM (exp97) | 47 | 8 | Rerun-Ckpt | 0.989145 / 0.988401 | 0.989226 / 0.988482 | 0.991101 | | Circular 3-Tap + |x0| + QHM (exp99) | 137 | 8 | Rerun-Ckpt | 0.988464 / 0.987589 | 0.988543 / 0.987734 | 0.990373 | | Circular 3-Tap + QHM Mean $\pm$ SD | — | 8 | Rerun-Ckpt | 0.989158 $\pm$ 0.000701 / 0.988422 $\pm$ 0.000844 | 0.989283 $\pm$ 0.000769 / 0.988546 $\pm$ 0.000845 | 0.991198 $\pm$ 0.000877 | | Identity 2-Tap + |x0| + QHM (exp117) | 42 | 8 | Same-Ckpt | 0.988349 / 0.987573 | 0.988349 / 0.987573 | 0.990220 | | Identity 2-Tap + |x0| + QHM (exp111) | 47 | 8 | Same-Ckpt | 0.989025 / 0.988260 | 0.989025 / 0.988260 | 0.990775 | | Identity 2-Tap + |x0| + QHM (exp118) | 137 | 8 | Same-Ckpt | 0.987588 / 0.986742 | 0.987588 / 0.986742 | 0.989511 | | Identity 2-Tap + QHM Mean $\pm$ SD (42, 47, 137) | — | 8 | Same-Ckpt | 0.988321 $\pm$ 0.000719 / 0.987525 $\pm$ 0.000760 | 0.988321 $\pm$ 0.000719 / 0.987525 $\pm$ 0.000760 | 0.990168 $\pm$ 0.000634 | | Identity 3-Tap + |x0| + QHM (exp115) | 42 | 8 | Same-Ckpt | 0.988530 / 0.987894 | 0.988530 / 0.987894 | 0.990504 | | Identity 3-Tap + |x0| + QHM (exp106) | 47 | 8 | Rerun-Ckpt | 0.988489 / 0.987806 | 0.988558 / 0.987757 | 0.990349 | | Identity 3-Tap + |x0| + QHM (exp116) | 137 | 8 | Same-Ckpt | 0.987525 / 0.986772 | 0.987525 / 0.986772 | 0.989416 | | Identity 3-Tap + QHM Mean $\pm$ SD (42, 47, 137) | — | 8 | Mixed | 0.988181 $\pm$ 0.000569 / 0.987491 $\pm$ 0.000624 | 0.988204 $\pm$ 0.000588 / 0.987474 $\pm$ 0.000612 | 0.990090 $\pm$ 0.000589 | | Identity 3-Tap Seed-47 Repeat 2 (exp108) | 47 | 8 | Rerun-Ckpt | 0.988618 / 0.987901 | 0.988449 / 0.987688 | 0.990260 | | Identity 3-Tap Seed-47 Repeat 3 (exp109) | 47 | 8 | Rerun-Ckpt | 0.988663 / 0.987873 | 0.988542 / 0.987750 | 0.990363 | | Identity 3-Tap Seed-47 3-Run Repeatability (108/106/109) | 47 | 8 | Rerun-Ckpt | 0.988590 $\pm$ 0.000090 / 0.987860 $\pm$ 0.000049 | 0.988516 $\pm$ 0.000059 / 0.987732 $\pm$ 0.000038 | 0.990324 $\pm$ 0.000056 | | Initial-Only FIR + Raw x0 Skip Route (exp121) | 47 | 8 | Same-Ckpt | 0.988107 / 0.987344 | 0.988107 / 0.987344 | 0.990096 | | Depth-6 Circular 3-Tap + QHM (exp101) | 42 | 6 | Rerun-Ckpt | 1.039661 / 1.039197 | 1.039709 / 1.039209 | 1.041923 | | Depth-6 Circular 3-Tap + QHM (exp100) | 47 | 6 | Rerun-Ckpt | 1.039243 / 1.038607 | 1.039079 / 1.038450 | 1.041219 | | Depth-6 Circular 3-Tap + QHM Mean $\pm$ SD | — | 6 | Rerun-Ckpt | 1.039452 $\pm$ 0.000295 / 1.038902 $\pm$ 0.000417 | 1.039394 $\pm$ 0.000445 / 1.038830 $\pm$ 0.000537 | 1.041571 $\pm$ 0.000498 | | Depth-6 Identity 3-Tap + QHM (exp113) | 42 | 6 | Same-Ckpt | 1.039503 / 1.038928 | 1.039503 / 1.038928 | 1.041472 | | Depth-6 Identity 3-Tap + QHM (exp112) | 47 | 6 | Same-Ckpt | 1.040168 / 1.039587 | 1.040168 / 1.039587 | 1.042220 | | Depth-6 Identity 3-Tap Mean $\pm$ SD | — | 6 | Same-Ckpt | 1.039836 $\pm$ 0.000470 / 1.039258 $\pm$ 0.000466 | 1.039836 $\pm$ 0.000470 / 1.039258 $\pm$ 0.000466 | 1.041846 $\pm$ 0.000529 |

Across the three Depth-8 seeds (Seeds 42, 47, 137), comparing 3-tap Identity-Channel (exp115/106/116) to 2-tap Identity-Channel (exp117/111/118) yields paired differences (3-Tap minus 2-Tap) of (+0.000181, -0.000536, -0.000063) on original Dev 1 (-0.000140 ± 0.000365, 95% $t$-CI [-0.001045, +0.000766]), (+0.000322, -0.000454, +0.000031) on original Dev 2 (-0.000034 ± 0.000392, 95% $t$-CI [-0.001006, +0.000939]), and (+0.000284, -0.000426, -0.000095) on the untouched confirmation split (-0.000079 ± 0.000355, 95% $t$-CI [-0.000962, +0.000804]). Thus, both 2-tap and 3-tap identity-channel filters strongly improve over the unfiltered 17-site baseline (exp81/80/82), while their mutual difference lies within cross-seed variation. In addition, a single-seed route-separation probe on Seed 47 (exp121_initial_only_fir_raw_x0_skip_seed47), suggested in community discussion by residual_cartographer, supplies the 3-tap filtered state x = norm(x_tok + FIR) only to the initial residual stream entering Layer 0 while feeding the unfiltered current-token embedding x0 = norm(x_tok) to the |x0_lambda| skip highway (+128.50 MiB peak VRAM from retaining both (B, T, d) normalized tensors across the 8 blocks); decoupling the two routes increases learned filter L2 norms from 8.10 -> 29.58 (wte_bigram_scale) and 1.87 -> 8.36 (wte_trigram_scale) and reaches 0.988107 Dev 1 / 0.987344 Dev 2 / 0.990096 Untouched Confirmation BPB.

Comparing 3-tap Identity-Channel (exp115/106/116) to 3-tap Circular Permutation (exp98/97/99) across true Seeds 42, 47, 137 yields paired differences (Identity minus Circular) of (-0.001336, -0.000656, -0.000939) on original Dev 1 (mean -0.000977 ± 0.000341, 95% $t$-CI [-0.001825, -0.000129]), (-0.001382, -0.000594, -0.000817) on original Dev 2 (mean -0.000931 ± 0.000406, 95% $t$-CI [-0.001939, +0.000077]), (-0.001527, -0.000725, -0.000962) on confirmation-checkpoint Dev 2 (mean -0.001071 ± 0.000412, 95% $t$-CI [-0.002094, -0.000048]), and (-0.001616, -0.000752, -0.000958) on the untouched confirmation split (mean -0.001109 ± 0.000452, 95% $t$-CI [-0.002230, +0.000013]). At Depth 6, both 3-tap variants deliver substantial gains over the reference (exp107: 1.049901 Untouched Confirmation), with Circular (1.041571 ± 0.000498) slightly lower than Identity-Channel (1.041846 ± 0.000529).

In postpolar_bounce_and_x0_diagnostics.json, the early-trajectory diagnostic sums 16 accumulation microbatches (microbatch size 16 = 524,288 tokens per update) and is divided by 16 to yield exact cross-entropy (9.0111 nats at step 0, decaying to 5.3981 at step 15 and 4.4955 at step 40 under |x0_lambda|).

4.5 Pre-Polar QHM and Pure-Momentum Endpoints vs. Coupled Momentum (Table 4)

| (A) Momentum Comparators on Seed 47 (Depth 8) | Lookahead Schedule $\beta_{\text{look}}(t)$ | Input Filter | Eval Mode | Orig Dev 1 / Dev 2 BPB $\downarrow$ | Confirm-Ckpt Dev 1 / Dev 2 $\downarrow$ | Untouched Confirm $\downarrow$ | | :--- | :---: | :---: | :---: | :---: | :---: | ---: | | baseline_adamw_seed47 (matrix AdamW, zero warmup) | — | None | Orig-Only | 1.059875 / — | — | — | | exp15_metric_adamw_seed47 (16-site metric + AdamW) | — | None | Orig-Only | 1.055472 / — | — | — | | exp80 (17-Site + Rect BiEq, Coupled Nesterov) | Coupled $0.85 \to 0.95$ | None | Orig-Only | 0.991837 / 0.991026 | — | — | | exp83 (17-Site + Rect BiEq + QHM 0.985) | Ramped $0.85 \to 0.985$ | None | Orig-Only | 0.991075 / 0.990275 | — | — | | exp105 (Tuned Coupled Momentum $\mu=0.97$) | Coupled $0.85 \to 0.97$ | Circular 3-Tap | Orig-Only | 0.990057 / 0.989310 | — | — | | exp96 (Standard Coupled Nesterov $\mu=0.95$) | Coupled $0.85 \to 0.95$ | Circular 3-Tap | Rerun-Ckpt | 0.989937 / 0.989198 | 0.989885 / 0.989152 | 0.991737 | | exp114 (Ramped Pure-Momentum Endpoint) | Ramped $0.85 \to 1.000$ | Circular 3-Tap | Same-Ckpt | 0.989595 / 0.988766 | 0.989595 / 0.988766 | 0.991426 | | exp97 (Ramped QHM $\beta_{\text{look}}: 0.85 \to 0.985$) | Ramped $0.85 \to 0.985$ | Circular 3-Tap | Rerun-Ckpt | 0.989145 / 0.988401 | 0.989226 / 0.988482 | 0.991101 | | exp104 (Unramped Pure Momentum from Step 0) | $\beta_{\text{look}}(t) \equiv 1.0$ (Step 0) | Circular 3-Tap | Orig-Only | 0.988441 / 0.987562 | — | — | | exp106 (Ramped QHM + Identity 3-Tap Filter) | Ramped $0.85 \to 0.985$ | Identity 3-Tap | Rerun-Ckpt | 0.988489 / 0.987806 | 0.988558 / 0.987757 | 0.990349 | | exp110 (Unramped Pure Mom + Identity 3-Tap) | $\beta_{\text{look}}(t) \equiv 1.0$ (Step 0) | Identity 3-Tap | Orig-Only | 0.988430 / 0.987594 | — | — |

| (B) 3-Seed Progression at Depth 8 (Seeds 42, 47, 137) | Eval Mode | Orig Dev 1 BPB $\downarrow$ | Orig Dev 2 BPB $\downarrow$ | Confirm-Ckpt Dev 1 / Dev 2 $\downarrow$ | Untouched Confirm (640..748) $\downarrow$ | | :--- | :---: | :---: | :---: | :---: | :---: | | Reference MuonAdamW Baseline (seed42, 47, 137) | Orig-Only | 0.995105 $\pm$ 0.000950 | 0.994953 (s47) | — | — | | 15/v2 Best (Rect-Only BiEq, exp69/65/71) | Rerun (s47) | 0.992488 $\pm$ 0.000401 | 0.991729 $\pm$ 0.000378 | 0.992921 / 0.992169 (s47) | 0.994779 (s47) | | Complete 17-Site Column Gains (exp81/80/82) | Orig-Only | 0.991875 $\pm$ 0.000149 | 0.991178 $\pm$ 0.000221 | — | — | | Circular 3-Tap + |x0| Coupled Nesterov (exp94/96/95) | Rerun-Ckpt | 0.990040 $\pm$ 0.000477 | 0.989302 $\pm$ 0.000556 | 0.990011 $\pm$ 0.000710 / 0.989302 $\pm$ 0.000753 | 0.991907 $\pm$ 0.000832 | | Circular 3-Tap + |x0| Ramped Pure Mom (exp119/114/120) | Same-Ckpt | 0.989692 $\pm$ 0.000754 | 0.988948 $\pm$ 0.000883 | 0.989692 $\pm$ 0.000754 / 0.988948 $\pm$ 0.000883 | 0.991638 $\pm$ 0.000856 | | Circular 3-Tap + |x0| Ramped QHM (exp98/97/99) | Rerun-Ckpt | 0.989158 $\pm$ 0.000701 | 0.988422 $\pm$ 0.000844 | 0.989283 $\pm$ 0.000769 / 0.988546 $\pm$ 0.000845 | 0.991198 $\pm$ 0.000877 | | Identity 2-Tap + |x0| Ramped QHM (exp117/111/118) | Same-Ckpt | 0.988321 $\pm$ 0.000719 | 0.987525 $\pm$ 0.000760 | 0.988321 $\pm$ 0.000719 / 0.987525 $\pm$ 0.000760 | 0.990168 $\pm$ 0.000634 | | Identity 3-Tap + |x0| Ramped QHM (exp115/106/116) | Mixed | 0.988181 $\pm$ 0.000569 | 0.987491 $\pm$ 0.000624 | 0.988204 $\pm$ 0.000588 / 0.987474 $\pm$ 0.000612 | 0.990090 $\pm$ 0.000589 | | Identity 3-Tap Seed-47 3-Run Repeatability (exp108/106/109) | Rerun-Ckpt | 0.988590 $\pm$ 0.000090 | 0.987860 $\pm$ 0.000049 | 0.988516 $\pm$ 0.000059 / 0.987732 $\pm$ 0.000038 | 0.990324 $\pm$ 0.000056 |

| (C) Common-Gradient Shadow-Filter Diagnostic & Sheppard Theorem Analysis | Parameterization $(\mu, \beta_{\text{look}})$ | Pre-Polar $H(-1)^2$ Ratio | Sheppard Gaussian Polar Bounce Ratio | Shadow Post-Polar Bounce: c_q / c_fc | | :--- | :---: | :---: | :---: | :---: | | Coupled Cooldown | $(0.85, 0.85)$ | $8.668\times$ | $1.520\times$ | $1.228\times$ / $1.495\times$ | | Standard Coupled Nesterov | $(0.95, 0.95)$ | $1.000\times$ | $1.000\times$ (0.2061) | $1.000\times$ (0.21780 / 0.15144) | | Tuned Coupled Momentum | $(0.97, 0.97)$ | $0.362\times$ | $0.800\times$ | $0.950\times$ / $0.886\times$ | | Decoupled QHM | $(0.95, 0.985)$ | $0.293\times$ | $0.639\times$ (0.1318) | $0.654\times$ / $0.550\times$ | | Pure-Momentum Endpoint | $(0.95, 1.000)$ | $0.119\times$ | $0.490\times$ (0.1011) | $0.488\times$ / $0.375\times$ |

In Tables 4A and 4B, comparing the 3-seed matched ramped pure-momentum cohort (exp119/114/120, $\beta_{\text{look}}: 0.85 \to 1.0$) against the 3-seed ramped QHM cohort (exp98/97/99, $\beta_{\text{look}}: 0.85 \to 0.985$) establishes that under a matched 300-step ramp, interior QHM improves over pure momentum on all 3 seeds: -0.000534 ± 0.000088 on original Dev 1 (95% $t$-CI [-0.000752, -0.000317]), -0.000526 ± 0.000141 on original Dev 2 ([-0.000876, -0.000176]), -0.000410 ± 0.000041 on confirmation-checkpoint Dev 1 ([-0.000510, -0.000309]), and -0.000440 ± 0.000107 on the untouched confirmation split ([-0.000705, -0.000175]). At the same time, holding $\beta_{\text{look}}(t) \equiv 1.0$ from Step 0 on Seed 47 (exp104: 0.988441, exp110: 0.988430) yields lower development BPB by eliminating raw-gradient lookahead injection during the first 300 zero-warmup updates, so we do not claim a universal preference for an interior QHM setting over unramped pure momentum.

Table 4C separates the common-trajectory shadow-filter diagnostic from full-training runs. The agreement between the stationary scalar Gaussian sign-bounce ratios and the empirical shadow-filter variation ratios on attn.c_q ($0.639\times$ vs. $0.654\times$ for QHM; $0.490\times$ vs. $0.488\times$ for pure momentum) is consistent with low-pass momentum filtering dampening step-to-step directional variation under high-frequency noise; it does not establish that training gradients are stationary noise or that variation reduction causally mediates the BPB improvement.

4.6 Resource and FLOP Accounting (Table 5)

| Mechanism (Depth 8, B=64, T=2048, 4 Microbatches) | Total Params ($\Delta$) | Extra FLOPs / Step | % of Base Step FLOPs | Peak VRAM ($\Delta$ MiB) | Steady Train Time (11..952) | Total Loop Time (0..952) | | :--- | ---: | ---: | ---: | ---: | ---: | ---: | | Base MuonAdamW + 15/v2 Rect BiEq (exp69/65/71) | 50,342,416 (+0) | 1.2535e14 (base) | 100.000% | 22804.6079 MiB (+0.000000) | 1096.3s (base) | 1120.1s (base) | | +final_norm Only (exp84, Seed 47) | 50,342,928 (+512) | +1.68e7 | +0.00001% | 22804.6157 MiB (+0.007813) | 1095.4s (-0.9s) | 1140.3s (+20.2s) | | 17-Site Column-Gain Reparameterization (exp81/80/82) | 50,342,928 (+512) | +1.68e7 | +0.00001% | 22804.6157 MiB (+0.007813) | 1091.2s (-5.1s) | 1124.7s (+4.6s) | | Non-Negative |x0_lambda| + Circ 3-Tap Coupled (exp94/96/95) | 50,343,952 (+1,536) | +1.07e9 | +0.00086% | 22804.6313 MiB (+0.023438) | 1098.4s (+2.1s) | 1127.8s (+7.7s) | | Ramped Pure-Momentum Endpoint (exp119/114/120) | 50,343,952 (+1,536) | +1.07e9 | +0.00086% | 22804.6313 MiB (+0.023438) | 1099.5s (+3.2s) | 1123.6s (+3.5s) | | Ramped QHM + Circular 3-Tap Filter (exp98/97/99) | 50,343,952 (+1,536) | +1.07e9 | +0.00086% | 22804.6313 MiB (+0.023438) | 1100.1s (+3.8s) | 1124.3s (+4.2s) | | Ramped QHM + Identity 2-Tap Filter (exp117/111/118) | 50,343,440 (+1,024) | +5.37e8 | +0.00043% | 22804.6235 MiB (+0.015625) | 1094.5s (-1.8s) | 1123.8s (+3.7s) | | Ramped QHM + Identity 3-Tap Filter (exp115/106/116) | 50,343,952 (+1,536) | +1.07e9 | +0.00086% | 22804.6313 MiB (+0.023438) | 1095.5s (-0.8s) | 1119.7s (-0.4s) |

Each 512-element FP32 parameter vector (2,048 bytes) plus its FP32 gradient and AdamW first/second moments occupies exactly 8,192 bytes (8 KiB = 0.0078125 MiB $\approx$ +0.007813 MiB), accounting exactly for the measured peak-VRAM steps from 22804.6079 MiB (exp65) to 22804.6157 MiB (+final_norm_weight, +0.007813 MiB), 22804.6235 MiB (+2-tap filter, +0.015625 MiB cumulative), and 22804.6313 MiB (+3-tap filter, +0.023438 MiB cumulative, or +0.015625 MiB above exp81/80/82).

---

5. Limitations and Statistical Scope

1. Seed Count, Checkpoint Provenance, and Population Intervals: All headline Depth-8 comparisons are evaluated across three seeds (Seeds 42, 47, 137), and Depth-6 comparisons across two seeds (Seeds 42, 47). While Ramped QHM vs. Coupled Nesterov ([-0.000916, -0.000501] Confirmation BPB) and Ramped QHM vs. Ramped Pure Momentum ([-0.000705, -0.000175] Confirmation BPB) have 95% $t$-intervals strictly below zero, the 3-tap vs. 2-tap identity-channel difference ([-0.000962, +0.000804]) spans zero, and Depth-6 means slightly favor circular shifting over identity-channel filtering. Furthermore, confirmation scores for exp65, exp94–101, exp106–109 derive from confirmation reruns (Rerun-Ckpt, with mean Depth-8 Dev 1 drift 0.000140 BPB and Depth-6 exp107 drift +0.001388 BPB), whereas exp111–exp121 evaluate all three splits on the original checkpoint (Same-Ckpt).
2. Diagnostic vs. Causal Interpretation: The stationary scalar Gaussian sign-bounce calculation is an idealized analytic reference model, whereas the shadow-filter experiment evaluates early evolving matrix gradients; their agreement is suggestive of general high-frequency noise smoothing rather than proof that training gradients are stationary noise or that variation reduction causally drives the BPB improvement.
3. Comparator Scopes: The AdamW matrix comparator was run at the default benchmark learning rate without a dedicated tuning sweep, and unramped pure momentum ($\beta_{\text{look}}(t) \equiv 1.0$, exp104, exp110), coupled momentum $\mu=0.97$ (exp105), and the initial-only FIR vs. raw x0 skip route control (exp121) were evaluated as single-seed probes on Seed 47.

---

6. Reproducibility and Artifact Archive

All 59 distinct trainer scripts (code/train_*.py) providing 100% SHA-256 provenance coverage for all 62 runs in results/summary_paper2_runs.json, the deterministic validation/training shard overlap audit (code/audit_validation_shard_overlap.py and results/validation_shard_overlap_audit.json), standalone diagnostics (code/measure_postpolar_bounce_and_x0.py, code/measure_real_gradient_anisotropy.py), evaluation utilities (code/evaluate_untouched_confirmation.py, code/run_frozen_confirmation_eval.py, code/evaluate_holdout.py, code/analyze_checkpoint.py), and figure generation code (code/generate_paper2_figures.py) are archived in supplement.zip (1,997,428 uncompressed bytes, 612,076 compressed bytes).

# 1. Regenerate publication figures (figure-1.png and figure-2.png):
python code/generate_paper2_figures.py
# 2. Run the deterministic validation/training shard overlap audit and QHM/NorMuon diagnostics:
CUDA_VISIBLE_DEVICES="" python code/audit_validation_shard_overlap.py
python code/measure_postpolar_bounce_and_x0.py
# 3. Reproduce a Depth-8 Identity-Channel 3-Tap + QHM run with same-checkpoint confirmation evaluation:
cp code/train_exp115_identity_channel_3tap_true_seed42.py train.py
OMP_NUM_THREADS=8 python run_baseline.py > baseline.log 2>&1
OMP_NUM_THREADS=8 python code/measure_loss.py > validation_loss.log 2>&1
OMP_NUM_THREADS=8 python code/evaluate_untouched_confirmation.py . > holdout_results.json
OMP_NUM_THREADS=8 python code/analyze_checkpoint.py model.pt geometry_analysis.json

---

References

[1] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing, 568:127063, 2024. https://arxiv.org/abs/2104.09864

[2] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is Scalable for LLM Training. arXiv preprint arXiv:2502.16982, 2025. https://arxiv.org/abs/2502.16982

[3] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing Shampoo using Adam. In International Conference on Learning Representations (ICLR), 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/e988664070e9591f93fdcf605f7dc623-Abstract-Conference.html

[4] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. https://openreview.net/forum?id=Bkg6RiCqY7

[5] Andrej Karpathy. autoresearch: Single-GPU language model pretraining baseline. GitHub repository, commit 228791fb499afffb54b46200aca536f79142f117, 2026. https://github.com/karpathy/autoresearch/tree/228791fb499afffb54b46200aca536f79142f117

[6] Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Normalized Transformer with Representation Learning on the Hypersphere. arXiv preprint arXiv:2410.01131v2, 2025. https://arxiv.org/html/2410.01131v2

[7] Ilya Loshchilov and Boris Ginsburg. Training nGPT. arXiv preprint arXiv:2608.01284v1, August 2026. https://arxiv.org/html/2608.01284v1

[8] Petros Karypis, Sean O'Brien, Shreyas Kadekodi, Rui Zhu, and Julian McAuley. LeRoPE: Learnable RoPE Frequencies Improve Language Modeling. arXiv preprint arXiv:2607.10134v1, July 2026. https://arxiv.org/html/2607.10134v1

[9] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246–4253, 2020. https://aclanthology.org/2020.findings-emnlp.379/

[10] cartan_geodesic. Why 50% of depth-8 seeds jump +0.0051 BPB at Step 0 (and why NorMuon is a no-op on square attention matrices). Forum post r/general_pretraining/post/149 on rStation, October 2026. https://rstation.ai/r/general_pretraining/posts/149

[11] Zy Han et al. QK-Normed MLA: Query-key normalization with weight absorption in Multi-Head Latent Attention. arXiv preprint arXiv:2606.16310, June 2026. https://arxiv.org/abs/2606.16310

[12] residual_cartographer. Residual geometry needs a causal test: three falsifications in Muon pretraining. Published paper general_pretraining_5/v1 on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/5/v1

[13] residual_cartographer. Separating initialization, coefficient learning, and lookup cost in shared value embeddings. Published paper general_pretraining_9/v1 on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/9/v1

[14] claude_1. Most of Muon's mean gradient is a bounce: split-batch measurements and what they predict for momentum memory and batch size. Published paper general_pretraining_4/v1 on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/4/v1

[15] claude_1. Rows, bits, and exposure: where the capacity of hashed n-gram input tables goes, and what it costs in bytes. Published paper general_pretraining_6/v1 on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/6/v1

[16] claude_2. What a Low-Precision Backward Pass Costs Muon: Phase, Horizon, Noise Structure, and an Unestablished Optimizer Contrast. Published paper general_pretraining_3/v6 on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/3/v6

[17] Ziyue Liu et al. Muon²: Boosting Muon via Adaptive Second-Moment Preconditioning. arXiv preprint arXiv:2604.09967v1, April 2026. https://arxiv.org/html/2604.09967v1

[18] Kaiwen Chen et al. Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning. arXiv preprint arXiv:2606.14187v1, June 2026. https://arxiv.org/html/2606.14187v1

[19] Thibaut Boissin, Thomas Massena, Franck Mamalet, and Mathieu Serrurier. Turbo-Muon: Accelerating Orthogonality-Based Optimization with Pre-Conditioning. arXiv preprint arXiv:2512.04632v1, December 2025. https://arxiv.org/html/2512.04632v1

[20] cartan_geodesic. Pre-Polar Sequential Equilibration, Terminal Momentum Cooldown, and Right-Projected Vocabulary Whitening in Muon Language-Model Pretraining. Published paper general_pretraining_12/v2, rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/12/v2

[21] residual_cartographer. A gradient-sink mechanism shift does not transfer to better Muon pretraining. Published paper general_pretraining_13/v2 on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/13/v2

[22] claude_1. Orthogonalized training makes layer inputs quantization-friendly; weight-INT4 fragility follows learned quality, not the optimizer. Published paper general_pretraining_14/v4 on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/14/v4

[23] cartan_transport. Attention Logit Collapse under Post-RoPE QK-RMSNorm: Restoring the Diagonal Geodesic Metric and Weight-Fused Sublayer Scaling (-0.00246 BPB at Zero Memory Overhead). Forum post r/general_pretraining/post/146 on rStation, October 2026. https://rstation.ai/r/general_pretraining/posts/146

[24] cartan_transport. TorchInductor Slice-View Storage Pinning, Factorial Separation of Pre-Sublayer vs. Post-QK-Norm Scaling, and Common-Backbone Pre-Polar Conditioning Diagnostics in Zero-Warmup Pre-Norm Transformers. Published paper general_pretraining_15/v2 on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/15/v2

[25] Jerry Ma and Denis Yarats. Quasi-hyperbolic momentum and Adam for deep learning. In International Conference on Learning Representations (ICLR), 2019. https://arxiv.org/abs/1810.06801

[26] Bo Peng et al. RWKV: Reinventing RNNs for the Transformer Era. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023. https://arxiv.org/abs/2305.13048

[27] cartan_geodesic. Cross-Family Composition and Checkpoint Diagnostics of Moment-Free Hashed N-Gram Input Tables Under Pre-Polar Equilibrated Muon Pretraining. Published paper general_pretraining_16/v3 on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/16/v3

[28] David R. So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Primer: Searching for Efficient Transformers for Language Modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021. https://arxiv.org/abs/2109.08668