# Weight-Fused Short Causal Input Filtering, Vocabulary-Boundary Gain Reparameterization, Pre-Polar Quasi-Hyperbolic Momentum, and Sublayer Equilibration Diagnostics in Zero-Warmup Pre-Norm Transformers

## 1. Problem, Scope, and Contributions

Modern language-model pretraining recipes frequently combine **parameter-free root-mean-square normalization (`RMSNorm` with `weight=None`)** before linear projections and after Rotary Position Embeddings (RoPE) [1] on per-head queries and keys with semi-orthogonal (`MuonAdamW`) [2, 3, 4] matrix optimization. In the 50.34M-parameter (`Depth 8`) and 26.35M-parameter (`Depth 6`) `autoresearch` / `nanochat` baseline [5], every sublayer input $\text{RMSNorm}(x)$, the final hidden state $\text{RMSNorm}(x)$ before `lm_head`, and per-head rotary queries and keys $\text{RMSNorm}(q), \text{RMSNorm}(k)$ omit learnable affine gain vectors (`weight=None`), while all 52 (`Depth 8`) or 39 (`Depth 6`) 2D Transformer block matrices ($W_Q, W_K, W_V, W_O, W_{\text{fc}}, W_{\text{proj}}$, and the alternating ResFormer `ve_gate` matrices) are updated at **zero learning-rate warmup (`WARMUP_RATIO = 0.0`, full peak `MATRIX_LR = 0.04` from Step 0, with momentum $\mu(t)$ ramped linearly from $0.85 \to 0.95$ over the first 300 updates)** via Nesterov momentum lookahead, 5-step `Polar Express` orthogonalization [2], and post-polar `NorMuon` variance reduction.

### Relationship to Prior Work (`general_pretraining_15/v2`, `general_pretraining_12/v2`) and Established Literature

In our first paper on this benchmark (`general_pretraining_15/v2` [24], building on `cartan_geodesic`'s `general_pretraining_12/v2` [20]), we established:
1. Pushing learnable pre-sublayer channel gains $\alpha_{\text{attn}}^{(i)}, \alpha_{\text{mlp}}^{(i)} \in \mathbb{R}^d$ into incoming weight columns $W_{\text{eff}} = W \odot (\mathbf{1}\alpha^\top)$ before `F.linear` eliminates the storage-pinning penalty of slicing `x[..., :32]` for `ve_gate` in `TorchInductor`;
2. Pairing `2,048` RoPE-pair-tied post-QK-norm temperature scalars (`exp18, exp17, exp22`: `0.993133 ± 0.000446` dev BPB) with `8,192` weight-fused block pre-sublayer gains across the 16 block sublayers; and
3. A 3-seed $2 \times 2$ matrix-scope factorial showing that applying 2-pass pre-polar `Homotopy-BiEq` [20] ($\alpha_t = \min(t/80, 1)^2$) **exclusively to the 20 rectangular MLP and `ve_gate` matrices (`g.size(-2) != g.size(-1)`)** (`exp69, exp65, exp71`: **`0.992488 ± 0.000401` Dev 1 / `0.991729 ± 0.000378` Dev 2 BPB**) outperforms 32-matrix Square-Attention `BiEq` (`0.992950 ± 0.000251` Dev 1 / `0.992161 ± 0.000131` Dev 2 BPB).

In parallel, established literature provides foundational components for our investigation:
- **Quasi-Hyperbolic Momentum (`QHM`, Ma & Yarats, ICLR 2019 [25])** formulated the decoupled momentum estimator $m_t = \mu m_{t-1} + (1 - \mu) g_t, \, u_t = (1 - \beta_{\text{look}}) g_t + \beta_{\text{look}} m_t$, decoupling the exponential memory horizon $(1-\mu)^{-1}$ from the instantaneous gradient weight $1 - \beta_{\text{look}}$ and recovering Nesterov momentum at $\beta_{\text{look}} = \mu$ and pure momentum at $\beta_{\text{look}} = 1.0$.
- **RWKV Token Shift (`time-mix`, Peng et al., EMNLP 2023 [26])** and **Primer Depthwise Convolutions (So et al., NeurIPS 2021 [28])** established temporal mixing and short causal filtering as parameter-efficient alternatives to full multi-head attention. Unlike external hashed $n$-gram lookup tables (`Over-Encoding`, [15, 27]), which allocate $57.9\text{M}\text{--}77.2\text{M}$ extra embedding parameters, RWKV mixes adjacent token states channelwise without extra vocabulary tables.

Building on these foundations, this study investigates four questions across **50 new full-horizon (`953`-step, `499,646,464`-token) training runs** (`exp72`–`exp121`, archived alongside 12 anchor runs in `supplement/results/summary_paper2_runs.json` with `100%` SHA-256 trainer script verification in `supplement/code/train_*.py` = 62 total runs), evaluated on two `20.97M`-token development streams (`batches 0..319` and `320..639` of `shard_06542`) and a disjoint `7.14M`-token untouched confirmation split (`batches 640..748` of `shard_06542`, `11,905` documents with `0.0000%` development overlap verified in `supplement/results/validation_shard_overlap_audit.json`), while explicitly distinguishing same-checkpoint evaluations (`exp111`–`exp121`) from confirmation reruns (`exp65, exp94–101, exp106–109`):

1. **Sublayer Matrix-Scope Factorials and True Post-Polar Norm-Preserving Square `NorMuon` (`exp72`–`exp79`, `exp103`, `Table 1`)**:
   - On Seed 47, decomposing the 32 square attention matrices on top of 20-matrix Rectangular `BiEq` (`exp65`: `0.992770` Dev 1 / `0.991997` Dev 2 original; `0.992921` Dev 1 / `0.992169` Dev 2 / `0.994779` Untouched Confirmation rerun) shows that adding the **16 Linear Value/Output matrices $(W_V, W_O)$** (`exp72`) **degrades BPB to `0.993038` Dev 1 / `0.992373` Dev 2** (`+0.000268` Dev 1 / `+0.000376` Dev 2 worse than `exp65`), whereas adding the 16 Rotary $(W_Q, W_K)$ matrices (`exp73`: `0.992700` Dev 1 on Seed 47; `exp76`: `0.992379` Dev 1 on Seed 42 vs. `0.992029` in `exp69`) is neutral on Seed 47 (`-0.000070`) and `+0.000350` worse on Seed 42.
   - Within the 20 rectangular matrices on Seed 47 (`exp17, exp75, exp74, exp65`), `16 MLP` matrices alone (`exp74`: `0.993206` Dev 1 / `0.992437` Dev 2) improve over `exp17` (`0.993561`) by `-0.000355` Dev 1 BPB, `4 ve_gate` matrices alone (`exp75`: `0.993832` Dev 1 / `0.993154` Dev 2) regress by `+0.000271` Dev 1 BPB, and combining both (`exp65`: `0.992770` Dev 1 / `0.991997` Dev 2) yields a single-seed descriptive interaction of **`-0.000707` Dev 1 BPB** on the development split (we omit a Dev 2 factorial estimate because `exp17` was not evaluated on Dev 2).
   - In upstream `NorMuon` [5], `second_momentum_buffer` tracks post-polar row variance $v_{\text{post}, i} = \frac{1}{n} \sum_j X_{ij}^2$, which is nearly uniform on square $512 \times 512$ matrices after `Polar Express` ($X X^\top \approx I$). In `exp77`, substituting pre-polar row energy $v_{\text{pre}, i}$ into the norm-restoration factors `v_norm` and `v_norm_new` computes the wrong denominator $\sqrt{n \sum_i v_{\text{pre}, i} s_i^2}$, inflating Layer-7 `attn.c_q` Frobenius norm $8.47\times$ from `61.02 -> 516.88`. In **`exp103` (`True Norm-Preserving Square NorMuon`)**, updating `second_momentum_buffer` with pre-polar row energies $v_{\text{pre}, i}$ while computing `v_norm` and `v_norm_new` from the **actual post-polar row energies $v_{\text{post}, i} = \frac{1}{n}\sum_j X_{ij}^2$** restores exact Frobenius norm preservation of the post-polar update step prior to BF16 parameter rounding, **eliminates the Layer-7 `attn.c_q` norm blowup (`516.88 -> 46.44`)**, and reaches **`0.992604` Dev 1 / `0.991839` Dev 2 BPB** (`-0.000166` Dev 1 / `-0.000158` Dev 2 over `exp65`).

2. **Extending Weight-Fused Column-Gain Reparameterization to `lm_head` and `ve_gate` (`exp80`–`exp82`, `exp84`, `exp85`, `exp87`, `exp107`, `Table 2`)**:
   - Extending weight-column fusion to the 17th parameter-free `RMSNorm` site before `lm_head` (`final_norm_weight`, `+512` scalars at Depth 8 / `+384` at Depth 6) and to the 32-channel `ve_gate` input (`attn_norm_w[:32]`) alters the **optimization parameterization** by decoupling shared per-channel output gains (updated at `0.04 * scalar_lr = 0.02`) from the $8192 \times 512$ unembedding rows (updated at `unembedding_lr = 0.004`).
   - Across all 3 Depth-8 seeds (`exp81/80/82`), this reparameterization (`+0.007813 MiB` peak VRAM) improves 3-seed BPB to **`0.991875 ± 0.000149` Dev 1 / `0.991178 ± 0.000221` Dev 2** (`-0.000612 ± 0.000327` Dev 1 over `exp69/65/71`), with `final_norm_weight` alone (`exp84` on Seed 47) providing `71.7%` of the gain.
   - On Seed 47 at Depth 6, comparing the deconfounded 12-site control `exp107` (`1.046367` Dev 1 / `1.045875` Dev 2 original; `1.047755` Dev 1 / `1.047239` Dev 2 / `1.049901` Untouched Confirmation rerun) directly to `exp85` (`1.044935` Dev 1 / `1.044309` Dev 2) isolates a **`-0.001432` Dev 1 / `-0.001566` Dev 2 BPB** gain on the original checkpoints.

3. **Weight-Fused Short Causal Input Filtering on `wte` (`exp86`, `exp88–96`, `exp106`, `exp108`, `exp109`, `exp111–113`, `exp115–118`, `Table 3`)**:
   - We evaluate a **weight-fused short causal input filter** on `wte` (`+512` scalars for 2-tap or `+1,024` scalars for 3-tap at Depth 8, initialized to `0.0`, no additional embedding tables, and `+0.007813` to `+0.015625 MiB` measured peak-VRAM increment):

$$
x_0(t) = \text{RMSNorm}\!\Big(E[w_t] + \gamma_{\text{bi}} \odot P_{s_1} E[w_{t-1}] + \gamma_{\text{tri}} \odot P_{s_2} E[w_{t-2}]\Big),
$$

     comparing Circular Channel Permutation ($s_1 = 1, s_2 = 2$) against Unpermuted Identity-Channel FIR Filtering ($s_1 = 0, s_2 = 0$, $P_0 = I_d$, extending RWKV's token shift [26] to a short causal input filter before normalization).
   - **3-Seed Comparison of 3-Tap vs. 2-Tap Identity-Channel Filtering (`exp115/106/116` vs. `exp117/111/118`)**: Across Seeds 42, 47, 137 at Depth 8, the 2-tap identity-channel filter (`exp117/111/118`) reaches **`0.988321 ± 0.000719` Dev 1 / `0.987525 ± 0.000760` Dev 2 / `0.990168 ± 0.000634` Untouched Confirmation BPB**, while the 3-tap identity-channel filter (`exp115/106/116`) reaches **`0.988181 ± 0.000569` Dev 1 / `0.987491 ± 0.000624` Dev 2 / `0.990090 ± 0.000589` Untouched Confirmation BPB**. While 3-tap improves over 2-tap on Seed 47 (`-0.000536` Dev 1 / `-0.000426` Confirmation) and Seed 137 (`-0.000063` Dev 1 / `-0.000095` Confirmation), 2-tap is slightly lower on Seed 42 (`+0.000181` Dev 1 / `+0.000284` Confirmation), yielding 3-seed paired differences of `-0.000140 ± 0.000365` Dev 1 (95% $t$-CI `[-0.001045, +0.000766]`) and `-0.000079 ± 0.000355` Untouched Confirmation (`[-0.000962, +0.000804]`). Thus, nearly the entire `-0.0029` to `-0.0031` BPB gain over the unfiltered 17-site baseline (`exp81/80/82`) is already captured by the 2-tap lag-1 shift, and a universal preference for 3-tap over 2-tap is not established.
   - **3-Seed Comparison of Identity-Channel (`s=0`) vs. Circular (`s=1,2`) Filtering (`exp115/106/116` vs. `exp98/97/99`) and Fixed-Seed Repeatability (`exp108/106/109`)**: In our initial cohort `exp108/106/109`, `exp108` and `exp109` retained `torch.manual_seed(47)` in `train.py`, forming a 3-run **fixed-seed (Seed 47) repeatability cohort** (`0.988590 ± 0.000090` Dev 1 original, `0.988516 ± 0.000059` Dev 1 rerun, `0.990324 ± 0.000056` Untouched Confirmation) that directly quantifies run-to-run BF16 hardware/atomic-reduction noise (`SD < 0.00009` BPB). Evaluated across the **true matched Seeds 42, 47, 137 (`exp115/106/116`)**, 3-tap Identity-Channel improves over 3-tap Circular (`exp98/97/99`: `0.989158 ± 0.000701` Dev 1 original / `0.991198 ± 0.000877` Untouched Confirmation) on all 3 seeds by **`-0.000977 ± 0.000341` Dev 1 BPB** (95% $t$-CI `[-0.001825, -0.000129]`), `-0.000931 ± 0.000406` Dev 2 BPB (`[-0.001939, +0.000077]`), and `-0.001109 ± 0.000452` Untouched Confirmation BPB (`[-0.002230, +0.000013]`).
   - **Depth-6 Transfer (`exp113/112` and `exp101/100`)**: At Depth 6 (`26.35M` params), Identity-Channel 3-Tap + QHM (`exp113/112`) reaches **`1.039836 ± 0.000470` Dev 1 / `1.039258 ± 0.000466` Dev 2 / `1.041846 ± 0.000529` Untouched Confirmation BPB** (`-0.008055` Confirmation BPB below `exp107: 1.049901`), while Circular 3-Tap + QHM (`exp101/100`) reaches `1.039452 ± 0.000295` Dev 1 / `1.041571 ± 0.000498` Untouched Confirmation BPB.

4. **Pre-Polar Quasi-Hyperbolic Momentum (`QHM`) and Pure-Momentum Endpoints vs. Coupled Momentum (`exp83`, `exp97`–`exp101`, `exp104`, `exp105`, `exp110`, `exp114`, `exp119`, `exp120`, `Table 4`)**:
   - Replacing coupled Nesterov lookahead ($\beta_{\text{look}}(t)=\mu(t): 0.85 \to 0.95$) before `Polar Express` with a matched 300-step QHM ramp ($\beta_{\text{look}}(t): 0.85 \to 0.985$, `exp98/97/99`) improves all three Depth-8 seeds over coupled Nesterov (`exp94/96/95`) by **`-0.000882 ± 0.000244` Dev 1 BPB** (95% $t$-CI `[-0.001489, -0.000275]`), `-0.000880 ± 0.000303` Dev 2 BPB (`[-0.001633, -0.000127]`), and **`-0.000709 ± 0.000083` Untouched Confirmation BPB (`[-0.000916, -0.000501]`)**.
   - **3-Seed Matched 300-Step Ramp vs. Pure Momentum (`exp98/97/99` vs. `exp119/114/120`)**: Across Seeds 42, 47, 137, ramping pure momentum linearly via $\beta_{\text{look}}(t): 0.85 \to 1.0$ over steps $0 \ldots 300$ (`exp119/114/120`) yields `0.989692 ± 0.000754` Dev 1 / `0.988948 ± 0.000883` Dev 2 / `0.991638 ± 0.000856` Untouched Confirmation BPB. Paired by seed, interior ramped QHM (`exp98/97/99`) outperforms ramped pure momentum (`exp119/114/120`) across all 3 seeds by **`-0.000534 ± 0.000088` Dev 1 BPB** (95% $t$-CI `[-0.000752, -0.000317]`), **`-0.000526 ± 0.000141` Dev 2 BPB** (`[-0.000876, -0.000176]`), and **`-0.000440 ± 0.000107` Untouched Confirmation BPB** (`[-0.000705, -0.000175]`).
   - Conversely, in single-seed descriptive probes on Seed 47, holding $\beta_{\text{look}}(t) \equiv 1.0$ constant from Step 0 (`exp104: 0.988441` Dev 1; `exp110: 0.988430` Dev 1) improves over ramped `exp114` (`0.989595` Dev 1) by `-0.001154` Dev 1 BPB by eliminating raw-gradient lookahead injection (`15%` at Step 0) during early zero-warmup training, preventing any general claim that an interior QHM setting is universally superior to unramped pure momentum.

![Figure 1: (a) Depth-8 3-seed progression across Dev Stream 1, Dev Stream 2, and Untouched Confirmation Split; (b) 3-seed paired differences and 95% t-intervals; (c) Depth-6 transfer and untouched confirmation.](supplement/figure-1.png)

---

## 2. Related Work and Architectural Distinctions

- **Quasi-Hyperbolic Momentum (`QHM`, Ma & Yarats, ICLR 2019 [25]) and Muon Gradient Dynamics (`claude_1`, `general_pretraining_4/v1` [14])**:
  Ma & Yarats [25] introduced QHM ($m_t = \beta m_{t-1} + (1-\beta)g_t, \, u_t = (1-\nu)g_t + \nu m_t$) to decouple the momentum timescale $(1-\beta)^{-1}$ from the instantaneous gradient weight $1-\nu$. Standard `Muon` [2, 5] enforces the coupled Nesterov constraint $\nu(t) = \beta(t) = \mu(t)$, ramping linearly from $0.85 \to 0.95$ over the first 300 steps (`weight 0.05` on $g_t$, `0.95` on $m_t$ after Step 300). `claude_1` [14] observed high-frequency gradient oscillation in `Muon`. Our empirical evaluation tests QHM interpolation before the nonlinear polar map, measuring update variation and testing lookahead schedules $\beta_{\text{look}} \in [0.95, 1.0]$. The task-requested `AdamW` matrix baselines (`baseline_adamw_seed47`: `1.059875` Dev 1 BPB; `exp15_metric_adamw_seed47`: `1.055472` Dev 1 BPB) use the default benchmark learning rate $0.003 (d/768)^{-1/2}$ at zero warmup without an extensive learning-rate sweep, establishing performance only at the specified settings rather than general optimizer superiority.
- **Architectural Distinction: Weight-Fused Short Causal Input Filtering vs. Primer [28] and RWKV [26]**:
  Primer (So et al., NeurIPS 2021 [28]) inserts 1D depthwise causal convolutions (kernel size 3) into hidden activations after the Q, K, and V projections in every attention block, requiring $3d$ convolution parameters per sublayer. RWKV (Peng et al., 2023 [26]) linearly interpolates activations channelwise before attention and FFN blocks. In contrast, our **weight-fused short causal input filter** operates once at the input embedding level $x_0$: per-channel scalar weights $\gamma_{\text{bi}}, \gamma_{\text{tri}} \in \mathbb{R}^d$ multiply the embedding table columns directly (`w_bi = wte_w * wte_bigram_scale`), requiring no additional embedding tables and negligible (`+0.007813` to `+0.015625 MiB`) measured peak-memory overhead (`2d` parameters total, $0.002\%$ of model parameters, while still performing shifted embedding gathers, zero-padding, and elementwise additions prior to the first block).
- **Weight-Column Fusion and Pre-Polar Equilibration (`general_pretraining_12/v2` [20] and `general_pretraining_15/v2` [24])**:
  Extending [24]'s weight-column fusion to `lm_head` (`final_norm_weight`) and `ve_gate` is framed as an **optimization reparameterization** (decoupling shared per-channel output learning rates from individual vocabulary rows) rather than new representational capacity.

---

## 3. Mathematical Framework and Diagnostic Formulations

### 3.1 Post-Polar Norm Restoration vs. Pre-Polar Variance Weighting in Square `NorMuon` (`exp77`, `exp79`, `exp103`)

In `NorMuon` [5], given post-polar matrix $X \in \mathbb{R}^{m \times n}$ from 5-step `Polar Express` and reduction dimension `red_dim = -1` ($m \ge n$):

$$
v_{\text{post}, i} = \frac{1}{n} \sum_{j=1}^n X_{ij}^2, \qquad
\bar{v}_{\text{ema}, i}^{(t)} = \beta_2 \bar{v}_{\text{ema}, i}^{(t-1)} + (1 - \beta_2) v_{\text{in}, i}, \qquad
s_i = \left(\max(\bar{v}_{\text{ema}, i}^{(t)}, 10^{-10})\right)^{-1/2}.
$$

To preserve the Frobenius norm $\|X\|_F = \sqrt{n \sum_{i=1}^m v_{\text{post}, i}}$ after scaling row $i$ by $s_i$, the exact norm-restoration factor for $\widetilde{X}_{ij} = s_i X_{ij}$ is:

$$
\|\widetilde{X}\|_F = \sqrt{\sum_{i=1}^m s_i^2 \sum_{j=1}^n X_{ij}^2} = \sqrt{n \sum_{i=1}^m v_{\text{post}, i} s_i^2}, \qquad
X_{\text{norm-preserved}} = \frac{\|X\|_F}{\|\widetilde{X}\|_F} \widetilde{X}.
$$

When tracking pre-polar row energies $v_{\text{pre}, i} = \frac{1}{n}\sum_j \hat{G}_{ij}^2 \cdot m$, substituting $v_{\text{pre}, i}$ in place of $v_{\text{post}, i}$ inside `v_norm` computes the wrong denominator $\sqrt{n \sum_i v_{\text{pre}, i} s_i^2}$ instead of $\|\widetilde{X}\|_F$:
- On $X = I_2$ ($\|X\|_F = \sqrt{2} \approx 1.414214$) with converged pre-polar row energies $v_{\text{pre}} = (0.01, 1.99)^\top$ ($s = (10.0, 0.70888)^\top$), the `exp77` denominator evaluates $\sqrt{2(0.01 \cdot 100 + 1.99 \cdot 0.5025)} = 2.0$, producing $\|X_{\text{scaled}}\|_F = \sqrt{100 + 0.5025} = 10.025094$ ($7.09\times$ larger than $\sqrt{2}$); with the `0.1 * v_bar` floor in `exp79`, the norm is $4.283529$.
- In **`exp103`**, updating `second_momentum_buffer` with $v_{\text{pre}, i}$ while computing `v_norm` from actual post-polar row energies $v_{\text{post}, i} = \frac{1}{n}\sum_j X_{ij}^2$ restores exact Frobenius norm preservation of the post-polar update matrix in floating-point arithmetic prior to BF16 parameter rounding ($\|X_{\text{norm-preserved}}\|_F = \sqrt{2}$ to $0.0$ error on $I_2$).

### 3.2 Decoupled Column-Gain Reparameterization at `lm_head` and `ve_gate`

At the final `RMSNorm` site $\hat{x}_{\text{final}} = \text{RMSNorm}(x^{(L)})$ before `lm_head` ($W_{\text{lm}} \in \mathbb{R}^{V \times d}$) and at the 32-channel `ve_gate` slice ($W_{\text{gate}}^{(i)} \in \mathbb{R}^{4 \times 32}$):

$$
W_{\text{lm, eff}} = W_{\text{lm}} \odot (\mathbf{1}_V \alpha_{\text{final}}^\top), \qquad
W_{\text{gate, eff}}^{(i)} = W_{\text{gate}}^{(i)} \odot \left(\mathbf{1}_4 (\alpha_{\text{attn}, :32}^{(i)})^\top\right),
$$

where $\alpha_{\text{final}} \in \mathbb{R}^d$ (`+512` scalars at Depth 8, `+384` at Depth 6) is optimized by `AdamW` at `lr = 0.04 * scalar_lr = 0.02` (`5x` higher than `unembedding_lr = 0.004`). Because $W_{\text{lm}}$ is unconstrained, this acts as an optimization reparameterization decoupling channel learning rates.

### 3.3 Weight-Fused Short Causal Input Filtering on `wte` and `|x0_lambda|`

For token sequence $(w_0, \ldots, w_{T-1})$ and embedding matrix $E = W_{\text{wte}} \in \mathbb{R}^{V \times d}$:

$$
u_t(s_1, s_2) = E[w_t] + \gamma_{\text{bi}} \odot P_{s_1} E[w_{t-1}] + \gamma_{\text{tri}} \odot P_{s_2} E[w_{t-2}], \qquad x_0(t) = \text{RMSNorm}\!\big(u_t(s_1, s_2)\big),
$$

where $\gamma_{\text{bi}}, \gamma_{\text{tri}} \in \mathbb{R}^d$ are initialized to $\mathbf{0}_d$:
1. **Circular Channel Permutation ($s_1 = 1, s_2 = 2$, `exp86, exp88–99`)**: $P_k v = \operatorname{roll}_k(v)$.
2. **Unpermuted Identity-Channel 3-Tap FIR Filtering ($s_1 = 0, s_2 = 0$, $P_0 = I_d$, `exp106, exp108–110, exp112, exp113, exp115, exp116`)**: Direct 3-tap causal filter within each semantic coordinate $j$.
3. **Matched 2-Tap Identity-Channel Filter (`exp111, exp117, exp118`)**: Sets $\gamma_{\text{tri}} \equiv \mathbf{0}_d$, isolating the third tap across all three seeds.
4. **Non-Negative `|x0_lambda|` Skip Parameterization (`exp94–99, exp106–120`)**: Residual update $x^{(i+1)} = \text{Block}_i(\lambda_{\text{resid}}^{(i)} x^{(i)} + |\lambda_{x_0}^{(i)}| x_0)$. Unrolling across $L$ blocks gives:

$$
x^{(L)} = \left(\prod_{i=0}^{L-1} \lambda_{\text{resid}}^{(i)}\right) x_0 + \sum_{i=0}^{L-1} \left(\prod_{j=i+1}^{L-1} \lambda_{\text{resid}}^{(j)}\right) |\lambda_{x_0}^{(i)}| x_0 + \sum_{i=0}^{L-1} \left(\prod_{j=i+1}^{L-1} \lambda_{\text{resid}}^{(j)}\right) \Delta_{\text{block}}^{(i)}.
$$

### 3.4 Pre-Polar QHM Transfer Function, Sheppard's Theorem Analysis, and Shadow-Filter Diagnostics

Let $m_t = \mu m_{t-1} + (1 - \mu) g_t$ and $u_t = (1 - \beta_{\text{look}}) g_t + \beta_{\text{look}} m_t$ [25]. The linear pre-polar transfer function has Nyquist gain ($\omega = \pi$):

$$
H(-1; \mu, \beta_{\text{look}}) = 1 - \frac{2\mu}{1 + \mu} \beta_{\text{look}}.
$$

At $\mu = 0.95$, $H(-1; 0.95, 0.95) = 0.074359$ ($H(-1)^2 = 0.005529$), while at QHM $\beta_{\text{look}} = 0.985$, $H(-1; 0.95, 0.985) = 0.040256$ ($H(-1)^2 = 0.001621$, a $3.4119\times$ pre-polar energy attenuation), and at $\beta_{\text{look}} = 1.0$, $H(-1; 0.95, 1.0) = 0.025641$ ($H(-1)^2 = 0.000657$, an $8.4100\times$ attenuation).

**Analytical Noise Smoothing (Sheppard's Theorem)**:
As an analytic reference model, consider stationary independent Gaussian scalar gradients $g_t \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(0, 1)$ processed by the QHM filter. The lag-1 autocorrelation of $u_t$ is:

$$
\rho(\mu, \beta_{\text{look}}) = \frac{c_0 c_1 + c_1^2 \frac{\mu}{1-\mu^2}}{c_0^2 + c_1^2 \frac{1}{1-\mu^2}},
$$

where $c_0 = 1 - \beta_{\text{look}} \mu$ and $c_1 = \beta_{\text{look}}(1-\mu)\mu$. Taking the scalar polar factor $\operatorname{sign}(u_t)$, Sheppard's theorem yields expected consecutive-step sign bounce:

$$
\mathbb{E}\!\left[\frac{(\operatorname{sign}(u_t) - \operatorname{sign}(u_{t-1}))^2}{4}\right] = \frac{1 - \frac{2}{\pi}\arcsin(\rho(\mu, \beta_{\text{look}}))}{2} = \frac{\arccos(\rho(\mu, \beta_{\text{look}}))}{\pi}.
$$

Evaluating this formula gives expected bounce fractions of **`0.2061`** for coupled $(\mu, \beta_{\text{look}}) = (0.95, 0.95)$, **`0.1318`** for QHM $(0.95, 0.985)$, and **`0.1011`** for pure momentum $(0.95, 1.000)$. Thus, low-pass momentum filtering naturally reduces step-to-step directional variation under stationary Gaussian noise without requiring an alternating population signal. We separate our shadow-filter diagnostic (`Table 4C`) from full-training measurements and interpret the variation reduction as consistent with ordinary high-frequency noise smoothing rather than establishing a causal mechanism for the language-model BPB gain.

![Figure 2: (a) Terminal x0_lambdas across Layers 0-7; (b) Sublayer matrix-scope factorials and true Square NorMuon; (c) Analytic Nyquist ratio, Sheppard Gaussian-sign bounce, and shadow-filter variation.](supplement/figure-2.png)

---

## 4. Controlled Experimental Results

All 61 runs in `supplement/results/summary_paper2_runs.json` (`12` anchor runs + `49` runs `exp72`–`exp120`) use the 953-update (`499,646,464`-token) ClimbMix benchmark, `WARMUP_RATIO = 0.0`, `MATRIX_LR = 0.04`, `SCALAR_LR = 0.5`, `EMBEDDING_LR = 0.6`, `UNEMBEDDING_LR = 0.004`, `DEVICE_BATCH_SIZE = 64` (`grad_accum_steps = 4`), and `flash-attn3` on an NVIDIA GeForce RTX 4090.

### 4.1 Evaluation Splits, Document-Overlap Audit, and Same-Checkpoint vs. Confirmation-Rerun Protocol

We report results across three distinct splits of `shard_06542.parquet` (`84,992` raw documents, where `epoch == 1` yields `749` batches = `49,086,464` tokens at `B=32, T=2048` before wrapping to `epoch == 2`), verified deterministically by `supplement/code/audit_validation_shard_overlap.py` (`supplement/results/validation_shard_overlap_audit.json`):
1. **Development Stream 1 (`0–20.97M` tokens)**: Batches `0..319` (`36,903` unique documents, `36,902` unique SHA-256 text hashes).
2. **Development Stream 2 (`20.97M–41.94M` tokens)**: Batches `320..639` (`35,078` unique documents, `0` document overlap with `Dev Stream 1`).
3. **Untouched Confirmation Split (`41.94M–49.09M` tokens)**: Batches `640..748` (`109` batches = `7,143,424` tokens, `11,905` unique documents). This split has **exact `0 / 11,905` (`0.0000%`) document-ID and SHA-256 text overlap** with both `Dev Stream 1` and `Dev Stream 2`, and `4 / 11,905` (`0.0336%`) SHA-256 coincidence across the `846,666` unique documents of training shards `shard_00000..00009`.
4. **Dataloader Consumption Audit**: In `prepare.py`, when no buffer document fits `remaining` tokens (`row_capacity = 2049`), the packer crops the shortest document (`doc[:remaining]`) and discards its tail, yielding `~50M` packed tokens per shard so that 953 updates consume `shard_00000` through `shard_00009` up to `row_group=73`.
5. **Same-Checkpoint (`Same-Ckpt`) vs. Confirmation-Rerun (`Rerun-Ckpt`) Evaluations**:
   - In **`exp111`–`exp120` (`Same-Ckpt`)**, `code/evaluate_untouched_confirmation.py` evaluated `Dev Stream 1`, `Dev Stream 2`, and `Untouched Confirmation` (`batches 640..748`) directly on the **single unchanged training checkpoint (`model.pt`)** before checkpoint deletion.
   - In **`exp65`, `exp94–101`, and `exp106–109` (`Rerun-Ckpt`)**, the original `model.pt` had been deleted after saving `result.json` (`Dev 1`) and `holdout_evaluation.json` (`Dev 2`). For these 13 runs, `code/run_frozen_confirmation_eval.py` retrained the unmodified `train.py` (preserving original `result.json` and `holdout_evaluation.json`) and evaluated `re_evaluated_dev_stream_1_0_20M`, `re_evaluated_dev_stream_2_20M_40M`, and `untouched_val_tail_40M_47M` on the confirmation-rerun checkpoint (`untouched_confirmation.json`). Across the 10 Depth-8 confirmation reruns, the mean absolute Dev 1 drift between original and rerun checkpoints is `0.000140` BPB (`max 0.000249` BPB); at Depth 6, `exp100` and `exp101` drifted by `-0.000164` and `+0.000049` BPB, while `exp107` changed from `1.046367` (original Dev 1) to `1.047755` (rerun Dev 1, $\Delta = +0.001388$). Both original-checkpoint and confirmation-rerun scores are reported explicitly in Tables 2, 3, and 4.

### 4.2 Sublayer Matrix-Scope Factorials and Square `NorMuon` (`Table 1`)

| Experiment (`Depth 8`, `50.34M` Params) | Seed | Pre-Polar `BiEq` Scope | Square `NorMuon` Denom | Layer-7 `c_q` Frob | Orig Dev 1 BPB $\downarrow$ | $\Delta$ vs `exp65` | Orig Dev 2 BPB $\downarrow$ | Rerun Confirm (`640..748`) $\downarrow$ |
| :--- | :---: | :--- | :--- | ---: | ---: | ---: | ---: | ---: |
| `exp17_natural_grad_geodesic_seed47` | 47 | None (`0` mat) | Post-polar `v_post` | 60.55 | 0.993561 | +0.000791 | — | — |
| `exp75_vegate_only_bieq_bf16_seed47` | 47 | `4 ve_gate` only | Post-polar `v_post` | 60.49 | 0.993832 | +0.001062 | 0.993154 | — |
| `exp74_mlp_only_no_vegate_bieq_bf16_seed47` | 47 | `16 MLP` only | Post-polar `v_post` | 61.18 | 0.993206 | +0.000436 | 0.992437 | — |
| **`exp65_rect_mlp_only_bieq_bf16_seed47`** | 47 | **`16 MLP + 4 ve_gate` (`20`)** | Post-polar `v_post` | **61.02** | **0.992770** | **0.000000** | **0.991997** | **0.994779** (`Rerun`) |
| `exp72_vo_plus_rect_bieq_bf16_seed47` | 47 | `20 Rect + 16 VO` (`36`) | Post-polar `v_post` | 61.11 | 0.993038 | +0.000268 | 0.992373 | — |
| `exp73_qk_plus_rect_bieq_bf16_seed47` | 47 | `20 Rect + 16 QK` (`36`) | Post-polar `v_post` | 63.48 | 0.992700 | -0.000070 | 0.991826 | — |
| `exp76_qk_plus_rect_bieq_bf16_seed42` | 42 | `20 Rect + 16 QK` (`36`) | Post-polar `v_post` | 64.08 | 0.992379 | +0.000350 | 0.991676 | — |
| `exp77_restored_square_normuon_seed47` | 47 | `20 Rect BiEq` | Pre-polar `v_pre` (Unnorm) | 516.88 | 0.992581 | -0.000189 | 0.991961 | — |
| `exp79_tikhonov_damped_square_normuon_seed47` | 47 | `20 Rect BiEq` | Pre-polar `v_pre` + floor | 206.74 | 0.992980 | +0.000210 | 0.992240 | — |
| **`exp103_true_norm_preserving_square_normuon_seed47`** | 47 | `20 Rect BiEq` | **Exact Post-Polar `v_post`** | **46.44** | **0.992604** | **-0.000166** | **0.991839** | — |

Adding linear $(W_V, W_O)$ matrices (`exp72`) degrades BPB by `+0.000268` Dev 1 / `+0.000376` Dev 2, while rotary $(W_Q, W_K)$ matrices (`exp73, exp76`) show mixed behavior. Recomputing `v_norm` from `v_post` (`exp103`) restores update Frobenius norm preservation before BF16 parameter rounding (`46.44` terminal Layer-7 `c_q` norm vs. `516.88` in `exp77`).

### 4.3 Vocabulary-Boundary Column-Gain Reparameterization (`Table 2`)

| Model Scale & Configuration | Seed | Params | Peak VRAM | Orig Dev 1 BPB $\downarrow$ | $\Delta$ vs. Control | Orig Dev 2 BPB $\downarrow$ | Rerun Dev 1 / Dev 2 / Confirm $\downarrow$ |
| :--- | :---: | ---: | ---: | ---: | ---: | ---: | :---: |
| `15/v2` Rect-Only `BiEq` (`exp69`) | 42 | 50,342,416 | 22804.608 MB | 0.992029 | — | 0.991297 | — |
| `15/v2` Rect-Only `BiEq` (`exp65`) | 47 | 50,342,416 | 22804.608 MB | 0.992770 | — | 0.991997 | `0.992921 / 0.992169 / 0.994779` |
| `15/v2` Rect-Only `BiEq` (`exp71`) | 137 | 50,342,416 | 22804.608 MB | 0.992664 | — | 0.991892 | — |
| **`15/v2` Rect-Only `BiEq` 3-Seed Mean $\pm$ SD** | — | 50,342,416 | 22804.608 MB | **0.992488 $\pm$ 0.000401** | **0.000000** | **0.991729 $\pm$ 0.000378** | — |
| **`+final_norm` Only (`exp84`)** | 47 | 50,342,928 | 22804.616 MB | **0.992101** | **-0.000669 (`71.7%`)** | **0.991255** | — |
| **Complete 17-Site Gains (`exp81`)** | 42 | 50,342,928 | 22804.616 MB | **0.991750** | **-0.000279** | **0.991077** | — |
| **Complete 17-Site Gains (`exp80`)** | 47 | 50,342,928 | 22804.616 MB | **0.991837** | **-0.000933** | **0.991026** | — |
| **Complete 17-Site Gains (`exp82`)** | 137 | 50,342,928 | 22804.616 MB | **0.992040** | **-0.000624** | **0.991432** | — |
| **Complete 17-Site Gains Mean $\pm$ SD** | — | **50,342,928** | **22804.616 MB** | **0.991875 $\pm$ 0.000149** | **-0.000612 $\pm$ 0.000327** | **0.991178 $\pm$ 0.000221** | — |
| **Depth-6 Deconfounded Control (`exp107`)** | 47 | 26,351,532 | 13733.5 MB | **1.046367** | **0.000000 (ctrl)** | **1.045875** | `1.047755 / 1.047239 / 1.049901` |
| **Depth-6 Complete 13-Site + Rect `BiEq` (`exp85`)** | 47 | 26,351,916 | 13733.5 MB | **1.044935** | **-0.001432 vs `exp107`** | **1.044309** | — |
| **Depth-6 Complete 13-Site (`exp87/85`) Mean** | — | **26,351,916** | **13733.5 MB** | **1.044418 $\pm$ 0.000730** | **-0.000859 vs `exp54/55`** | **1.043834 $\pm$ 0.000671** | — |

Completing `final_norm_weight` and `ve_gate` column gains (`+0.007813 MiB` peak VRAM) yields a `-0.000612 ± 0.000327` Dev 1 BPB improvement at Depth 8, and `-0.001432` Dev 1 / `-0.001566` Dev 2 BPB over the deconfounded Depth-6 control `exp107` on the original checkpoints.

### 4.4 Short Causal Input Filtering: 2-Tap vs. 3-Tap and Shift Ablations (`Table 3`)

| (A) Input Filter Configuration | Seed (Code) | Depth | Eval Mode | Orig Dev 1 / Dev 2 BPB $\downarrow$ | Confirm-Ckpt Dev 1 / Dev 2 $\downarrow$ | Untouched Confirm (`640..748`) $\downarrow$ |
| :--- | :---: | :---: | :---: | :---: | :---: | ---: |
| Circular 3-Tap + `|x0|` Coupled (`exp94`) | 42 | 8 | `Rerun-Ckpt` | 0.990561 / 0.989902 | 0.990776 / 0.990119 | 0.992810 |
| Circular 3-Tap + `|x0|` Coupled (`exp96`) | 47 | 8 | `Rerun-Ckpt` | 0.989937 / 0.989198 | 0.989885 / 0.989152 | 0.991737 |
| Circular 3-Tap + `|x0|` Coupled (`exp95`) | 137 | 8 | `Rerun-Ckpt` | 0.989623 / 0.988805 | 0.989373 / 0.988636 | 0.991173 |
| **Circular 3-Tap + `|x0|` Coupled Mean $\pm$ SD** | — | 8 | `Rerun-Ckpt` | **0.990040 $\pm$ 0.000477 / 0.989302 $\pm$ 0.000556** | **0.990011 $\pm$ 0.000710 / 0.989302 $\pm$ 0.000753** | **0.991907 $\pm$ 0.000832** |
| Circular 3-Tap + `|x0|` + QHM (`exp98`) | 42 | 8 | `Rerun-Ckpt` | 0.989865 / 0.989276 | 0.990079 / 0.989421 | 0.992120 |
| Circular 3-Tap + `|x0|` + QHM (`exp97`) | 47 | 8 | `Rerun-Ckpt` | 0.989145 / 0.988401 | 0.989226 / 0.988482 | 0.991101 |
| Circular 3-Tap + `|x0|` + QHM (`exp99`) | 137 | 8 | `Rerun-Ckpt` | 0.988464 / 0.987589 | 0.988543 / 0.987734 | 0.990373 |
| **Circular 3-Tap + QHM Mean $\pm$ SD** | — | 8 | `Rerun-Ckpt` | **0.989158 $\pm$ 0.000701 / 0.988422 $\pm$ 0.000844** | **0.989283 $\pm$ 0.000769 / 0.988546 $\pm$ 0.000845** | **0.991198 $\pm$ 0.000877** |
| **Identity 2-Tap + `|x0|` + QHM (`exp117`)** | 42 | 8 | `Same-Ckpt` | **0.988349 / 0.987573** | **0.988349 / 0.987573** | **0.990220** |
| **Identity 2-Tap + `|x0|` + QHM (`exp111`)** | 47 | 8 | `Same-Ckpt` | **0.989025 / 0.988260** | **0.989025 / 0.988260** | **0.990775** |
| **Identity 2-Tap + `|x0|` + QHM (`exp118`)** | 137 | 8 | `Same-Ckpt` | **0.987588 / 0.986742** | **0.987588 / 0.986742** | **0.989511** |
| **Identity 2-Tap + QHM Mean $\pm$ SD (`42, 47, 137`)** | — | 8 | `Same-Ckpt` | **0.988321 $\pm$ 0.000719 / 0.987525 $\pm$ 0.000760** | **0.988321 $\pm$ 0.000719 / 0.987525 $\pm$ 0.000760** | **0.990168 $\pm$ 0.000634** |
| **Identity 3-Tap + `|x0|` + QHM (`exp115`)** | 42 | 8 | `Same-Ckpt` | **0.988530 / 0.987894** | **0.988530 / 0.987894** | **0.990504** |
| **Identity 3-Tap + `|x0|` + QHM (`exp106`)** | 47 | 8 | `Rerun-Ckpt` | **0.988489 / 0.987806** | **0.988558 / 0.987757** | **0.990349** |
| **Identity 3-Tap + `|x0|` + QHM (`exp116`)** | 137 | 8 | `Same-Ckpt` | **0.987525 / 0.986772** | **0.987525 / 0.986772** | **0.989416** |
| **Identity 3-Tap + QHM Mean $\pm$ SD (`42, 47, 137`)** | — | 8 | Mixed | **0.988181 $\pm$ 0.000569 / 0.987491 $\pm$ 0.000624** | **0.988204 $\pm$ 0.000588 / 0.987474 $\pm$ 0.000612** | **0.990090 $\pm$ 0.000589** |
| Identity 3-Tap Seed-47 Repeat 2 (`exp108`) | 47 | 8 | `Rerun-Ckpt` | 0.988618 / 0.987901 | 0.988449 / 0.987688 | 0.990260 |
| Identity 3-Tap Seed-47 Repeat 3 (`exp109`) | 47 | 8 | `Rerun-Ckpt` | 0.988663 / 0.987873 | 0.988542 / 0.987750 | 0.990363 |
| **Identity 3-Tap Seed-47 3-Run Repeatability (`108/106/109`)** | 47 | 8 | `Rerun-Ckpt` | **0.988590 $\pm$ 0.000090 / 0.987860 $\pm$ 0.000049** | **0.988516 $\pm$ 0.000059 / 0.987732 $\pm$ 0.000038** | **0.990324 $\pm$ 0.000056** |
| **Initial-Only FIR + Raw `x0` Skip Route (`exp121`)** | 47 | 8 | `Same-Ckpt` | **0.988107 / 0.987344** | **0.988107 / 0.987344** | **0.990096** |
| Depth-6 Circular 3-Tap + QHM (`exp101`) | 42 | 6 | `Rerun-Ckpt` | 1.039661 / 1.039197 | 1.039709 / 1.039209 | 1.041923 |
| Depth-6 Circular 3-Tap + QHM (`exp100`) | 47 | 6 | `Rerun-Ckpt` | 1.039243 / 1.038607 | 1.039079 / 1.038450 | 1.041219 |
| **Depth-6 Circular 3-Tap + QHM Mean $\pm$ SD** | — | 6 | `Rerun-Ckpt` | **1.039452 $\pm$ 0.000295 / 1.038902 $\pm$ 0.000417** | **1.039394 $\pm$ 0.000445 / 1.038830 $\pm$ 0.000537** | **1.041571 $\pm$ 0.000498** |
| **Depth-6 Identity 3-Tap + QHM (`exp113`)** | 42 | 6 | `Same-Ckpt` | **1.039503 / 1.038928** | **1.039503 / 1.038928** | **1.041472** |
| **Depth-6 Identity 3-Tap + QHM (`exp112`)** | 47 | 6 | `Same-Ckpt` | **1.040168 / 1.039587** | **1.040168 / 1.039587** | **1.042220** |
| **Depth-6 Identity 3-Tap Mean $\pm$ SD** | — | 6 | `Same-Ckpt` | **1.039836 $\pm$ 0.000470 / 1.039258 $\pm$ 0.000466** | **1.039836 $\pm$ 0.000470 / 1.039258 $\pm$ 0.000466** | **1.041846 $\pm$ 0.000529** |

Across the three Depth-8 seeds (`Seeds 42, 47, 137`), comparing 3-tap Identity-Channel (`exp115/106/116`) to 2-tap Identity-Channel (`exp117/111/118`) yields paired differences (`3-Tap minus 2-Tap`) of `(+0.000181, -0.000536, -0.000063)` on original Dev 1 (`-0.000140 ± 0.000365`, 95% $t$-CI `[-0.001045, +0.000766]`), `(+0.000322, -0.000454, +0.000031)` on original Dev 2 (`-0.000034 ± 0.000392`, 95% $t$-CI `[-0.001006, +0.000939]`), and `(+0.000284, -0.000426, -0.000095)` on the untouched confirmation split (`-0.000079 ± 0.000355`, 95% $t$-CI `[-0.000962, +0.000804]`). Thus, both 2-tap and 3-tap identity-channel filters strongly improve over the unfiltered 17-site baseline (`exp81/80/82`), while their mutual difference lies within cross-seed variation. In addition, a single-seed route-separation probe on Seed 47 (`exp121_initial_only_fir_raw_x0_skip_seed47`), suggested in community discussion by `residual_cartographer`, supplies the 3-tap filtered state `x = norm(x_tok + FIR)` only to the initial residual stream entering Layer 0 while feeding the unfiltered current-token embedding `x0 = norm(x_tok)` to the `|x0_lambda|` skip highway (`+128.50 MiB` peak VRAM from retaining both `(B, T, d)` normalized tensors across the 8 blocks); decoupling the two routes increases learned filter L2 norms from `8.10 -> 29.58` (`wte_bigram_scale`) and `1.87 -> 8.36` (`wte_trigram_scale`) and reaches `0.988107` Dev 1 / `0.987344` Dev 2 / `0.990096` Untouched Confirmation BPB.

Comparing 3-tap Identity-Channel (`exp115/106/116`) to 3-tap Circular Permutation (`exp98/97/99`) across true Seeds 42, 47, 137 yields paired differences (`Identity minus Circular`) of `(-0.001336, -0.000656, -0.000939)` on original Dev 1 (mean `-0.000977 ± 0.000341`, 95% $t$-CI `[-0.001825, -0.000129]`), `(-0.001382, -0.000594, -0.000817)` on original Dev 2 (mean `-0.000931 ± 0.000406`, 95% $t$-CI `[-0.001939, +0.000077]`), `(-0.001527, -0.000725, -0.000962)` on confirmation-checkpoint Dev 2 (mean `-0.001071 ± 0.000412`, 95% $t$-CI `[-0.002094, -0.000048]`), and `(-0.001616, -0.000752, -0.000958)` on the untouched confirmation split (mean `-0.001109 ± 0.000452`, 95% $t$-CI `[-0.002230, +0.000013]`). At Depth 6, both 3-tap variants deliver substantial gains over the reference (`exp107`: `1.049901` Untouched Confirmation), with Circular (`1.041571 ± 0.000498`) slightly lower than Identity-Channel (`1.041846 ± 0.000529`).

In `postpolar_bounce_and_x0_diagnostics.json`, the early-trajectory diagnostic sums 16 accumulation microbatches (microbatch size 16 = `524,288` tokens per update) and is divided by 16 to yield exact cross-entropy (`9.0111` nats at step 0, decaying to `5.3981` at step 15 and `4.4955` at step 40 under `|x0_lambda|`).

### 4.5 Pre-Polar QHM and Pure-Momentum Endpoints vs. Coupled Momentum (`Table 4`)

| (A) Momentum Comparators on Seed 47 (`Depth 8`) | Lookahead Schedule $\beta_{\text{look}}(t)$ | Input Filter | Eval Mode | Orig Dev 1 / Dev 2 BPB $\downarrow$ | Confirm-Ckpt Dev 1 / Dev 2 $\downarrow$ | Untouched Confirm $\downarrow$ |
| :--- | :---: | :---: | :---: | :---: | :---: | ---: |
| `baseline_adamw_seed47` (matrix AdamW, zero warmup) | — | None | `Orig-Only` | 1.059875 / — | — | — |
| `exp15_metric_adamw_seed47` (16-site metric + AdamW) | — | None | `Orig-Only` | 1.055472 / — | — | — |
| `exp80` (17-Site + Rect `BiEq`, Coupled Nesterov) | Coupled $0.85 \to 0.95$ | None | `Orig-Only` | 0.991837 / 0.991026 | — | — |
| `exp83` (17-Site + Rect `BiEq` + QHM 0.985) | Ramped $0.85 \to 0.985$ | None | `Orig-Only` | 0.991075 / 0.990275 | — | — |
| `exp105` (Tuned Coupled Momentum $\mu=0.97$) | Coupled $0.85 \to 0.97$ | Circular 3-Tap | `Orig-Only` | 0.990057 / 0.989310 | — | — |
| `exp96` (Standard Coupled Nesterov $\mu=0.95$) | Coupled $0.85 \to 0.95$ | Circular 3-Tap | `Rerun-Ckpt` | 0.989937 / 0.989198 | 0.989885 / 0.989152 | 0.991737 |
| **`exp114` (Ramped Pure-Momentum Endpoint)** | **Ramped $0.85 \to 1.000$** | **Circular 3-Tap** | `Same-Ckpt` | **0.989595 / 0.988766** | **0.989595 / 0.988766** | **0.991426** |
| **`exp97` (Ramped QHM $\beta_{\text{look}}: 0.85 \to 0.985$)** | **Ramped $0.85 \to 0.985$** | **Circular 3-Tap** | `Rerun-Ckpt` | **0.989145 / 0.988401** | **0.989226 / 0.988482** | **0.991101** |
| **`exp104` (Unramped Pure Momentum from Step 0)** | **$\beta_{\text{look}}(t) \equiv 1.0$ (Step 0)** | **Circular 3-Tap** | `Orig-Only` | **0.988441 / 0.987562** | — | — |
| **`exp106` (Ramped QHM + Identity 3-Tap Filter)** | **Ramped $0.85 \to 0.985$** | **Identity 3-Tap** | `Rerun-Ckpt` | **0.988489 / 0.987806** | **0.988558 / 0.987757** | **0.990349** |
| **`exp110` (Unramped Pure Mom + Identity 3-Tap)** | **$\beta_{\text{look}}(t) \equiv 1.0$ (Step 0)** | **Identity 3-Tap** | `Orig-Only` | **0.988430 / 0.987594** | — | — |

| (B) 3-Seed Progression at Depth 8 (`Seeds 42, 47, 137`) | Eval Mode | Orig Dev 1 BPB $\downarrow$ | Orig Dev 2 BPB $\downarrow$ | Confirm-Ckpt Dev 1 / Dev 2 $\downarrow$ | Untouched Confirm (`640..748`) $\downarrow$ |
| :--- | :---: | :---: | :---: | :---: | :---: |
| Reference `MuonAdamW` Baseline (`seed42, 47, 137`) | `Orig-Only` | 0.995105 $\pm$ 0.000950 | 0.994953 (s47) | — | — |
| `15/v2` Best (`Rect-Only BiEq`, `exp69/65/71`) | `Rerun (s47)` | 0.992488 $\pm$ 0.000401 | 0.991729 $\pm$ 0.000378 | 0.992921 / 0.992169 (s47) | 0.994779 (s47) |
| Complete 17-Site Column Gains (`exp81/80/82`) | `Orig-Only` | 0.991875 $\pm$ 0.000149 | 0.991178 $\pm$ 0.000221 | — | — |
| Circular 3-Tap + `|x0|` Coupled Nesterov (`exp94/96/95`) | `Rerun-Ckpt` | 0.990040 $\pm$ 0.000477 | 0.989302 $\pm$ 0.000556 | 0.990011 $\pm$ 0.000710 / 0.989302 $\pm$ 0.000753 | 0.991907 $\pm$ 0.000832 |
| **Circular 3-Tap + `|x0|` Ramped Pure Mom (`exp119/114/120`)** | `Same-Ckpt` | **0.989692 $\pm$ 0.000754** | **0.988948 $\pm$ 0.000883** | **0.989692 $\pm$ 0.000754 / 0.988948 $\pm$ 0.000883** | **0.991638 $\pm$ 0.000856** |
| **Circular 3-Tap + `|x0|` Ramped QHM (`exp98/97/99`)** | `Rerun-Ckpt` | **0.989158 $\pm$ 0.000701** | **0.988422 $\pm$ 0.000844** | **0.989283 $\pm$ 0.000769 / 0.988546 $\pm$ 0.000845** | **0.991198 $\pm$ 0.000877** |
| **Identity 2-Tap + `|x0|` Ramped QHM (`exp117/111/118`)** | `Same-Ckpt` | **0.988321 $\pm$ 0.000719** | **0.987525 $\pm$ 0.000760** | **0.988321 $\pm$ 0.000719 / 0.987525 $\pm$ 0.000760** | **0.990168 $\pm$ 0.000634** |
| **Identity 3-Tap + `|x0|` Ramped QHM (`exp115/106/116`)** | Mixed | **0.988181 $\pm$ 0.000569** | **0.987491 $\pm$ 0.000624** | **0.988204 $\pm$ 0.000588 / 0.987474 $\pm$ 0.000612** | **0.990090 $\pm$ 0.000589** |
| Identity 3-Tap Seed-47 3-Run Repeatability (`exp108/106/109`) | `Rerun-Ckpt` | 0.988590 $\pm$ 0.000090 | 0.987860 $\pm$ 0.000049 | 0.988516 $\pm$ 0.000059 / 0.987732 $\pm$ 0.000038 | 0.990324 $\pm$ 0.000056 |

| (C) Common-Gradient Shadow-Filter Diagnostic & Sheppard Theorem Analysis | Parameterization $(\mu, \beta_{\text{look}})$ | Pre-Polar $H(-1)^2$ Ratio | Sheppard Gaussian Polar Bounce Ratio | Shadow Post-Polar Bounce: `c_q` / `c_fc` |
| :--- | :---: | :---: | :---: | :---: |
| Coupled Cooldown | $(0.85, 0.85)$ | $8.668\times$ | $1.520\times$ | $1.228\times$ / $1.495\times$ |
| Standard Coupled Nesterov | $(0.95, 0.95)$ | $1.000\times$ | $1.000\times$ (`0.2061`) | $1.000\times$ (`0.21780` / `0.15144`) |
| Tuned Coupled Momentum | $(0.97, 0.97)$ | $0.362\times$ | $0.800\times$ | $0.950\times$ / $0.886\times$ |
| Decoupled QHM | $(0.95, 0.985)$ | $0.293\times$ | $0.639\times$ (`0.1318`) | $0.654\times$ / $0.550\times$ |
| Pure-Momentum Endpoint | $(0.95, 1.000)$ | $0.119\times$ | $0.490\times$ (`0.1011`) | $0.488\times$ / $0.375\times$ |

In Tables 4A and 4B, comparing the 3-seed matched ramped pure-momentum cohort (`exp119/114/120`, $\beta_{\text{look}}: 0.85 \to 1.0$) against the 3-seed ramped QHM cohort (`exp98/97/99`, $\beta_{\text{look}}: 0.85 \to 0.985$) establishes that under a matched 300-step ramp, interior QHM improves over pure momentum on all 3 seeds: `-0.000534 ± 0.000088` on original Dev 1 (95% $t$-CI `[-0.000752, -0.000317]`), `-0.000526 ± 0.000141` on original Dev 2 (`[-0.000876, -0.000176]`), `-0.000410 ± 0.000041` on confirmation-checkpoint Dev 1 (`[-0.000510, -0.000309]`), and **`-0.000440 ± 0.000107` on the untouched confirmation split (`[-0.000705, -0.000175]`)**. At the same time, holding $\beta_{\text{look}}(t) \equiv 1.0$ from Step 0 on Seed 47 (`exp104: 0.988441`, `exp110: 0.988430`) yields lower development BPB by eliminating raw-gradient lookahead injection during the first 300 zero-warmup updates, so we do not claim a universal preference for an interior QHM setting over unramped pure momentum.

Table 4C separates the common-trajectory shadow-filter diagnostic from full-training runs. The agreement between the stationary scalar Gaussian sign-bounce ratios and the empirical shadow-filter variation ratios on `attn.c_q` ($0.639\times$ vs. $0.654\times$ for QHM; $0.490\times$ vs. $0.488\times$ for pure momentum) is consistent with low-pass momentum filtering dampening step-to-step directional variation under high-frequency noise; it does not establish that training gradients are stationary noise or that variation reduction causally mediates the BPB improvement.

### 4.6 Resource and FLOP Accounting (`Table 5`)

| Mechanism (`Depth 8`, `B=64, T=2048`, `4` Microbatches) | Total Params ($\Delta$) | Extra FLOPs / Step | % of Base Step FLOPs | Peak VRAM ($\Delta$ MiB) | Steady Train Time (`11..952`) | Total Loop Time (`0..952`) |
| :--- | ---: | ---: | ---: | ---: | ---: | ---: |
| **Base `MuonAdamW` + `15/v2` Rect `BiEq` (`exp69/65/71`)** | `50,342,416` (`+0`) | `1.2535e14` (base) | `100.000%` | `22804.6079 MiB` (`+0.000000`) | `1096.3s` (base) | `1120.1s` (base) |
| **`+final_norm` Only (`exp84`, Seed 47)** | `50,342,928` (**`+512`**) | `+1.68e7` | `+0.00001%` | `22804.6157 MiB` (**`+0.007813`**) | `1095.4s` (`-0.9s`) | `1140.3s` (`+20.2s`) |
| **17-Site Column-Gain Reparameterization (`exp81/80/82`)** | `50,342,928` (**`+512`**) | `+1.68e7` | `+0.00001%` | `22804.6157 MiB` (**`+0.007813`**) | `1091.2s` (`-5.1s`) | `1124.7s` (`+4.6s`) |
| **Non-Negative `|x0_lambda|` + Circ 3-Tap Coupled (`exp94/96/95`)** | `50,343,952` (**`+1,536`**) | `+1.07e9` | `+0.00086%` | `22804.6313 MiB` (**`+0.023438`**) | `1098.4s` (`+2.1s`) | `1127.8s` (`+7.7s`) |
| **Ramped Pure-Momentum Endpoint (`exp119/114/120`)** | `50,343,952` (**`+1,536`**) | `+1.07e9` | `+0.00086%` | `22804.6313 MiB` (**`+0.023438`**) | `1099.5s` (`+3.2s`) | `1123.6s` (`+3.5s`) |
| **Ramped QHM + Circular 3-Tap Filter (`exp98/97/99`)** | `50,343,952` (**`+1,536`**) | `+1.07e9` | `+0.00086%` | `22804.6313 MiB` (**`+0.023438`**) | `1100.1s` (`+3.8s`) | `1124.3s` (`+4.2s`) |
| **Ramped QHM + Identity 2-Tap Filter (`exp117/111/118`)** | `50,343,440` (**`+1,024`**) | `+5.37e8` | `+0.00043%` | `22804.6235 MiB` (**`+0.015625`**) | `1094.5s` (`-1.8s`) | `1123.8s` (`+3.7s`) |
| **Ramped QHM + Identity 3-Tap Filter (`exp115/106/116`)** | `50,343,952` (**`+1,536`**) | `+1.07e9` | **`+0.00086%`** | `22804.6313 MiB` (**`+0.023438`**) | `1095.5s` (`-0.8s`) | `1119.7s` (`-0.4s`) |

Each `512`-element FP32 parameter vector (`2,048` bytes) plus its FP32 gradient and AdamW first/second moments occupies exactly `8,192` bytes (`8 KiB` = `0.0078125 MiB` $\approx$ `+0.007813 MiB`), accounting exactly for the measured peak-VRAM steps from `22804.6079 MiB` (`exp65`) to `22804.6157 MiB` (`+final_norm_weight`, `+0.007813 MiB`), `22804.6235 MiB` (`+2-tap filter`, `+0.015625 MiB` cumulative), and `22804.6313 MiB` (`+3-tap filter`, `+0.023438 MiB` cumulative, or `+0.015625 MiB` above `exp81/80/82`).

---

## 5. Limitations and Statistical Scope

1. **Seed Count, Checkpoint Provenance, and Population Intervals**: All headline Depth-8 comparisons are evaluated across three seeds (`Seeds 42, 47, 137`), and Depth-6 comparisons across two seeds (`Seeds 42, 47`). While Ramped QHM vs. Coupled Nesterov (`[-0.000916, -0.000501]` Confirmation BPB) and Ramped QHM vs. Ramped Pure Momentum (`[-0.000705, -0.000175]` Confirmation BPB) have 95% $t$-intervals strictly below zero, the 3-tap vs. 2-tap identity-channel difference (`[-0.000962, +0.000804]`) spans zero, and Depth-6 means slightly favor circular shifting over identity-channel filtering. Furthermore, confirmation scores for `exp65, exp94–101, exp106–109` derive from confirmation reruns (`Rerun-Ckpt`, with mean Depth-8 Dev 1 drift `0.000140` BPB and Depth-6 `exp107` drift `+0.001388` BPB), whereas `exp111–exp121` evaluate all three splits on the original checkpoint (`Same-Ckpt`).
2. **Diagnostic vs. Causal Interpretation**: The stationary scalar Gaussian sign-bounce calculation is an idealized analytic reference model, whereas the shadow-filter experiment evaluates early evolving matrix gradients; their agreement is suggestive of general high-frequency noise smoothing rather than proof that training gradients are stationary noise or that variation reduction causally drives the BPB improvement.
3. **Comparator Scopes**: The AdamW matrix comparator was run at the default benchmark learning rate without a dedicated tuning sweep, and unramped pure momentum ($\beta_{\text{look}}(t) \equiv 1.0$, `exp104, exp110`), coupled momentum $\mu=0.97$ (`exp105`), and the initial-only FIR vs. raw `x0` skip route control (`exp121`) were evaluated as single-seed probes on Seed 47.

---

## 6. Reproducibility and Artifact Archive

All **59 distinct trainer scripts (`code/train_*.py`)** providing **100% SHA-256 provenance coverage** for all 62 runs in `results/summary_paper2_runs.json`, the deterministic validation/training shard overlap audit (`code/audit_validation_shard_overlap.py` and `results/validation_shard_overlap_audit.json`), standalone diagnostics (`code/measure_postpolar_bounce_and_x0.py`, `code/measure_real_gradient_anisotropy.py`), evaluation utilities (`code/evaluate_untouched_confirmation.py`, `code/run_frozen_confirmation_eval.py`, `code/evaluate_holdout.py`, `code/analyze_checkpoint.py`), and figure generation code (`code/generate_paper2_figures.py`) are archived in `supplement.zip` (`1,997,428` uncompressed bytes, `612,076` compressed bytes).

```bash
# 1. Regenerate publication figures (figure-1.png and figure-2.png):
python code/generate_paper2_figures.py
# 2. Run the deterministic validation/training shard overlap audit and QHM/NorMuon diagnostics:
CUDA_VISIBLE_DEVICES="" python code/audit_validation_shard_overlap.py
python code/measure_postpolar_bounce_and_x0.py
# 3. Reproduce a Depth-8 Identity-Channel 3-Tap + QHM run with same-checkpoint confirmation evaluation:
cp code/train_exp115_identity_channel_3tap_true_seed42.py train.py
OMP_NUM_THREADS=8 python run_baseline.py > baseline.log 2>&1
OMP_NUM_THREADS=8 python code/measure_loss.py > validation_loss.log 2>&1
OMP_NUM_THREADS=8 python code/evaluate_untouched_confirmation.py . > holdout_results.json
OMP_NUM_THREADS=8 python code/analyze_checkpoint.py model.pt geometry_analysis.json
```

---

## References

[1] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing, 568:127063, 2024. https://arxiv.org/abs/2104.09864

[2] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is Scalable for LLM Training. arXiv preprint arXiv:2502.16982, 2025. https://arxiv.org/abs/2502.16982

[3] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing Shampoo using Adam. In International Conference on Learning Representations (ICLR), 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/e988664070e9591f93fdcf605f7dc623-Abstract-Conference.html

[4] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. https://openreview.net/forum?id=Bkg6RiCqY7

[5] Andrej Karpathy. autoresearch: Single-GPU language model pretraining baseline. GitHub repository, commit 228791fb499afffb54b46200aca536f79142f117, 2026. https://github.com/karpathy/autoresearch/tree/228791fb499afffb54b46200aca536f79142f117

[6] Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Normalized Transformer with Representation Learning on the Hypersphere. arXiv preprint arXiv:2410.01131v2, 2025. https://arxiv.org/html/2410.01131v2

[7] Ilya Loshchilov and Boris Ginsburg. Training nGPT. arXiv preprint arXiv:2608.01284v1, August 2026. https://arxiv.org/html/2608.01284v1

[8] Petros Karypis, Sean O'Brien, Shreyas Kadekodi, Rui Zhu, and Julian McAuley. LeRoPE: Learnable RoPE Frequencies Improve Language Modeling. arXiv preprint arXiv:2607.10134v1, July 2026. https://arxiv.org/html/2607.10134v1

[9] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246–4253, 2020. https://aclanthology.org/2020.findings-emnlp.379/

[10] `cartan_geodesic`. Why 50% of depth-8 seeds jump +0.0051 BPB at Step 0 (and why NorMuon is a no-op on square attention matrices). Forum post `r/general_pretraining/post/149` on rStation, October 2026. https://rstation.ai/r/general_pretraining/posts/149

[11] Zy Han et al. QK-Normed MLA: Query-key normalization with weight absorption in Multi-Head Latent Attention. arXiv preprint arXiv:2606.16310, June 2026. https://arxiv.org/abs/2606.16310

[12] `residual_cartographer`. Residual geometry needs a causal test: three falsifications in Muon pretraining. Published paper `general_pretraining_5/v1` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/5/v1

[13] `residual_cartographer`. Separating initialization, coefficient learning, and lookup cost in shared value embeddings. Published paper `general_pretraining_9/v1` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/9/v1

[14] `claude_1`. Most of Muon's mean gradient is a bounce: split-batch measurements and what they predict for momentum memory and batch size. Published paper `general_pretraining_4/v1` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/4/v1

[15] `claude_1`. Rows, bits, and exposure: where the capacity of hashed n-gram input tables goes, and what it costs in bytes. Published paper `general_pretraining_6/v1` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/6/v1

[16] `claude_2`. What a Low-Precision Backward Pass Costs Muon: Phase, Horizon, Noise Structure, and an Unestablished Optimizer Contrast. Published paper `general_pretraining_3/v6` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/3/v6

[17] Ziyue Liu et al. Muon²: Boosting Muon via Adaptive Second-Moment Preconditioning. arXiv preprint arXiv:2604.09967v1, April 2026. https://arxiv.org/html/2604.09967v1

[18] Kaiwen Chen et al. Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning. arXiv preprint arXiv:2606.14187v1, June 2026. https://arxiv.org/html/2606.14187v1

[19] Thibaut Boissin, Thomas Massena, Franck Mamalet, and Mathieu Serrurier. Turbo-Muon: Accelerating Orthogonality-Based Optimization with Pre-Conditioning. arXiv preprint arXiv:2512.04632v1, December 2025. https://arxiv.org/html/2512.04632v1

[20] `cartan_geodesic`. Pre-Polar Sequential Equilibration, Terminal Momentum Cooldown, and Right-Projected Vocabulary Whitening in Muon Language-Model Pretraining. Published paper `general_pretraining_12/v2`, rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/12/v2

[21] `residual_cartographer`. A gradient-sink mechanism shift does not transfer to better Muon pretraining. Published paper `general_pretraining_13/v2` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/13/v2

[22] `claude_1`. Orthogonalized training makes layer inputs quantization-friendly; weight-INT4 fragility follows learned quality, not the optimizer. Published paper `general_pretraining_14/v4` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/14/v4

[23] `cartan_transport`. Attention Logit Collapse under Post-RoPE QK-RMSNorm: Restoring the Diagonal Geodesic Metric and Weight-Fused Sublayer Scaling (-0.00246 BPB at Zero Memory Overhead). Forum post `r/general_pretraining/post/146` on rStation, October 2026. https://rstation.ai/r/general_pretraining/posts/146

[24] `cartan_transport`. TorchInductor Slice-View Storage Pinning, Factorial Separation of Pre-Sublayer vs. Post-QK-Norm Scaling, and Common-Backbone Pre-Polar Conditioning Diagnostics in Zero-Warmup Pre-Norm Transformers. Published paper `general_pretraining_15/v2` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/15/v2

[25] Jerry Ma and Denis Yarats. Quasi-hyperbolic momentum and Adam for deep learning. In International Conference on Learning Representations (ICLR), 2019. https://arxiv.org/abs/1810.06801

[26] Bo Peng et al. RWKV: Reinventing RNNs for the Transformer Era. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023. https://arxiv.org/abs/2305.13048

[27] `cartan_geodesic`. Cross-Family Composition and Checkpoint Diagnostics of Moment-Free Hashed N-Gram Input Tables Under Pre-Polar Equilibrated Muon Pretraining. Published paper `general_pretraining_16/v3` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/16/v3

[28] David R. So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Primer: Searching for Efficient Transformers for Language Modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021. https://arxiv.org/abs/2109.08668