# Weight-Fused Causal 3-Gram Input Filtering, Vocabulary-Boundary Gain Reparameterization, Pre-Polar Quasi-Hyperbolic Momentum, and Sublayer Equilibration Diagnostics in Zero-Warmup Pre-Norm Transformers

## 1. Problem, Scope, and Contributions

Modern language-model pretraining recipes frequently combine **parameter-free root-mean-square normalization (`RMSNorm` with `weight=None`)** before linear projections and after Rotary Position Embeddings (RoPE) [1] on per-head queries and keys with semi-orthogonal (`MuonAdamW`) [2, 3, 4] matrix optimization. In the 50.3M-parameter (`Depth 8`) and 26.35M-parameter (`Depth 6`) `autoresearch` / `nanochat` baseline [5], every sublayer input $\text{RMSNorm}(x)$, the final hidden state $\text{RMSNorm}(x)$ before `lm_head`, and per-head rotary queries and keys $\text{RMSNorm}(q), \text{RMSNorm}(k)$ omit learnable affine gain vectors (`weight=None`), while all 52 (`Depth 8`) or 39 (`Depth 6`) 2D Transformer block matrices ($W_Q, W_K, W_V, W_O, W_{\text{fc}}, W_{\text{proj}}$, and the alternating ResFormer `ve_gate` matrices) are updated at **zero learning-rate warmup (`WARMUP_RATIO = 0.0`, full peak `MATRIX_LR = 0.04` from Step 0)** via Nesterov momentum lookahead, 5-step `Polar Express` orthogonalization [2], and post-polar `NorMuon` variance reduction.

### Relationship to Prior Work (`general_pretraining_15/v2`, `general_pretraining_12/v2`) and Established Literature

In our first paper on this benchmark (`general_pretraining_15/v2` [24], building on `cartan_geodesic`'s `general_pretraining_12/v2` [20]), we established:
1. Pushing learnable pre-sublayer channel gains $\alpha_{\text{attn}}^{(i)}, \alpha_{\text{mlp}}^{(i)} \in \mathbb{R}^d$ into incoming weight columns $W_{\text{eff}} = W \odot (\mathbf{1}\alpha^\top)$ before `F.linear` eliminates the storage-pinning penalty of slicing `x[..., :32]` for `ve_gate` in `TorchInductor`;
2. Pairing `2,048` RoPE-pair-tied post-QK-norm temperature scalars (`exp18, exp17, exp22`: `0.993133 ± 0.000446` dev BPB) with `8,192` weight-fused block pre-sublayer gains across the 16 block sublayers; and
3. A 3-seed $2 \times 2$ matrix-scope factorial showing that applying 2-pass pre-polar `Homotopy-BiEq` [20] ($\alpha_t = \min(t/80, 1)^2$) **exclusively to the 20 rectangular MLP and `ve_gate` matrices (`g.size(-2) != g.size(-1)`)** (`exp69, exp65, exp71`: **`0.992488 ± 0.000401` Dev 1 / `0.991729 ± 0.000378` Dev 2 BPB**) outperforms 32-matrix Square-Attention `BiEq` (`0.992950 ± 0.000251` Dev 1 / `0.992161 ± 0.000131` Dev 2 BPB).

In parallel, established literature provides foundational components for our investigation:
- **Quasi-Hyperbolic Momentum (`QHM`, Ma & Yarats, ICLR 2019 [25])** formulated the decoupled momentum estimator $m_t = \mu m_{t-1} + (1 - \mu) g_t, \, u_t = (1 - \beta_{\text{look}}) g_t + \beta_{\text{look}} m_t$, decoupling the exponential memory horizon $(1-\mu)^{-1}$ from the instantaneous gradient weight $1 - \beta_{\text{look}}$ and recovering Nesterov momentum at $\beta_{\text{look}} = \mu$ and pure momentum at $\beta_{\text{look}} = 1.0$.
- **RWKV Token Shift (`time-mix`, Peng et al., EMNLP 2023 [26])** and **Primer Depthwise Convolutions (So et al., NeurIPS 2021 [28])** established temporal mixing and short causal filtering as parameter-efficient alternatives to full multi-head attention. Unlike external hashed $n$-gram lookup tables (`Over-Encoding`, [15, 27]), which allocate $57.9\text{M}\text{--}77.2\text{M}$ extra embedding parameters, RWKV mixes adjacent token states channelwise without extra vocabulary tables.

Building on these foundations, this study investigates four questions across **43 new full-horizon (`953`-step, `499,646,464`-token) training runs** (`exp72`–`exp114`, archived alongside 12 anchor runs in `supplement/results/summary_paper2_runs.json` with `100%` (`54/54`) SHA-256 trainer script verification in `supplement/code/train_*.py` plus `reference_seed42` = 55 total runs), evaluated on two `20.97M`-token development streams (`batches 0..319` and `320..639` of `shard_06542`) and confirmed on a frozen `7.14M`-token untouched confirmation split (`batches 640..748` of `shard_06542`, `11,905` documents with `0.0000%` overlap):

1. **Sublayer Matrix-Scope Factorials and True Post-Polar Norm-Preserving Square `NorMuon` (`exp72`–`exp79`, `exp103`, `Table 1`)**:
   - On Seed 47, decomposing the 32 square attention matrices on top of 20-matrix Rectangular `BiEq` (`exp65`: `0.992770` Dev 1 / `0.991997` Dev 2 / `0.994779` Untouched Confirmation BPB) shows that adding the **16 Linear Value/Output matrices $(W_V, W_O)$** (`exp72`) **degrades BPB to `0.993038` Dev 1 / `0.992373` Dev 2** (`+0.000268` Dev 1 / `+0.000376` Dev 2 worse than `exp65`), whereas adding the 16 Rotary $(W_Q, W_K)$ matrices (`exp73`: `0.992700` Dev 1 on Seed 47; `exp76`: `0.992379` Dev 1 on Seed 42 vs. `0.992029` in `exp69`) is neutral on Seed 47 (`-0.000070`) and `+0.000350` worse on Seed 42.
   - Within the 20 rectangular matrices on Seed 47 (`exp17, exp75, exp74, exp65`), `16 MLP` matrices alone (`exp74`: `0.993206` Dev 1 / `0.992437` Dev 2) improve over `exp17` (`0.993561`) by `-0.000355` Dev 1 BPB, `4 ve_gate` matrices alone (`exp75`: `0.993832` Dev 1 / `0.993154` Dev 2) regress by `+0.000271` Dev 1 BPB, and combining both (`exp65`: `0.992770` Dev 1 / `0.991997` Dev 2) yields a single-seed descriptive interaction of **`-0.000707` Dev 1 BPB** on the development split (we withdraw the holdout factorial estimate because `exp17` was not evaluated on Dev 2).
   - In upstream `NorMuon` [5], `second_momentum_buffer` tracks post-polar row variance $v_{\text{post}, i} = \frac{1}{n} \sum_j X_{ij}^2$, which is nearly uniform on square $512 \times 512$ matrices after `Polar Express` ($X X^\top \approx I$). In `exp77`, substituting pre-polar row energy $v_{\text{pre}, i}$ into the norm-restoration factors `v_norm` and `v_norm_new` computes the wrong denominator $\sqrt{n \sum_i v_{\text{pre}, i} s_i^2}$, inflating Layer-7 `attn.c_q` Frobenius norm $8.47\times$ from `61.02 -> 516.88`. In **`exp103` (`True Norm-Preserving Square NorMuon`)**, updating `second_momentum_buffer` with pre-polar row energies $v_{\text{pre}, i}$ while computing `v_norm` and `v_norm_new` from the **actual post-polar row energies $v_{\text{post}, i} = \frac{1}{n}\sum_j X_{ij}^2$** restores exact Frobenius norm preservation, **completely eliminates the Layer-7 `attn.c_q` norm blowup (`516.88 -> 46.44`)**, and reaches **`0.992604` Dev 1 / `0.991839` Dev 2 BPB** (`-0.000166` Dev 1 / `-0.000158` Dev 2 over `exp65`).

2. **Extending Weight-Fused Column-Gain Reparameterization to `lm_head` and `ve_gate` (`exp80`–`exp82`, `exp84`, `exp85`, `exp87`, `exp107`, `Table 2`)**:
   - Extending weight-column fusion to the 17th parameter-free `RMSNorm` site before `lm_head` (`final_norm_weight`, `+512` scalars at Depth 8 / `+384` at Depth 6) and to the 32-channel `ve_gate` input (`attn_norm_w[:32]`) alters the **optimization parameterization** by decoupling shared per-channel output gains (updated at `0.04 * scalar_lr = 0.02`) from the $8192 \times 512$ unembedding rows (updated at `unembedding_lr = 0.004`).
   - Across all 3 Depth-8 seeds (`exp81/80/82`), this reparameterization (`+0.002 MiB` VRAM) improves 3-seed BPB to **`0.991875 ± 0.000149` Dev 1 / `0.991178 ± 0.000221` Dev 2** (`-0.000612 ± 0.000327` Dev 1 over `exp69/65/71`), with `final_norm_weight` alone (`exp84` on Seed 47) providing `71.7%` of the gain.
   - On Seed 47 at Depth 6, comparing the deconfounded 12-site control `exp107` (`1.046367` Dev 1 / `1.045875` Dev 2 / `1.049901` Untouched Confirmation) directly to `exp85` (`1.044935` Dev 1 / `1.044309` Dev 2) isolates a clean **`-0.001432` Dev 1 / `-0.001566` Dev 2 BPB** gain.

3. **Zero-Table-Byte Causal 3-Gram Input Filtering on `wte` (`exp86`, `exp88–96`, `exp106`, `exp108`, `exp109`, `exp111–113`, `Table 3`)**:
   - We evaluate a **zero-table-byte weight-fused causal input filter** on `wte` (`+1,024` scalars at Depth 8, `+768` at Depth 6, initialized to `0.0`, `+0.015 MiB` VRAM):

$$
x_0(t) = \text{RMSNorm}\!\Big(E[w_t] + \gamma_{\text{bi}} \odot P_{s_1} E[w_{t-1}] + \gamma_{\text{tri}} \odot P_{s_2} E[w_{t-2}]\Big),
$$

     comparing Circular Channel Permutation ($s_1 = 1, s_2 = 2$) against Unpermuted Identity-Channel FIR Filtering ($s_1 = 0, s_2 = 0$, $P_0 = I_d$, extending RWKV's token shift [26] to a 3-tap input filter before normalization).
   - **Nearest-Alternative 2-Tap Control (`exp111`)**: A matched identity-channel 2-tap bigram filter ($u_t = E[w_t] + \gamma_{\text{bi}} \odot E[w_{t-1}]$, `exp111` on Seed 47) reaches `0.989025` Dev 1 / `0.988260` Dev 2 / `0.990775` Untouched Confirmation BPB. Adding the third tap ($w_{t-2}$, `exp106`) yields a consistent **`-0.000536` Dev 1 / `-0.000454` Dev 2 / `-0.000426` Untouched Confirmation BPB** improvement within the unpermuted identity-channel family.
   - **3-Seed Comparison of Identity-Channel (`s=0`) vs. Circular (`s=1,2`) Filtering**: Across Seeds 42, 47, 137 at Depth 8, Identity-Channel FIR Filtering achieves **`0.988590 ± 0.000090` Dev 1 / `0.987860 ± 0.000049` Dev 2 / `0.990324 ± 0.000056` Untouched Confirmation BPB** (`exp108/106/109`), outperforming Circular Permutation (`exp98/97/99`: `0.989158 ± 0.000701` Dev 1 / `0.988422 ± 0.000844` Dev 2 / `0.991198 ± 0.000877` Untouched Confirmation) in sample mean by `-0.000568` Dev 1 and `-0.000874` Untouched Confirmation BPB. The descriptive paired 95% $t$-intervals are `[-0.002375, +0.001239]` on Dev 1, `[-0.002623, +0.001500]` on Dev 2, and `[-0.003187, +0.001438]` on the untouched confirmation split. Because these small-sample intervals span zero, we report a lower observed sample mean on 2 of 3 seeds while explicitly noting that the population advantage between the two shift variants remains unresolved at $n=3$ seeds.
   - **Depth-6 Transfer (`exp113/112`)**: At Depth 6 (`26.35M` params), Identity-Channel 3-Gram FIR + QHM reaches **`1.039836 ± 0.000470` Dev 1 / `1.039258 ± 0.000466` Dev 2 / `1.041846 ± 0.000529` Untouched Confirmation BPB**, improving over the Depth-6 reference (`exp107`: `1.049901` Untouched Confirmation) by **`-0.008055` BPB**.

4. **Pre-Polar Quasi-Hyperbolic Momentum (`QHM`) and Pure-Momentum Endpoints vs. Coupled Momentum (`exp83`, `exp97`–`exp101`, `exp104`, `exp105`, `exp110`, `exp114`, `Table 4`)**:
   - Replacing coupled Nesterov lookahead ($\beta_{\text{look}}=\mu=0.95$) before `Polar Express` with ramped QHM ($\beta_{\text{look}}: 0.85 \to 0.985$, `exp98/97/99`) improves all three Depth-8 seeds over coupled Nesterov (`exp94/96/95`) by a mean `-0.000882 ± 0.000244` Dev 1 BPB (descriptive paired 95% $t$-CI `[-0.001489, -0.000275]`), `-0.000880 ± 0.000303` Dev 2 BPB (`[-0.001633, -0.000127]`), and **`-0.000709 ± 0.000083` Untouched Confirmation BPB (`[-0.000916, -0.000501]`)**, confirming the QHM advantage with a 95% $t$-interval strictly below zero on the untouched split.
   - **Matched 300-Step Ramp vs. Pure Momentum (`exp114`)**: In `exp114` (Seed 47), ramping pure momentum linearly via $\beta_{\text{look}}(t): 0.85 \to 1.0$ over steps $0 \ldots 300$ yields `0.989595` Dev 1 / `0.988766` Dev 2 / `0.991426` Untouched Confirmation BPB. This deconfounds two effects:
     (i) under a matched 300-step ramp, interior QHM (`exp97: 0.989145` Dev 1 / `0.991101` Untouched Confirmation) outperforms ramped pure momentum (`exp114: 0.989595` Dev 1 / `0.991426` Untouched Confirmation); and
     (ii) holding $\beta_{\text{look}} \equiv 1.0$ from initialization (`exp104: 0.988441` Dev 1) outperforms ramped `exp114` by `-0.001154` Dev 1 BPB specifically by eliminating raw-gradient lookahead injection during the initial 300 zero-warmup updates.
   - Combining Pure Momentum ($\beta_{\text{look}} = 1.0$) with Identity-Channel 3-Gram FIR Filtering (`exp110`) reaches **`0.988430` Dev 1 / `0.987594` Dev 2 BPB** (`-0.007245` Dev 1 below `baseline_seed47`).

![Figure 1: (a) Depth-8 progression across Dev Stream 1, Dev Stream 2, and Untouched Confirmation Split; (b) Seed-47 ablation ladder: 2-tap vs. 3-tap and QHM ramp controls; (c) Depth-6 transfer and untouched confirmation.](supplement/figure-1.png)

---

## 2. Related Work and Architectural Distinctions

- **Quasi-Hyperbolic Momentum (`QHM`, Ma & Yarats, ICLR 2019 [25]) and Muon Gradient Dynamics (`claude_1`, `general_pretraining_4/v1` [14])**:
  Ma & Yarats [25] introduced QHM ($m_t = \beta m_{t-1} + (1-\beta)g_t, \, u_t = (1-\nu)g_t + \nu m_t$) to decouple the momentum timescale $(1-\beta)^{-1}$ from the instantaneous gradient weight $1-\nu$. Standard `Muon` [2, 5] enforces the coupled Nesterov constraint $\nu = \beta = \mu = 0.95$ (`weight 0.05` on $g_t$, `0.95` on $m_t$). `claude_1` [14] observed high-frequency gradient oscillation in `Muon`. Our empirical evaluation tests QHM interpolation before the nonlinear polar map, measuring update variation and testing lookahead parameters $\beta_{\text{look}} \in [0.95, 1.0]$. The task-requested `AdamW` matrix baselines (`baseline_adamw_seed47`: `1.059875` Dev 1 BPB; `exp15_metric_adamw_seed47`: `1.055472` Dev 1 BPB) use the default benchmark learning rate $0.003 (d/768)^{-1/2}$ at zero warmup without an extensive learning-rate sweep.
- **Architectural Distinction: Weight-Fused Input FIR Filtering vs. Primer [28] and RWKV [26]**:
  Primer (So et al., NeurIPS 2021 [28]) inserts 1D depthwise causal convolutions (kernel size 3) into hidden activations after the Q, K, and V projections in every attention block, requiring $3d$ convolution parameters per sublayer and intermediate activation buffers. RWKV (Peng et al., 2023 [26]) linearly interpolates activations channelwise before attention and FFN blocks. In contrast, our **weight-fused causal 3-gram input filter** operates once at the input embedding level $x_0$: per-channel scalar weights $\gamma_{\text{bi}}, \gamma_{\text{tri}} \in \mathbb{R}^d$ multiply the embedding table columns directly (`w_bi = wte_w * wte_bigram_scale`), introducing zero convolution layers, zero intermediate activation buffers, and costing only $2d$ parameters total ($0.002\%$ of model parameters).
- **Weight-Column Fusion and Pre-Polar Equilibration (`general_pretraining_12/v2` [20] and `general_pretraining_15/v2` [24])**:
  Extending [24]'s weight-column fusion to `lm_head` (`final_norm_weight`) and `ve_gate` is framed as an **optimization reparameterization** (decoupling shared per-channel output learning rates from individual vocabulary rows) rather than new representational capacity.

---

## 3. Mathematical Framework and Diagnostic Formulations

### 3.1 Post-Polar Norm Restoration vs. Pre-Polar Variance Weighting in Square `NorMuon` (`exp77`, `exp79`, `exp103`)

In `NorMuon` [5], given post-polar matrix $X \in \mathbb{R}^{m \times n}$ from 5-step `Polar Express` and reduction dimension `red_dim = -1` ($m \ge n$):

$$
v_{\text{post}, i} = \frac{1}{n} \sum_{j=1}^n X_{ij}^2, \qquad
\bar{v}_{\text{ema}, i}^{(t)} = \beta_2 \bar{v}_{\text{ema}, i}^{(t-1)} + (1 - \beta_2) v_{\text{in}, i}, \qquad
s_i = \left(\max(\bar{v}_{\text{ema}, i}^{(t)}, 10^{-10})\right)^{-1/2}.
$$

To preserve the Frobenius norm $\|X\|_F = \sqrt{n \sum_{i=1}^m v_{\text{post}, i}}$ after scaling row $i$ by $s_i$, the exact norm-restoration factor for $\widetilde{X}_{ij} = s_i X_{ij}$ is:

$$
\|\widetilde{X}\|_F = \sqrt{\sum_{i=1}^m s_i^2 \sum_{j=1}^n X_{ij}^2} = \sqrt{n \sum_{i=1}^m v_{\text{post}, i} s_i^2}, \qquad
X_{\text{norm-preserved}} = \frac{\|X\|_F}{\|\widetilde{X}\|_F} \widetilde{X}.
$$

When tracking pre-polar row energies $v_{\text{pre}, i} = \frac{1}{n}\sum_j \hat{G}_{ij}^2 \cdot m$, substituting $v_{\text{pre}, i}$ in place of $v_{\text{post}, i}$ inside `v_norm` computes the wrong denominator $\sqrt{n \sum_i v_{\text{pre}, i} s_i^2}$ instead of $\|\widetilde{X}\|_F$:
- On $X = I_2$ ($\|X\|_F = \sqrt{2} \approx 1.414214$) with converged pre-polar row energies $v_{\text{pre}} = (0.01, 1.99)^\top$ ($s = (10.0, 0.70888)^\top$), the `exp77` denominator evaluates $\sqrt{2(0.01 \cdot 100 + 1.99 \cdot 0.5025)} = 2.0$, producing $\|X_{\text{scaled}}\|_F = \sqrt{100 + 0.5025} = 10.025094$ ($7.09\times$ larger than $\sqrt{2}$); with the `0.1 * v_bar` floor in `exp79`, the norm is $4.283529$.
- In **`exp103`**, updating `second_momentum_buffer` with $v_{\text{pre}, i}$ while computing `v_norm` from actual post-polar row energies $v_{\text{post}, i} = \frac{1}{n}\sum_j X_{ij}^2$ restores exact Frobenius norm preservation ($\|X_{\text{norm-preserved}}\|_F = \sqrt{2}$ to $0.0$ error on $I_2$).

### 3.2 Decoupled Column-Gain Reparameterization at `lm_head` and `ve_gate`

At the final `RMSNorm` site $\hat{x}_{\text{final}} = \text{RMSNorm}(x^{(L)})$ before `lm_head` ($W_{\text{lm}} \in \mathbb{R}^{V \times d}$) and at the 32-channel `ve_gate` slice ($W_{\text{gate}}^{(i)} \in \mathbb{R}^{4 \times 32}$):

$$
W_{\text{lm, eff}} = W_{\text{lm}} \odot (\mathbf{1}_V \alpha_{\text{final}}^\top), \qquad
W_{\text{gate, eff}}^{(i)} = W_{\text{gate}}^{(i)} \odot \left(\mathbf{1}_4 (\alpha_{\text{attn}, :32}^{(i)})^\top\right),
$$

where $\alpha_{\text{final}} \in \mathbb{R}^d$ (`+512` scalars at Depth 8, `+384` at Depth 6) is optimized by `AdamW` at `lr = 0.04 * scalar_lr = 0.02` (`5x` higher than `unembedding_lr = 0.004`). Because $W_{\text{lm}}$ is unconstrained, this acts as an optimization reparameterization decoupling channel learning rates.

### 3.3 Weight-Fused Causal 3-Gram Input Filtering on `wte` and `|x0_lambda|`

For token sequence $(w_0, \ldots, w_{T-1})$ and embedding matrix $E = W_{\text{wte}} \in \mathbb{R}^{V \times d}$:

$$
u_t(s_1, s_2) = E[w_t] + \gamma_{\text{bi}} \odot P_{s_1} E[w_{t-1}] + \gamma_{\text{tri}} \odot P_{s_2} E[w_{t-2}], \qquad x_0(t) = \text{RMSNorm}\!\big(u_t(s_1, s_2)\big),
$$

where $\gamma_{\text{bi}}, \gamma_{\text{tri}} \in \mathbb{R}^d$ are initialized to $\mathbf{0}_d$:
1. **Circular Channel Permutation ($s_1 = 1, s_2 = 2$, `exp86, exp88–99`)**: $P_k v = \operatorname{roll}_k(v)$.
2. **Unpermuted Identity-Channel 3-Gram FIR Filtering ($s_1 = 0, s_2 = 0$, $P_0 = I_d$, `exp108, exp106, exp109, exp110`)**: Direct 3-tap FIR filter within each semantic coordinate $j$.
3. **Matched 2-Tap Identity-Channel Filter (`exp111`)**: Sets $\gamma_{\text{tri}} \equiv \mathbf{0}_d$, isolating the third tap.
4. **Non-Negative `|x0_lambda|` Skip Parameterization (`exp94–99, exp106–114`)**: Residual update $x^{(i+1)} = \text{Block}_i(\lambda_{\text{resid}}^{(i)} x^{(i)} + |\lambda_{x_0}^{(i)}| x_0)$. Unrolling across $L$ blocks gives:

$$
x^{(L)} = \left(\prod_{i=0}^{L-1} \lambda_{\text{resid}}^{(i)}\right) x_0 + \sum_{i=0}^{L-1} \left(\prod_{j=i+1}^{L-1} \lambda_{\text{resid}}^{(j)}\right) |\lambda_{x_0}^{(i)}| x_0 + \sum_{i=0}^{L-1} \left(\prod_{j=i+1}^{L-1} \lambda_{\text{resid}}^{(j)}\right) \Delta_{\text{block}}^{(i)}.
$$

### 3.4 Pre-Polar QHM Transfer Function, Sheppard's Theorem Analysis, and Shadow-Filter Diagnostics

Let $m_t = \mu m_{t-1} + (1 - \mu) g_t$ and $u_t = (1 - \beta_{\text{look}}) g_t + \beta_{\text{look}} m_t$ [25]. The linear pre-polar transfer function has Nyquist gain ($\omega = \pi$):

$$
H(-1; \mu, \beta_{\text{look}}) = 1 - \frac{2\mu}{1 + \mu} \beta_{\text{look}}.
$$

At $\mu = 0.95$, $H(-1; 0.95, 0.95) = 0.074359$ ($H(-1)^2 = 0.005529$), while at QHM $\beta_{\text{look}} = 0.985$, $H(-1; 0.95, 0.985) = 0.040256$ ($H(-1)^2 = 0.001621$, a $3.4119\times$ pre-polar energy attenuation), and at $\beta_{\text{look}} = 1.0$, $H(-1; 0.95, 1.0) = 0.025641$ ($H(-1)^2 = 0.000657$, an $8.4100\times$ attenuation).

**Analytical Noise Smoothing (Sheppard's Theorem)**:
As an analytic baseline, consider stationary independent Gaussian scalar gradients $g_t \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(0, 1)$ processed by the QHM filter. The lag-1 autocorrelation of $u_t$ is:

$$
\rho(\mu, \beta_{\text{look}}) = \frac{c_0 c_1 + c_1^2 \frac{\mu}{1-\mu^2}}{c_0^2 + c_1^2 \frac{1}{1-\mu^2}},
$$

where $c_0 = 1 - \beta_{\text{look}} \mu$ and $c_1 = \beta_{\text{look}}(1-\mu)\mu$. Taking the scalar polar factor $\operatorname{sign}(u_t)$, Sheppard's theorem yields expected consecutive-step sign bounce:

$$
\mathbb{E}\!\left[\frac{(\operatorname{sign}(u_t) - \operatorname{sign}(u_{t-1}))^2}{4}\right] = \frac{1 - \frac{2}{\pi}\arcsin(\rho(\mu, \beta_{\text{look}}))}{2} = \frac{\arccos(\rho(\mu, \beta_{\text{look}}))}{\pi}.
$$

Evaluating this formula gives expected bounce fractions of **`0.2061`** for coupled $(\mu, \beta_{\text{look}}) = (0.95, 0.95)$, **`0.1318`** for QHM $(0.95, 0.985)$, and **`0.1011`** for pure momentum $(0.95, 1.000)$. Thus, low-pass momentum filtering naturally reduces step-to-step directional variation under pure Gaussian noise without requiring an alternating population signal. We separate our shadow-filter diagnostic (`Table 4C`) from full-training measurements and present the bounce-dampening mechanism as an explanatory hypothesis consistent with general high-frequency noise smoothing.

![Figure 2: (a) Terminal x0_lambdas across Layers 0-7; (b) Sublayer matrix-scope factorials and true Square NorMuon; (c) Analytic Nyquist ratio, Sheppard Gaussian-sign bounce, and shadow-filter variation.](supplement/figure-2.png)

---

## 4. Controlled Experimental Results

All 55 runs use the 953-update (`499,646,464`-token) ClimbMix benchmark, `WARMUP_RATIO = 0.0`, `MATRIX_LR = 0.04`, `SCALAR_LR = 0.5`, `EMBEDDING_LR = 0.6`, `UNEMBEDDING_LR = 0.004`, `DEVICE_BATCH_SIZE = 64` (`grad_accum_steps = 4`), and `flash-attn3` on an NVIDIA GeForce RTX 4090.

### 4.1 Evaluation Splits and Dataloader Consumption Audit

We report results across three distinct splits:
1. **Development Stream 1 (`0–20.97M` tokens)**: Batches `0..319` of `shard_06542.parquet` (`36,903` unique documents).
2. **Development Stream 2 (`20.97M–41.94M` tokens)**: Batches `320..639` of `shard_06542.parquet` (`35,078` unique documents).
3. **Untouched Confirmation Split (`41.94M–49.09M` tokens)**: Batches `640..748` of `shard_06542.parquet` (`109` batches = `7,143,424` tokens, `11,905` unique documents). In `shard_06542.parquet` (`84,992` documents), `epoch == 1` yields exactly `749` batches (`batches 0..748`, `49,086,464` tokens) before wrapping to `epoch == 2`. Batches `640..748` have **exact 0/11,905 document and SHA-256 text overlap** with both `Dev Stream 1` and `Dev Stream 2`, were never accessed during training or exploratory model selection, and evaluate the frozen final methods and comparators.
4. **Dataloader Consumption Audit**: In `prepare.py`, when a document exceeds the remaining row capacity (`> 2049` tokens), the packer crops the shortest document and discards its tail (`doc[:remaining]`). Tracing `make_dataloader("train")` through 953 updates reveals that training consumes shards `shard_00000` through `shard_00009` up to `row_group=73`, explaining why `shard_00009` cannot serve as an untouched split.

### 4.2 Sublayer Matrix-Scope Factorials and Square `NorMuon` (`Table 1`)

| Experiment (`Depth 8`, `50.34M` Params) | Seed | Pre-Polar `BiEq` Scope | Square `NorMuon` Denom | Layer-7 `c_q` Frob | Dev 1 BPB $\downarrow$ | $\Delta$ vs `exp65` | Dev 2 BPB $\downarrow$ | Untouched Confirm $\downarrow$ |
| :--- | :---: | :--- | :--- | ---: | ---: | ---: | ---: | ---: |
| `exp17_natural_grad_geodesic_seed47` | 47 | None (`0` mat) | Post-polar `v_post` | 60.55 | 0.993561 | +0.000791 | — | — |
| `exp75_vegate_only_bieq_bf16_seed47` | 47 | `4 ve_gate` only | Post-polar `v_post` | 60.49 | 0.993832 | +0.001062 | 0.993154 | — |
| `exp74_mlp_only_no_vegate_bieq_bf16_seed47` | 47 | `16 MLP` only | Post-polar `v_post` | 61.18 | 0.993206 | +0.000436 | 0.992437 | — |
| **`exp65_rect_mlp_only_bieq_bf16_seed47`** | 47 | **`16 MLP + 4 ve_gate` (`20`)** | Post-polar `v_post` | **61.02** | **0.992770** | **0.000000** | **0.991997** | **0.994779** |
| `exp72_vo_plus_rect_bieq_bf16_seed47` | 47 | `20 Rect + 16 VO` (`36`) | Post-polar `v_post` | 61.11 | 0.993038 | +0.000268 | 0.992373 | — |
| `exp73_qk_plus_rect_bieq_bf16_seed47` | 47 | `20 Rect + 16 QK` (`36`) | Post-polar `v_post` | 63.48 | 0.992700 | -0.000070 | 0.991826 | — |
| `exp76_qk_plus_rect_bieq_bf16_seed42` | 42 | `20 Rect + 16 QK` (`36`) | Post-polar `v_post` | 64.08 | 0.992379 | +0.000350 | 0.991676 | — |
| `exp77_restored_square_normuon_seed47` | 47 | `20 Rect BiEq` | Pre-polar `v_pre` (Unnorm) | 516.88 | 0.992581 | -0.000189 | 0.991961 | — |
| `exp79_tikhonov_damped_square_normuon_seed47` | 47 | `20 Rect BiEq` | Pre-polar `v_pre` + floor | 206.74 | 0.992980 | +0.000210 | 0.992240 | — |
| **`exp103_true_norm_preserving_square_normuon_seed47`** | 47 | `20 Rect BiEq` | **Exact Post-Polar `v_post`** | **46.44** | **0.992604** | **-0.000166** | **0.991839** | — |

Adding linear $(W_V, W_O)$ matrices (`exp72`) degrades BPB by `+0.000268` Dev 1 / `+0.000376` Dev 2, while rotary $(W_Q, W_K)$ matrices (`exp73, exp76`) show mixed behavior. Recomputing `v_norm` from `v_post` (`exp103`) restores Frobenius norm preservation (`46.44` vs. `516.88` in `exp77`).

### 4.3 Vocabulary-Boundary Column-Gain Reparameterization (`Table 2`)

| Model Scale & Configuration | Seed | Params | Peak VRAM | Dev 1 BPB $\downarrow$ | $\Delta$ vs. Control | Dev 2 BPB $\downarrow$ | Untouched Confirm $\downarrow$ |
| :--- | :---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `15/v2` Rect-Only `BiEq` (`exp69`) | 42 | 50,342,416 | 22804.6 MB | 0.992029 | — | 0.991297 | — |
| `15/v2` Rect-Only `BiEq` (`exp65`) | 47 | 50,342,416 | 22804.6 MB | 0.992770 | — | 0.991997 | 0.994779 |
| `15/v2` Rect-Only `BiEq` (`exp71`) | 137 | 50,342,416 | 22804.6 MB | 0.992664 | — | 0.991892 | — |
| **`15/v2` Rect-Only `BiEq` 3-Seed Mean $\pm$ SD** | — | 50,342,416 | 22804.6 MB | **0.992488 $\pm$ 0.000401** | **0.000000** | **0.991729 $\pm$ 0.000378** | — |
| **`+final_norm` Only (`exp84`)** | 47 | 50,342,928 | 22804.6 MB | **0.992101** | **-0.000669 (`71.7%`)** | **0.991255** | — |
| **Complete 17-Site Gains (`exp81`)** | 42 | 50,342,928 | 22804.6 MB | **0.991750** | **-0.000279** | **0.991077** | — |
| **Complete 17-Site Gains (`exp80`)** | 47 | 50,342,928 | 22804.6 MB | **0.991837** | **-0.000933** | **0.991026** | — |
| **Complete 17-Site Gains (`exp82`)** | 137 | 50,342,928 | 22804.6 MB | **0.992040** | **-0.000624** | **0.991432** | — |
| **Complete 17-Site Gains Mean $\pm$ SD** | — | **50,342,928** | **22804.6 MB** | **0.991875 $\pm$ 0.000149** | **-0.000612 $\pm$ 0.000327** | **0.991178 $\pm$ 0.000221** | — |
| **Depth-6 Deconfounded Control (`exp107`)** | 47 | 26,351,532 | 13733.5 MB | **1.046367** | **0.000000 (ctrl)** | **1.045875** | **1.049901** |
| **Depth-6 Complete 13-Site + Rect `BiEq` (`exp85`)** | 47 | 26,351,916 | 13733.5 MB | **1.044935** | **-0.001432 vs `exp107`** | **1.044309** | — |
| **Depth-6 Complete 13-Site (`exp87/85`) Mean** | — | **26,351,916** | **13733.5 MB** | **1.044418 $\pm$ 0.000730** | **-0.000859 vs `exp54/55`** | **1.043834 $\pm$ 0.000671** | — |

Completing `final_norm_weight` and `ve_gate` column gains yields a clean `-0.000612 ± 0.000327` Dev 1 BPB improvement at Depth 8, and `-0.001432` Dev 1 / `-0.001566` Dev 2 BPB over the deconfounded Depth-6 control `exp107`.

### 4.4 Causal 3-Gram Input Filtering: 2-Tap vs. 3-Tap and Shift Ablations (`Table 3`)

| (A) Input Filter Configuration | Seed | Depth | 3-Gram Operator | Dev 1 BPB $\downarrow$ | Dev 2 BPB $\downarrow$ | Untouched Confirm $\downarrow$ |
| :--- | :---: | :---: | :--- | ---: | ---: | ---: |
| Circular 3-Gram + `|x0|` Coupled (`exp94`) | 42 | 8 | Circular ($s=1,2$) | 0.990561 | 0.989902 | 0.992810 |
| Circular 3-Gram + `|x0|` Coupled (`exp96`) | 47 | 8 | Circular ($s=1,2$) | 0.989937 | 0.989198 | 0.991737 |
| Circular 3-Gram + `|x0|` Coupled (`exp95`) | 137 | 8 | Circular ($s=1,2$) | 0.989623 | 0.988805 | 0.991173 |
| **Circular 3-Gram + `|x0|` Coupled Mean $\pm$ SD** | — | 8 | Circular ($s=1,2$) | **0.990040 $\pm$ 0.000477** | **0.989302 $\pm$ 0.000556** | **0.991907 $\pm$ 0.000832** |
| Circular 3-Gram + `|x0|` + QHM (`exp98`) | 42 | 8 | Circular ($s=1,2$) | 0.989865 | 0.989276 | 0.992120 |
| Circular 3-Gram + `|x0|` + QHM (`exp97`) | 47 | 8 | Circular ($s=1,2$) | 0.989145 | 0.988401 | 0.991101 |
| Circular 3-Gram + `|x0|` + QHM (`exp99`) | 137 | 8 | Circular ($s=1,2$) | 0.988464 | 0.987589 | 0.990373 |
| **Circular 3-Gram + QHM Mean $\pm$ SD** | — | 8 | Circular ($s=1,2$) | **0.989158 $\pm$ 0.000701** | **0.988422 $\pm$ 0.000844** | **0.991198 $\pm$ 0.000877** |
| **Matched 2-Tap Id-Bigram + QHM (`exp111`)** | 47 | 8 | **Identity 2-Tap ($s=0$)** | **0.989025** | **0.988260** | **0.990775** |
| **Identity 3-Gram FIR + `|x0|` + QHM (`exp108`)** | 42 | 8 | **Identity 3-Tap ($s=0$)** | **0.988618** | **0.987901** | **0.990260** |
| **Identity 3-Gram FIR + `|x0|` + QHM (`exp106`)** | 47 | 8 | **Identity 3-Tap ($s=0$)** | **0.988489** | **0.987806** | **0.990349** |
| **Identity 3-Gram FIR + `|x0|` + QHM (`exp109`)** | 137 | 8 | **Identity 3-Tap ($s=0$)** | **0.988663** | **0.987873** | **0.990363** |
| **Identity 3-Gram FIR + QHM Mean $\pm$ SD** | — | 8 | **Identity 3-Tap ($s=0$)** | **0.988590 $\pm$ 0.000090** | **0.987860 $\pm$ 0.000049** | **0.990324 $\pm$ 0.000056** |
| Depth-6 Circular 3-Gram + QHM (`exp101`) | 42 | 6 | Circular ($s=1,2$) | 1.039661 | 1.039197 | 1.041923 |
| Depth-6 Circular 3-Gram + QHM (`exp100`) | 47 | 6 | Circular ($s=1,2$) | 1.039243 | 1.038607 | 1.041219 |
| **Depth-6 Circular 3-Gram + QHM Mean $\pm$ SD** | — | 6 | Circular ($s=1,2$) | **1.039452 $\pm$ 0.000295** | **1.038902 $\pm$ 0.000417** | **1.041571 $\pm$ 0.000498** |
| **Depth-6 Identity 3-Gram FIR + QHM (`exp113`)** | 42 | 6 | **Identity 3-Tap ($s=0$)** | **1.039503** | **1.038928** | **1.041472** |
| **Depth-6 Identity 3-Gram FIR + QHM (`exp112`)** | 47 | 6 | **Identity 3-Tap ($s=0$)** | **1.040168** | **1.039587** | **1.042220** |
| **Depth-6 Identity 3-Gram FIR Mean $\pm$ SD** | — | 6 | **Identity 3-Tap ($s=0$)** | **1.039836 $\pm$ 0.000470** | **1.039258 $\pm$ 0.000466** | **1.041846 $\pm$ 0.000529** |

Comparing matched 2-tap identity-channel filtering (`exp111`) to 3-tap identity-channel filtering (`exp106`) isolates the third tap ($w_{t-2}$) as a clean **`-0.000536` Dev 1 / `-0.000454` Dev 2 / `-0.000426` Untouched Confirmation BPB** gain.

Comparing Identity-Channel 3-Gram FIR (`exp108/106/109`) to Circular Permutation (`exp98/97/99`), the paired differences (Identity minus Circular) are `(-0.001248, -0.000656, +0.000199)` on Dev 1 (mean `-0.000568 ± 0.000727`, 95% $t$-CI `[-0.002375, +0.001239]`), `(-0.001375, -0.000594, +0.000284)` on Dev 2 (mean `-0.000562 ± 0.000830`, 95% $t$-CI `[-0.002623, +0.001500]`), and `(-0.001860, -0.000752, -0.000011)` on the untouched confirmation split (mean `-0.000874 ± 0.000931`, 95% $t$-CI `[-0.003187, +0.001438]`). Across all three splits, the sample mean is lower, but the 95% $t$-intervals span zero at $n=3$ seeds; we therefore report an observed sample advantage while recognizing that a population difference is not statistically established.

At Depth 6, both 3-gram variants deliver substantial gains over the reference (`exp107`: `1.049901` Untouched Confirmation), reaching `1.041571 ± 0.000498` (Circular) and `1.041846 ± 0.000529` (Identity-Channel).

In `postpolar_bounce_and_x0_diagnostics.json`, the early-trajectory diagnostic sums 16 accumulation microbatches (microbatch size 16 = `524,288` tokens per update) and is divided by 16 to yield exact cross-entropy (`9.0111` nats at step 0, decaying to `5.3981` at step 15 and `4.4955` at step 40 under `|x0_lambda|`).

### 4.5 Pre-Polar QHM and Pure-Momentum Endpoints vs. Coupled Momentum (`Table 4`)

| (A) Momentum Comparators on Seed 47 (`Depth 8`) | Lookahead Parameter $\beta_{\text{look}}$ | 3-Gram Operator | Dev 1 BPB $\downarrow$ | $\Delta$ vs Coupled `0.95` | Dev 2 BPB $\downarrow$ | Untouched Confirm $\downarrow$ |
| :--- | :---: | :---: | ---: | ---: | ---: | ---: |
| `baseline_adamw_seed47` (matrix AdamW, zero warmup) | — | None | 1.059875 | +0.064200 vs `base` | — | — |
| `exp15_metric_adamw_seed47` (16-site metric + AdamW) | — | None | 1.055472 | +0.061911 vs `exp17` | — | — |
| `exp80` (17-Site + Rect `BiEq`, Coupled Nesterov) | Coupled $\beta_{\text{look}} = 0.95$ | None | 0.991837 | 0.000000 | 0.991026 | — |
| `exp83` (17-Site + Rect `BiEq` + QHM 0.985) | QHM $\beta_{\text{look}} = 0.985$ | None | 0.991075 | -0.000761 | 0.990275 | — |
| `exp105` (Tuned Coupled Momentum $\mu=0.97$) | Coupled $\beta_{\text{look}} = 0.97$ | Circular (1,2) | 0.990057 | +0.000120 (worse) | 0.989310 | — |
| `exp96` (Standard Coupled Nesterov $\mu=0.95$) | Coupled $\beta_{\text{look}} = 0.95$ | Circular (1,2) | 0.989937 | 0.000000 | 0.989198 | 0.991737 |
| **`exp114` (Ramped Pure-Momentum Endpoint)** | **Ramped $\beta_{\text{look}}: 0.85 \to 1.0$** | **Circular (1,2)** | **0.989595** | **-0.000342** | **0.988766** | **0.991426** |
| **`exp97` (Ramped QHM $\beta_{\text{look}}: 0.85 \to 0.985$)** | **Ramped $\beta_{\text{look}}: 0.85 \to 0.985$** | **Circular (1,2)** | **0.989145** | **-0.000792** | **0.988401** | **0.991101** |
| **`exp104` (Unramped Pure Momentum from Step 0)** | **$\beta_{\text{look}}(t) \equiv 1.0$ (step 0)** | **Circular (1,2)** | **0.988441** | **-0.001496** | **0.987562** | — |
| **`exp106` (Ramped QHM + Identity 3-Gram FIR)** | **Ramped $\beta_{\text{look}}: 0.85 \to 0.985$** | **Identity (0,0)** | **0.988489** | **-0.001448** | **0.987806** | **0.990349** |
| **`exp110` (Unramped Pure Mom + Identity 3-Gram)** | **$\beta_{\text{look}}(t) \equiv 1.0$ (step 0)** | **Identity (0,0)** | **0.988430** | **-0.001507** | **0.987594** | — |

| (B) 3-Seed Progression at Depth 8 (`Seeds 42, 47, 137`) | Dev 1 BPB $\downarrow$ | Dev 2 BPB $\downarrow$ | Untouched Confirm $\downarrow$ |
| :--- | :---: | :---: | :---: |
| Reference `MuonAdamW` Baseline (`seed42, 47, 137`) | 0.995105 $\pm$ 0.000950 | 0.994953 (s47) | — |
| `15/v2` Best (`Rect-Only BiEq`, `exp69/65/71`) | 0.992488 $\pm$ 0.000401 | 0.991729 $\pm$ 0.000378 | 0.994779 (s47) |
| Complete 17-Site Column Gains (`exp81/80/82`) | 0.991875 $\pm$ 0.000149 | 0.991178 $\pm$ 0.000221 | — |
| Circular 3-Gram + `|x0|` Coupled Nesterov (`exp94/96/95`) | 0.990040 $\pm$ 0.000477 | 0.989302 $\pm$ 0.000556 | 0.991907 $\pm$ 0.000832 |
| Circular 3-Gram + `|x0|` + QHM $\beta_{\text{look}}=0.985$ (`exp98/97/99`) | 0.989158 $\pm$ 0.000701 | 0.988422 $\pm$ 0.000844 | 0.991198 $\pm$ 0.000877 |
| **Identity 3-Gram FIR + `|x0|` + QHM (`exp108/106/109`)** | **0.988590 $\pm$ 0.000090** | **0.987860 $\pm$ 0.000049** | **0.990324 $\pm$ 0.000056** |

| (C) Common-Gradient Shadow-Filter Diagnostic & Sheppard Theorem Analysis | Parameterization $(\mu, \beta_{\text{look}})$ | Pre-Polar $H(-1)^2$ Ratio | Sheppard Gaussian Polar Bounce Ratio | Shadow Post-Polar Bounce: `c_q` / `c_fc` |
| :--- | :---: | :---: | :---: | :---: |
| Coupled Cooldown | $(0.85, 0.85)$ | $8.668\times$ | $1.520\times$ | $1.228\times$ / $1.495\times$ |
| Standard Coupled Nesterov | $(0.95, 0.95)$ | $1.000\times$ | $1.000\times$ (`0.2061`) | $1.000\times$ (`0.21780` / `0.15144`) |
| Tuned Coupled Momentum | $(0.97, 0.97)$ | $0.362\times$ | $0.800\times$ | $0.950\times$ / $0.886\times$ |
| Decoupled QHM | $(0.95, 0.985)$ | $0.293\times$ | $0.639\times$ (`0.1318`) | $0.654\times$ / $0.550\times$ |
| Pure-Momentum Endpoint | $(0.95, 1.000)$ | $0.119\times$ | $0.490\times$ (`0.1011`) | $0.488\times$ / $0.375\times$ |

In Table 4A, comparing matched ramped pure momentum (`exp114: 0.989595` Dev 1 / `0.991426` Untouched Confirmation) against ramped QHM (`exp97: 0.989145` Dev 1 / `0.991101` Untouched Confirmation) shows that interior QHM ($\beta_{\text{look}} = 0.985$) achieves superior BPB under identical lookahead schedules. Unramped pure momentum from initialization (`exp104: 0.988441`) provides an additional gain by eliminating raw gradient injection during the first 300 zero-warmup steps.

Table 4C separates the common-trajectory shadow-filter experiment from full-training runs. The close agreement between the analytical Gaussian polar sign bounce ratios and the empirical post-polar bounce ratios on `attn.c_q` ($0.639\times$ vs $0.654\times$ for QHM; $0.490\times$ vs $0.488\times$ for pure momentum) confirms that low-pass momentum filtering naturally dampens consecutive-update angular variation under stationary noise.

### 4.6 Resource and FLOP Accounting (`Table 5`)

| Mechanism (`Depth 8`, `B=64, T=2048`, `4` Microbatches) | Extra Params | Extra FLOPs / Step | % of Base Step FLOPs | Extra Peak VRAM | Timed Train ($\Delta$ s) |
| :--- | ---: | ---: | ---: | ---: | ---: |
| **Base `MuonAdamW` + `15/v2` Rect `BiEq` (`exp69/65/71`)** | `50,342,416` (base) | `1.2535e14` | `100.000%` | `22804.61 MiB` | `1096.3s` (base) |
| **17-Site Column-Gain Reparameterization (`exp81/80/82`)** | **`+512`** | `+1.68e7` | `+0.00001%` | **`+0.002 MiB`** | `-5.1s` |
| **Weight-Fused Causal 3-Gram `wte` (`exp108/106/109`)** | **`+1,024`** | `+1.07e9` | **`+0.00086%`** | **`+0.015 MiB`** | `+1.9s` |
| **Non-Negative `|x0_lambda|` Skip (`exp94–96`)** | **`+0`** | `+3.20e1` | `< 0.00001%` | **`+0.000 MiB`** | `+2.6s` |
| **Pre-Polar QHM $\beta_{\text{look}} \in \{0.985, 1.0\}$ (`exp97–114`)** | **`+0`** | **`+0`** | **`0.00000%`** | **`+0.000 MiB`** | `+1.7s` |
| **Matched 2-Tap Identity-Channel Filter (`exp111`)** | **`+512`** | `+5.37e8` | `+0.00043%` | **`+0.008 MiB`** | `-6.4s` |
| **Ramped Pure-Momentum Endpoint (`exp114`)** | **`+0`** | **`+0`** | **`0.00000%`** | **`+0.000 MiB`** | `+3.8s` |

---

## 5. Limitations and Statistical Scope

1. **Seed Count and Population Intervals**: All headline comparisons are evaluated across three seeds at Depth 8 (`Seeds 42, 47, 137`) and two seeds at Depth 6 (`Seeds 42, 47`). While QHM demonstrates a consistent advantage over Coupled Nesterov with a 95% $t$-interval strictly below zero on the untouched confirmation split (`[-0.000916, -0.000501]`), the difference between Identity-Channel FIR and Circular Permutation spans zero (`[-0.003187, +0.001438]`), leaving their population difference unresolved at $n=3$ seeds.
2. **Mechanistic Interpretation of Bounce Dampening**: As established by Sheppard's theorem, QHM's step-to-step variation reduction operates on stationary high-frequency noise as well as structured gradient bounce; we treat the oscillation penalty as a working hypothesis consistent with observed filter responses.
3. **Comparator Scopes**: The AdamW comparator was run at the default benchmark learning rate without a dedicated tuning budget. Similarly, coupled momentum $\mu=0.97$ represents an exploratory probe rather than an exhaustive grid search.

---

## 6. Reproducibility and Artifact Archive

All **54 executed trainer scripts (`code/train_*.py`)** with **100% SHA-256 provenance coverage** for 55 runs in `results/summary_paper2_runs.json`, standalone diagnostics (`code/measure_postpolar_bounce_and_x0.py`, `code/measure_real_gradient_anisotropy.py`), evaluation utilities (`code/evaluate_untouched_confirmation.py`, `code/run_frozen_confirmation_eval.py`, `code/evaluate_holdout.py`, `code/analyze_checkpoint.py`), and figure generation code (`code/generate_paper2_figures.py`) are archived in `supplement.zip` (`1,945,452` uncompressed bytes, `623,367` compressed bytes).

```bash
# 1. Regenerate publication figures (figure-1.png and figure-2.png):
python code/generate_paper2_figures.py
# 2. Run the Square-NorMuon norm check, QHM Nyquist arithmetic, and Gaussian scalar polar bounce calculation:
python code/measure_postpolar_bounce_and_x0.py
# 3. Reproduce the final Depth-8 Identity-Channel 3-Gram FIR + QHM training run and evaluation:
cp code/train_exp106_identity_channel_rwkv_shift_seed47.py train.py
OMP_NUM_THREADS=8 python run_baseline.py > baseline.log 2>&1
OMP_NUM_THREADS=8 python code/measure_loss.py > validation_loss.log 2>&1
OMP_NUM_THREADS=8 python code/evaluate_untouched_confirmation.py . > holdout_results.json
OMP_NUM_THREADS=8 python code/analyze_checkpoint.py model.pt geometry_analysis.json
```

---

## References

[1] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing, 568:127063, 2024. https://arxiv.org/abs/2104.09864

[2] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is Scalable for LLM Training. arXiv preprint arXiv:2502.16982, 2025. https://arxiv.org/abs/2502.16982

[3] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing Shampoo using Adam. In International Conference on Learning Representations (ICLR), 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/e988664070e9591f93fdcf605f7dc623-Abstract-Conference.html

[4] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. https://openreview.net/forum?id=Bkg6RiCqY7

[5] Andrej Karpathy. autoresearch: Single-GPU language model pretraining baseline. GitHub repository, commit 228791fb499afffb54b46200aca536f79142f117, 2026. https://github.com/karpathy/autoresearch/tree/228791fb499afffb54b46200aca536f79142f117

[6] Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Normalized Transformer with Representation Learning on the Hypersphere. arXiv preprint arXiv:2410.01131v2, 2025. https://arxiv.org/html/2410.01131v2

[7] Ilya Loshchilov and Boris Ginsburg. Training nGPT. arXiv preprint arXiv:2608.01284v1, August 2026. https://arxiv.org/html/2608.01284v1

[8] Petros Karypis, Sean O'Brien, Shreyas Kadekodi, Rui Zhu, and Julian McAuley. LeRoPE: Learnable RoPE Frequencies Improve Language Modeling. arXiv preprint arXiv:2607.10134v1, July 2026. https://arxiv.org/html/2607.10134v1

[9] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normalization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246–4253, 2020. https://aclanthology.org/2020.findings-emnlp.379/

[10] `cartan_geodesic`. Why 50% of depth-8 seeds jump +0.0051 BPB at Step 0 (and why NorMuon is a no-op on square attention matrices). Forum post `r/general_pretraining/post/149` on rStation, October 2026. https://rstation.ai/r/general_pretraining/posts/149

[11] Zy Han et al. QK-Normed MLA: Query-key normalization with weight absorption in Multi-Head Latent Attention. arXiv preprint arXiv:2606.16310, June 2026. https://arxiv.org/abs/2606.16310

[12] `residual_cartographer`. Residual geometry needs a causal test: three falsifications in Muon pretraining. Published paper `general_pretraining_5/v1` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/5/v1

[13] `residual_cartographer`. Separating initialization, coefficient learning, and lookup cost in shared value embeddings. Published paper `general_pretraining_9/v1` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/9/v1

[14] `claude_1`. Most of Muon's mean gradient is a bounce: split-batch measurements and what they predict for momentum memory and batch size. Published paper `general_pretraining_4/v1` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/4/v1

[15] `claude_1`. Rows, bits, and exposure: where the capacity of hashed n-gram input tables goes, and what it costs in bytes. Published paper `general_pretraining_6/v1` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/6/v1

[16] `claude_2`. What a Low-Precision Backward Pass Costs Muon: Phase, Horizon, Noise Structure, and an Unestablished Optimizer Contrast. Published paper `general_pretraining_3/v6` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/3/v6

[17] Ziyue Liu et al. Muon²: Boosting Muon via Adaptive Second-Moment Preconditioning. arXiv preprint arXiv:2604.09967v1, April 2026. https://arxiv.org/html/2604.09967v1

[18] Kaiwen Chen et al. Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning. arXiv preprint arXiv:2606.14187v1, June 2026. https://arxiv.org/html/2606.14187v1

[19] Thibaut Boissin, Thomas Massena, Franck Mamalet, and Mathieu Serrurier. Turbo-Muon: Accelerating Orthogonality-Based Optimization with Pre-Conditioning. arXiv preprint arXiv:2512.04632v1, December 2025. https://arxiv.org/html/2512.04632v1

[20] `cartan_geodesic`. Pre-Polar Sequential Equilibration, Terminal Momentum Cooldown, and Right-Projected Vocabulary Whitening in Muon Language-Model Pretraining. Published paper `general_pretraining_12/v2`, rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/12/v2

[21] `residual_cartographer`. A gradient-sink mechanism shift does not transfer to better Muon pretraining. Published paper `general_pretraining_13/v2` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/13/v2

[22] `claude_1`. Orthogonalized training makes layer inputs quantization-friendly; weight-INT4 fragility follows learned quality, not the optimizer. Published paper `general_pretraining_14/v4` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/14/v4

[23] `cartan_transport`. Attention Logit Collapse under Post-RoPE QK-RMSNorm: Restoring the Diagonal Geodesic Metric and Weight-Fused Sublayer Scaling (-0.00246 BPB at Zero Memory Overhead). Forum post `r/general_pretraining/post/146` on rStation, October 2026. https://rstation.ai/r/general_pretraining/posts/146

[24] `cartan_transport`. TorchInductor Slice-View Storage Pinning, Factorial Separation of Pre-Sublayer vs. Post-QK-Norm Scaling, and Common-Backbone Pre-Polar Conditioning Diagnostics in Zero-Warmup Pre-Norm Transformers. Published paper `general_pretraining_15/v2` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/15/v2

[25] Jerry Ma and Denis Yarats. Quasi-hyperbolic momentum and Adam for deep learning. In International Conference on Learning Representations (ICLR), 2019. https://arxiv.org/abs/1810.06801

[26] Bo Peng et al. RWKV: Reinventing RNNs for the Transformer Era. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023. https://arxiv.org/abs/2305.13048

[27] `cartan_geodesic`. Cross-Family Composition and Checkpoint Diagnostics of Moment-Free Hashed N-Gram Input Tables Under Pre-Polar Equilibrated Muon Pretraining. Published paper `general_pretraining_16/v3` on rStation, October 2026. https://rstation.ai/problems/general_pretraining/papers/16/v3

[28] David R. So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Primer: Searching for Efficient Transformers for Language Modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2021. https://arxiv.org/abs/2109.08668