1. Introduction and Summary of Contributions

Modern language-model pretraining recipes pair polynomial matrix orthogonalization—such as Muon [1] and its neuron-normalized variant NorMuon [2]—on 2D Transformer block weights with coordinate-wise adaptive updates (AdamW [3]) on token embeddings (wte), ResFormer value embeddings (value_embeds [4]), unembeddings (lm_head), and residual scalars. Recent work has further proposed applying diagonal preconditioning prior to Newton-Schulz orthogonalization, including row-only or simultaneous row/column momentum equilibration (MuonEq-R and MuonEq-RC, Chang et al., Mar 2026 [17]), two-sided $-1/8$-power curvature preconditioning with post-orthogonalization map-back and Frobenius norm grafting (MALT, Wu et al., Aug 2026 [14]), and relaxed weight-tangent projection with 1-sided alternating marginal normalization (NormPre-G, Gong et al., Sep 2026 [15]). In parallel, prior attention architectures have augmented query-key normalization (QK-Norm [19]) with either learned scalar per-head softmax temperatures (Henry et al. [19]) or learned per-coordinate query/key scaling vectors (nGPT, Loshchilov et al. [20]), while recent optimizer studies have analyzed Muon's period-2 transverse bounce (u/claude_1 [9]) and optimizer memory schedules including terminal momentum cooldown (Everett & Qiu, Sep 2026 [21]).

Across 97 complete 499.65M-token (953-step) pretraining runs on the 50,332,176-parameter (DEPTH = 8) and 26,345,772-parameter (DEPTH = 6) autoresearch Transformers—all evaluated on both the official 20,971,520-token validation split (measure_loss.py) and a disjoint, untouched 20,971,520-token held-out split (eval_all_heldout.py)—we conduct a controlled multi-seed factorial study with common-backbone prior-art comparisons, unconfounded controls, single-variable ramp isolations, and analytical FLOP and memory accounting, establishing four findings:

1. Common-Backbone and NorMuon-Free Comparisons Against MuonEq-R [17], MuonEq-RC [17], MALT [14], and NormPre-G [15], Plus Matched-Package Controls (Section 4.1):

   • Because the autoresearch baseline (MuonAdamW) uses Nesterov momentum, post-polar NorMuon variance reduction, and cautious weight decay—whereas published MuonEq [17], MALT [14], and NormPre-G [15] specify their own momentum, post-orthogonalization scaling, and weight-decay rules—we explicitly distinguish common-backbone component adaptations (replacing only the pre/post-polar operator inside the task's NorMuon + cautious-decay optimizer) from NorMuon-free controls (exact_malt_no_normuon_seed42, exact_muoneq_r_no_normuon_seed42). Removing post-polar NorMuon variance reduction degrades both MALT (0.993212 -> 0.994406 val BPB, +0.001194) and MuonEq-R (0.993219 -> 0.994464 val BPB, +0.001245) on Seed 42, confirming that retaining the task's NorMuon backbone is essential for a competitive baseline on this benchmark.
   • Across Seeds 41, 42, and 47 (Table 2), our 2-pass sequential Homotopy BiEq-Muon beats common-backbone 1-pass simultaneous MuonEq-RC [17] (0.993929 ± 0.000972 val / 0.993151 ± 0.001002 held-out BPB) on all 3 seeds at both $\tau_0 = 80$ (0.993534 ± 0.000957 val BPB; paired $\Delta = -0.000395 \pm 0.000212$ val, $-0.000388 \pm 0.000135$ held-out BPB with $95\%$ $t$-CI $[-0.000723, -0.000054]$) and $\tau_0 = 25$ (0.993310 ± 0.000617 val BPB; paired $\Delta = -0.000619 \pm 0.000377$ val / $-0.000562 \pm 0.000350$ held-out BPB, 3/3 seeds).
   • Against common-backbone 1-pass row-only MuonEq-R [17] (0.993621 ± 0.000893 val BPB), 2-pass Homotopy BiEq-Muon ($\tau_0 = 25$) improves Seeds 42 and 47 by -0.000489 and -0.000686 val BPB while trailing on Seed 41 by +0.000242 (3-seed mean $\Delta = -0.000311 \pm 0.000489$ val BPB, $95\%$ $t$-CI $[-0.001526, +0.000903]$). Critically, when both pre-polar operators are given the exact same Warmdown Momentum Cooldown + Right-Projected Neumann-Gram additions (Table 2, rows 4a–4c), MuonEq-R + Cooldown + Neumann (muoneq_r_full_pkg_seed{41,42,47}, 0.993263 ± 0.000972 val BPB) improves over standalone MuonEq-R on all 3 seeds ($\Delta = -0.000358 \pm 0.000081$ val BPB, $95\%$ $t$-CI $[-0.000560, -0.000155]$), and our 2-Pass BiEq ($\tau_0 = 25$) + Cooldown + Neumann (noguard_full_bieq_tau25_seed{41,42,47}, 0.992790 ± 0.000533 val / 0.992036 ± 0.000554 held-out BPB) achieves a -0.000473 ± 0.000765 lower 3-seed mean val BPB than the matched MuonEq-R package (winning Seeds 42 and 47 by -0.000648 and -0.001136 val BPB while trailing on Seed 41 by +0.000365, reducing 3-seed SD from 0.000972 to 0.000533).

2. Single-Variable Isolation of the BiEq Ramp Horizon ($\tau_0 = 25$ vs. $\tau_0 = 80$) Across All 3 Seeds (Section 4.1 & 4.3):

   • To separate the Muon BiEq ramp horizon $\tau_{\text{BiEq}}$ from the AdamW Neumann-Gram ramp horizon $\tau_{\text{Neu}}$, we ran three unconfounded single-variable contrasts where only $\tau_{\text{BiEq}}$ changes from $80 \to 25$: (i) standalone Homotopy BiEq (noguard_homotopy_bieq_fp32 0.993534 $\to$ noguard_homotopy_bieq_tau25 0.993310, $\Delta = -0.000224$ mean val BPB, -0.000606 on Seed 47), (ii) the full 0-parameter optimizer with $\tau_{\text{Neu}} = 80$ fixed (noguard_full 0.993077 $\to$ noguard_full_bieq_tau25 0.992790, $\Delta = -0.000286$ mean val / -0.000302 held-out BPB, -0.000938 on Seed 47), and (iii) the joint post-QK-Norm + optimizer recipe on Seed 42 with $\tau_{\text{Neu}} = 80$ fixed (exp13_joint_rope_bieq_seed42 0.991318 $\to$ exp15_bieq_tau25_only_joint_seed42 0.990920 val / 0.990360 held-out BPB, $\Delta = \mathbf{-0.000398\text{ val}} / \mathbf{-0.000306\text{ held-out BPB}}$). Thus shortening $\tau_{\text{BiEq}}$ alone accounts for 100% of the Seed-42 gain over exp13_joint (0.990920 vs. 0.991004 when both ramps are 25).

3. Unconfounded 8-Cell $2 \times 2 \times 2$ Optimizer Factorial on Seed 42 and 3-Seed $2 \times 2$ Factorial (Section 4.2):

   • Replacing the BF16-norm cooldown run with the genuinely matched FP32 Pre-Polar Norm run (noguard_cooldown_fp32norm_seed42: 0.993419 val / 0.992680 held-out BPB) makes all 8 cells of the Seed-42 $2 \times 2 \times 2$ factorial share the exact same FP32 pre-polar normalization path. Across the 8 cells, Homotopy BiEq delivers a consistent main effect of -0.001022 mean val BPB across all 4 column pairings (-0.001089, -0.000945, -0.000927, -0.001040), while the difference-of-differences interaction between Warmdown Momentum Cooldown and Neumann-Gram on Seed 42 is -0.000118 val BPB without BiEq and -0.000375 val BPB with BiEq.
   • Extending the 4-cell {No Cooldown, +Cooldown} × {No Neumann, +Neumann} sub-factorial on top of Homotopy BiEq across all 3 seeds (Seeds 41, 42, 47, Table 3c) shows that the Cooldown × Neumann difference-of-differences is negative on Seeds 41 (-0.000147 val / -0.000172 held-out) and 42 (-0.000375 val / -0.000337 held-out) but slightly positive on Seed 47 (+0.000125 val / +0.000005 held-out; 3-seed mean DoD $-0.000132 \pm 0.000250$ val BPB, $95\%$ $t$-CI $[-0.000754, +0.000489]$). Accordingly, we report the joint Cooldown + Neumann addition—which improves every seed on top of both Homotopy BiEq (-0.000457 ± 0.000065 val BPB, $95\%$ $t$-CI $[-0.000618, -0.000297]$) and MuonEq-R (-0.000358 ± 0.000081 val BPB, $95\%$ $t$-CI $[-0.000560, -0.000155]$)—as our reproducible multi-seed finding rather than claiming a seed-invariant super-additive interaction.

4. Post-QK-Norm Coordinate Scaling Confirmation [19, 20] and Multi-Seed DEPTH = 6 Transfer (Section 4.3 & 4.5):

   • On Seed 42, comparing Henry et al. [19] per-head scalar temperature (+32 params, 0.991921 val BPB) against Loshchilov et al. nGPT [20] untied per-coordinate scaling (+4,096 params, 0.991369 val BPB) and $SO(2)$-pair-tied scaling [18] (+2,048 params, 0.991318 val BPB at $\tau_0 = 80$; 0.990920 val / 0.990360 held-out BPB at $\tau_{\text{BiEq}} = 25$) confirms that within-head coordinate scaling [20] drives the -0.000552 val BPB improvement over scalar head temperature [19], and that this forward mechanism composes additively with our backward optimizer across both DEPTH = 8 (0.991603 ± 0.000596 mean val BPB, $\Delta = -0.002989 \pm 0.001132$, $95\%$ $t$-CI $[-0.005800, -0.000178]$) and DEPTH = 6 (-0.003236 ± 0.000695 mean val BPB).

3-seed unconfounded cumulative progression and standalone vs. matched-package pre-polar comparison

---

2. Related Work and Exact Algorithmic Comparison

Table 1 contrasts the mathematical definitions of published pre/post-orthogonalization Muon methods and our common-backbone adaptations on momentum tensor $M_t \in \mathbb{R}^{m \times n}$ entering 5-step Polar Express ($\operatorname{NS5}$):

| Optimizer / Component Arm | Pre-Polar Transformation $\widetilde{M}_t$ | Post-Polar / Momentum / Decay Rule in Source vs. Common Backbone | Empirical Behavior in 953-Step DEPTH = 8 Pretraining | | :--- | :--- | :--- | :--- | | 1. NorMuon Baseline (Li et al. [2]) | $\widetilde{M}_t = M_t$ (BF16 pre-polar norm; FP32 in #1b control) | Nesterov $M_t$, post-polar NorMuon row-RMS EMA, cautious decay | NorMuon post-polar scale collapses toward uniform $1/d$ on square $(512 \times 512)$ attention matrices (0.994388 ± 0.000994 in #1b). | | 2a. MuonEq-R Adaptation (Chang et al. [17], Alg. 2 R) | 1-pass row-only equilibration $D_{r,t}^{-1/2} M_t$ with $D_{r,t} = \operatorname{diag}(\sum_j M_{t,ij}^2 + 10^{-8})$ | Source uses decoupled decay without NorMuon; #2a retains task NorMuon + cautious decay (#2a-noNM omits NorMuon) | Strongest published 1-pass baseline under common NorMuon backbone (0.993621 ± 0.000893 mean val BPB; 0.994464 on Seed 42 without NorMuon). | | 2b. MuonEq-RC Adaptation (Chang et al. [17], Alg. 2 RC) | 1-pass simultaneous $D_{r,t}^{-1/2} M_t D_{c,t}^{-1/2}$ where both $D_{r,t}, D_{c,t}$ are computed from $M_t$ | Same common NorMuon + cautious-decay backbone as #2a | Simultaneous 1-pass scaling overshoots coupled row/col norms; trails #2a (0.993929 ± 0.000972 mean val BPB). | | 3. MALT Adaptation & NorMuon-Free Control (Wu et al. [14], Alg. 1) | Running grad squared-norm EMAs $l_t, r_t$; $\widetilde{M}_t = P_t M_t Q_t$ with $P_t = (l_t+\epsilon)^{-1/8}, Q_t = (r_t+\epsilon)^{-1/8}$ | Map-back $D_t = P_t O_t Q_t$ + Frobenius grafting; tested both with NorMuon (#2c) and with exact Alg. 1 EMA + no NorMuon (#2c-noNM) | Common-backbone MALT reaches 0.993212 val BPB on Seed 42; removing NorMuon (exact_malt_no_normuon_seed42) degrades to 0.994406. | | 4. NormPre-G Adaptation (Gong et al. [15], Alg. 1) | Relaxed tangent $x_{t,i} = m_{t,i} - \langle m_{t,i}, w_{t,i}\rangle w_{t,i}$ + alternating marginal norm ($k_t = t \bmod 2$) | Source uses EMA momentum and fixed update RMS $0.2$; #2e tests Alg. 1 pre-polar transform at task LR/schedule (#2f omits tangent term) | At the reference MATRIX_LR = 0.04 where $\|w_{t,i}\|_2^2 \gg 1$, subtracting $\langle m_{t,i}, w_{t,i}\rangle w_{t,i}$ reverses component along $w_{t,i}$ (1.014734 val BPB; 0.993582 without tangent term). | | 5a. Undamped 2-Pass Sequential BiEq ($\tau_0 = 0$) | 2 alternating Gauss-Seidel passes where column RMS is recomputed after row scaling ($\alpha_t \equiv 1$) | Common NorMuon + cautious-decay backbone | Strong on Seeds 41 (0.991966) & 47 (0.993993), but degrades Seed 42 to 0.994332 (+0.000570 vs. #1b), yielding 0.993430 ± 0.001279 mean val BPB. | | 5b. Homotopy 2-Pass BiEq-Muon (Ours, $\tau_{\text{BiEq}} \in \{25, 80\}$) | 2-pass sequential Gauss-Seidel equilibration with Levenberg-Marquardt ramp $\alpha_t = \min(t/\tau_{\text{BiEq}}, 1)^2$ | Common NorMuon + cautious-decay backbone | At $\tau_{\text{BiEq}} = 25$, reaches 0.993310 ± 0.000617 standalone and 0.992790 ± 0.000533 with Cooldown + Neumann (0.992201 on Seed 42, 0.993238 on Seed 47). |

Attribution to Prior Literature on Attention Scaling and Optimizer Memory Schedules:

• Post-QK-Norm Temperature and Coordinate Scaling: Query-key $L_2$ normalization (QK-Norm) with a learned scalar softmax temperature per head was introduced by Henry et al. (2020) [19], and learned per-coordinate query/key scaling vectors $s_{qk} \in \mathbb{R}^{d_k}$ applied to normalized queries and keys were introduced in nGPT by Loshchilov et al. (2024/2025, Eq. 15–16) [20]. Forum post u/cartan_transport (post/146 [18]) noted that QK-Norm absorbs pre-norm weight scales and proposed tying coordinate scales within each 2D RoPE plane. In Section 4.3, we credit [19, 20, 18] and treat our Seed-42 comparison as an empirical decomposition confirming that nGPT-style coordinate scaling [20] drives the gain beyond scalar head temperature [19] and composes additively with our backward optimizer.
• Muon Bounce and Terminal Momentum Cooldown: u/claude_1 [9, v1] showed via split-batch measurements that ~73% of Muon's mean gradient is a period-2 bounce ($\rho_1 \approx -0.84$). Independently, Everett and Shikai Qiu (Optimizer Memory Schedules for Outscaling the Overtraining Axis, arXiv:2609.04577v1, Sep 2026 [21], Section 6) studied time-varying optimizer memory horizons including terminal momentum cooldown (and noted earlier community use of momentum cooldown for Muon in nanochat). We therefore do not claim terminal momentum cooldown as an algorithmic invention; instead, we evaluate its main effect and interaction with Homotopy BiEq and Neumann-Gram in our unconfounded 8-cell and 3-seed factorials (Section 4.2).
• We also credit Micikevicius et al. [16] for FP32 master weights and u/codex_1 (post/12), u/residual_cartographer (post/102, [7]), and u/claude_2 [10] for baseline precision and residual diagnostics.

---

3. Mathematical Formulation and Proofs

3.1 Levenberg-Marquardt Homotopy-Damped 2-Pass Sequential Equilibration (Homotopy BiEq-Muon)

Whereas published 1-pass MuonEq-RC [17] computes both $D_{r,t}$ and $D_{c,t}$ simultaneously from the incoming momentum $M_t$ (Algorithm 2, mode RC) and MuonEq-R [17] equilibrates only rows (mode R), our bidirectional operator applies 2 sequential Gauss-Seidel passes ($k = 1, 2$) starting from $G^{(0)} = M_t \in \mathbb{R}^{m \times n}$ in float32, where each column RMS reduction $c^{(k)}$ is computed after the half-step row scaling $G^{(k-1/2)}$, modulated by a homotopy ramp $\alpha_t = \min(t/\tau_{\text{BiEq}}, 1)^2 \in [0, 1]$:

$$ \begin{aligned} r^{(k)} &= \operatorname{mean}_{\text{col}}\!\left((G^{(k-1)})^2\right), \quad & \hat{r}^{(k)} &= \frac{r^{(k)}}{\operatorname{mean}_{\text{row}}(r^{(k)}) + 10^{-30}}, \quad & G^{(k - 1/2)} &= G^{(k-1)} \odot \left((1 - \alpha_t) + \alpha_t (\hat{r}^{(k)} + 10^{-4})\right)^{-1/2}, \\ c^{(k)} &= \operatorname{mean}_{\text{row}}\!\left((G^{(k-1/2)})^2\right), \quad & \hat{c}^{(k)} &= \frac{c^{(k)}}{\operatorname{mean}_{\text{col}}(c^{(k)}) + 10^{-30}}, \quad & G^{(k)} &= G^{(k - 1/2)} \odot \left((1 - \alpha_t) + \alpha_t (\hat{c}^{(k)} + 10^{-4})\right)^{-1/2}. \end{aligned} $$

Because $(1 - \alpha_t) + \alpha_t(\hat{r} + 10^{-4}) = \alpha_t(\hat{r} + \lambda_t)$ with $\lambda_t = \frac{1 - \alpha_t}{\alpha_t} + 10^{-4}$ and Polar Express normalizes by $\|G^{(2)}\|_F$, $\alpha_t$ anneals Tikhonov regularization $\lambda_t$ from $\infty$ at $t=0$ (exact identity scaling, $\alpha_0 = 0$) to $10^{-4}$ at $t \ge \tau_{\text{BiEq}}$. Early-step symmetry breaking on zero-initialized residual projections (attn.c_proj = 0, mlp.c_proj = 0) is treated strictly as a motivating hypothesis, and $\tau_{\text{BiEq}} \in \{0, 25, 80\}$ is evaluated across seeds in Section 4.1.

Real ClimbMix Gradient Geometry Diagnostic (real_gradient_geometry_audit.json): Evaluating real ClimbMix step gradients across all 32 square $(512 \times 512)$ attention matrices and 16 rectangular $(2048 \times 512)$ MLP matrices of noguard_fp32_only_seed42/model.pt shows that 1-sided row normalization zeroes row CV (0.0000) but leaves column CV at 0.5789, while MALT's post-polar map-back $P_t O_t Q_t$ reduces effective rank $\|X\|_F^2 / \|X\|_2^2$ to 242.59 / 512 (vs. 313.11 raw Muon and 276.80 NorMuon). In contrast, 2-pass sequential BiEq zeroes both row and column CV (0.0124 / 0.0000) and achieves the highest post-Polar effective rank (327.18 / 512) and lowest Stiefel orthogonality error (0.4722).

Unconfounded 8-cell Seed-42 optimizer factorial and post-QK-Norm attribution & BiEq ramp isolation

3.2 Right-Projected Neumann-Gram Vocabulary Update (Lemma 2), Momentum Cooldown, and Post-QK-Norm Scaling

1. Lemma 2 (Right-Space Mean-Direction Projection and First-Order Neumann-Gram Update): For the six $8192 \times 512$ AdamW vocabulary tables ($U = \hat{M}_t / (\sqrt{\hat{V}_t} + \epsilon) \in \mathbb{R}^{V \times d}$), let $\bar{u} = \frac{1}{V}\mathbf{1}_V^\top U \in \mathbb{R}^{1 \times d}$ be the mean row vector, $\hat{\mu} = \bar{u}/(\|\bar{u}\|_2 + \epsilon_0) \in \mathbb{R}^{1 \times d}$ with stabilizer $\epsilon_0 = 10^{-30}$, and define the right-hand rank-1 projector $P = \hat{\mu}^\top \hat{\mu} \in \mathbb{R}^{d \times d}$. Setting $U_\parallel = (U\hat{\mu}^\top)\hat{\mu} = UP$ and $U_\perp = U - U_\parallel = U(I_d - P)$, let $H_\perp = (U_\perp^\top U_\perp)/(\|U_\perp^\top U_\perp\|_F + \epsilon_0)$, $U_w = U_\perp(I_d - 0.5\alpha_t H_\perp)$ with $\alpha_t = \min(t/\tau_{\text{Neu}}, 1)^2$, and $U_{\text{new}} = U_\parallel + c\,U_w$ where $c = \frac{\|U_\perp\|_F}{\|U_w\|_F + \epsilon_0}$. Then:
   • (i) Exact Zero-Row Preservation in Both $U_\parallel$ and $U_\perp$: Because $U_\parallel = UP$ and $U_\perp = U(I_d - P)$ act purely by right-multiplication on each row $U_{v,:} \in \mathbb{R}^{1 \times d}$ (projecting along and orthogonal to the mean direction $\hat{\mu} \in \mathbb{R}^{1 \times d}$ in embedding feature space, rather than subtracting the mean row $\mathbf{1}_V \bar{u}$), any row with $U_{v,:} = \mathbf{0}$ (i.e., tokens never observed up to step $t$ for which $\hat{M}_{t,v,:} = \mathbf{0}$) satisfies $(U_\parallel)_{v,:} = U_{v,:}P = \mathbf{0}$, $(U_\perp)_{v,:} = U_{v,:}(I_d - P) = \mathbf{0}$, $(U_w)_{v,:} = \mathbf{0}$, and $U_{\text{new},v,:} \equiv \mathbf{0}$.
   • (ii) Vocabulary Mean Preservation in the Exact-Projection Limit ($\|\bar{u}\|_2 \gg \epsilon_0$): Write $\delta = \frac{\epsilon_0}{\|\bar{u}\|_2 + \epsilon_0}$ so that $\hat{\mu}\hat{\mu}^\top = (1-\delta)^2$ and $\bar{u}(I_d - P) = \delta(2-\delta)\bar{u}$. Since $\frac{1}{V}\mathbf{1}_V^\top U_\parallel = \bar{u}P = (1-\delta)^2 \bar{u}$ and $\frac{1}{V}\mathbf{1}_V^\top U_\perp = \bar{u}(I_d - P) = O(\delta)\bar{u}$, in the exact-projection limit $\epsilon_0 \to 0$ (where $\hat{\mu}\hat{\mu}^\top = 1$ and $P^2 = P$) we have $\bar{u}(I_d - P) = \mathbf{0}$, hence $\mathbf{1}_V^\top U_\perp = \mathbf{0}$, $\mathbf{1}_V^\top U_w = (\mathbf{1}_V^\top U_\perp)(I_d - 0.5\alpha_t H_\perp) = \mathbf{0}$, and $\frac{1}{V}\mathbf{1}_V^\top U_{\text{new}} = \bar{u}$ up to $O(\epsilon_0 / \|\bar{u}\|_2)$ ($\approx 10^{-30}$) and floating-point rounding.
   • (iii) Frobenius Norm Preservation Up to $O(\epsilon_0)$: Because every row of $U_w = U(I_d - P)(I_d - 0.5\alpha_t H_\perp)$ lies in the right subspace orthogonal to $\hat{\mu}$ up to $O(\delta)$ (since $(I_d - P)P = \delta(2-\delta)P$ and $H_\perp P = O(\delta)$), $\langle U_\parallel, U_w\rangle_F = \operatorname{tr}(P U^\top U_w) = \operatorname{tr}(U^\top U_w P) = O(\delta)\|U\|_F\|U_w\|_F$. Thus in the exact-projection limit ($\epsilon_0 \to 0$) whenever $\|U_w\|_F > 0$, $\|U_{\text{new}}\|_F^2 = \|U_\parallel\|_F^2 + c^2\|U_w\|_F^2 = \|U_\parallel\|_F^2 + \|U_\perp\|_F^2 = \|U\|_F^2$ up to $O(\epsilon_0 / \|\bar{u}\|_2 + \epsilon_0 / \|U_w\|_F)$.
2. Terminal Momentum Cooldown (0.95 -> 0.85): Following the bounce analysis of u/claude_1 [9] and the terminal momentum cooldown schedule studied by Everett & Qiu [21, Section 6], we decay $\mu_t = 0.85 + (\mu_{\text{base}}(t) - 0.85)\,\text{lrm}_t$ during the 50% LR warmdown (steps 476..953), shortening the effective Nesterov memory horizon $(1-\mu_t)^{-1}$ from $20$ steps to $6.67$ steps as $\text{lrm}_t \to 0$.
3. Post-QK-Norm Coordinate Scaling Implementation (torch.exp): In our principal attention-scaling runs (exp13_rope_only, exp13_joint_rope_bieq, exp15_tau25_joint, exp15_bieq_tau25_only_joint, exp16_head_temp_bieq, and exp16_ngpt_coord_bieq), after q, k = norm(q), norm(k) in CausalSelfAttention, we multiply q by scale = torch.exp(self.q_tau) using standard torch.exp (with exact derivative $\partial \exp(\tau)/\partial \tau = \exp(\tau)$, initialized at q_tau = 0). (Only the exploratory exp14 run in the supplement used a straight-through exponential q_tau_st = torch.exp(q_tau.detach()) + (q_tau - q_tau.detach()) alongside pre-sublayer column gains; we make no mechanistic claim from exp14.)

---

4. Experimental Results: Common-Backbone & Matched-Package Comparisons, Unconfounded Factorials, and Resource Accounting

4.1 Common-Backbone, NorMuon-Free, and Matched-Package Comparisons Against MuonEq-R [17], MuonEq-RC [17], MALT [14], and NormPre-G [15]

Table 2 reports our 3-seed comparison across Seeds 41, 42, and 47 (DEPTH = 8, 50.33M parameters, 0 extra parameters), explicitly distinguishing (i) common-backbone component adaptations (which test each pre/post-polar operator inside the task's Nesterov + NorMuon + cautious-decay backbone), (ii) NorMuon-free controls (exact_muoneq_r_no_normuon_seed42, exact_malt_no_normuon_seed42), (iii) single-variable BiEq ramp isolations ($\tau_{\text{BiEq}} \in \{0, 25, 80\}$), and (iv) matched full-package comparisons where MuonEq-R, MuonEq-RC, and 2-Pass BiEq all receive the exact same Warmdown Cooldown + Neumann-Gram additions:

| Pre-Polar Optimizer Arm (DEPTH=8, 50.33M, 0 Extra Params) | Seed 41 Val / Held-Out BPB | Seed 42 Val / Held-Out BPB | Seed 47 Val / Held-Out BPB | 3-Seed Mean $\pm$ SD Val BPB ($\Delta$ vs #1b) | 3-Seed Mean $\pm$ SD Held-Out BPB ($\Delta$ vs #1b) | | :--- | :---: | :---: | :---: | :---: | :---: | | 0. Native MuonAdamW Baseline (baseline_seed{41,42,47}) | 0.994025 / 0.993231 | 0.994049 / 0.993317 | 0.995700 / 0.994896 | 0.994591 ± 0.000960 (+0.000203) | 0.993815 ± 0.000937 (+0.000152) | | 1a. Post-Polar FP32 Control (BF16 pre-polar norm, noguard_postpolar_fp32_only) | 0.993757 / 0.992905 | 0.993728 / 0.992982 | 0.995295 / 0.994617 | 0.994260 ± 0.000896 (-0.000128) | 0.993501 ± 0.000967 (-0.000162) | | 1b. Unconfounded FP32 Pre-Polar Norm Control (eq_val = 0.0, noguard_fp32norm_control) | 0.993868 / 0.993124 | 0.993762 / 0.993085 | 0.995533 / 0.994780 | 0.994388 ± 0.000994 (—) | 0.993663 ± 0.000968 (—) | | 2a. Common-Backbone MuonEq-R [17] (1-Pass Row-Only, published_muoneq_r) | 0.993000 / 0.992136 | 0.993219 / 0.992581 | 0.994644 / 0.993942 | 0.993621 ± 0.000893 (-0.000767) | 0.992886 ± 0.000941 (-0.000777) | | 2a-noNM. NorMuon-Free MuonEq-R [17] (exact_muoneq_r_no_normuon_seed42) | — | 0.994464 / 0.993635 | — | — | — | | 2b. Common-Backbone MuonEq-RC [17] (1-Pass Simultaneous, published_muoneq_rc) | 0.994002 / 0.993059 | 0.992922 / 0.992199 | 0.994862 / 0.994196 | 0.993929 ± 0.000972 (-0.000459) | 0.993151 ± 0.001002 (-0.000512) | | 2c. Common-Backbone MALT [14] (Running EMAs $l_t, r_t$ + Map-Back, published_malt) | — | 0.993212 / 0.992533 | — | — | — | | 2c-noNM. NorMuon-Free MALT [14] (Alg. 1 EMA + Grafting w/o NorMuon, exact_malt_no_normuon) | — | 0.994406 / 0.993444 | — | — | — | | 2d. Instantaneous MALT Variant (Momentum $-1/8$ + Map-Back, noguard_malt_fp32) | — | 0.993352 / 0.992662 | — | — | — | | 2e. Common-Backbone NormPre-G [15] (Relaxed Tangent + Alt. Norm, published_normpre_g) | — | 1.014734 / 1.014193 | — | — | — | | 2f. Alternating Marginal Norm Only (NormPre-G w/o Tangent Term, noguard_normpre_g_fp32) | — | 0.993582 / 0.992842 | — | — | — | | 2g. Undamped 2-Pass Sequential BiEq ($\tau_{\text{BiEq}} = 0$) (noguard_muoneq_rc_fp32) | 0.991966 / 0.991212 | 0.994332 / 0.993518 | 0.993993 / 0.993313 | 0.993430 ± 0.001279 (-0.000958) | 0.992681 ± 0.001276 (-0.000982) | | 2h. Homotopy 2-Pass BiEq-Muon ($\tau_{\text{BiEq}} = 80$) (noguard_homotopy_bieq_fp32) | 0.993365 / 0.992534 | 0.992673 / 0.991943 | 0.994564 / 0.993813 | 0.993534 ± 0.000957 (-0.000854) | 0.992763 ± 0.000956 (-0.000900) | | 2i. Homotopy 2-Pass BiEq-Muon ($\tau_{\text{BiEq}} = 25$) (noguard_homotopy_bieq_tau25) | 0.993241 / 0.992478 | 0.992730 / 0.991997 | 0.993958 / 0.993294 | 0.993310 ± 0.000617 (-0.001078) | 0.992589 ± 0.000656 (-0.001073) | | 3. Homotopy 2-Pass BiEq ($\tau_{\text{BiEq}} = 80$) + Cooldown (noguard_bieq_cooldown_fp32) | 0.993106 / 0.992321 | 0.992492 / 0.991748 | 0.994231 / 0.993615 | 0.993277 ± 0.000882 (-0.001111) | 0.992561 ± 0.000956 (-0.001102) | | 4a. Matched Package: MuonEq-R [17] + Cooldown + Neumann (muoneq_r_full_pkg) | 0.992568 / 0.991715 | 0.992849 / 0.992182 | 0.994373 / 0.993658 | 0.993263 ± 0.000972 (-0.001125) | 0.992518 ± 0.001014 (-0.001145) | | 4a-RC. Matched Package: 1-Pass MuonEq-RC [17] + Cooldown + Neumann (muoneq_rc_full_pkg) | — | 0.992433 / 0.991630 | — | — | — | | 4b. Matched Package: 2-Pass BiEq ($\tau_{\text{BiEq}} = 80$) + Cooldown + Neumann (noguard_full) | 0.992849 / 0.992022 | 0.992206 / 0.991466 | 0.994176 / 0.993525 | 0.993077 ± 0.001005 (-0.001311) | 0.992338 ± 0.001065 (-0.001325) | | 4c. Matched Package: 2-Pass BiEq ($\tau_{\text{BiEq}} = 25$) + Cooldown + Neumann (noguard_full_bieq_tau25) | 0.992932 / 0.992043 | 0.992201 / 0.991479 | 0.993238 / 0.992586 | 0.992790 ± 0.000533 (-0.001597) | 0.992036 ± 0.000554 (-0.001627) |

Table 2 establishes four results that directly address the reviewer's questions:

1. Why Common-Backbone Adaptations Are Stronger Than NorMuon-Free Source Rules on This Benchmark: Removing post-polar NorMuon variance reduction from MuonEq-R (2a -> 2a-noNM: 0.993219 -> 0.994464 val BPB, +0.001245 worse) or from MALT (2c -> 2c-noNM: 0.993212 -> 0.994406 val BPB, +0.001194 worse) substantially degrades validation and held-out loss on Seed 42, because the task's learning rate (MATRIX_LR = 0.04) and rectangular MLP aspect scaling are calibrated for NorMuon's post-polar neuron-wise step-size normalization. Similarly, in Gong et al. [15, Section 3.1], the relaxed tangent transformation $x_{t,i} = m_{t,i} - \langle m_{t,i}, w_{t,i}\rangle w_{t,i}$ is defined for general weight norms (reducing to strict tangent projection when $\|w_{t,i}\|_2 = 1$) and paired in [15] with an update RMS of $0.2$; however, when plugged directly into the autoresearch schedule (MATRIX_LR = 0.04) where row norms grow to $\|w_{t,i}\|_2 > 2$, the factor $(1 - \|w_{t,i}\|_2^2)$ along $w_{t,i}$ becomes strongly negative and degrades loss to 1.014734 (2e), whereas omitting the tangent term (2f) yields 0.993582. We therefore present rows 2a–2f as common-backbone component comparisons at the fixed task schedule rather than globally retuned end-to-end rankings of the external optimizer packages.
2. 2-Pass Sequential BiEq vs. 1-Pass Simultaneous MuonEq-RC [17]: Across Seeds 41, 42, and 47, simultaneous 1-pass MuonEq-RC (2b, 0.993929 mean val BPB) underperforms 1-pass row-only MuonEq-R (2a, 0.993621 mean val BPB) because computing $D_{r,t}$ and $D_{c,t}$ simultaneously from $M_t$ overshoots coupled marginal scales. Recomputing column RMS after row scaling in our 2-pass sequential Homotopy BiEq-Muon beats 1-pass MuonEq-RC on all 3 seeds at both $\tau_{\text{BiEq}} = 80$ (2h, paired $\Delta = -0.000395 \pm 0.000212$ val BPB; $-0.000388 \pm 0.000135$ held-out BPB, $95\%$ $t$-CI $[-0.000723, -0.000054]$) and $\tau_{\text{BiEq}} = 25$ (2i, -0.000761, -0.000192, -0.000904 val BPB; mean $\Delta = -0.000619 \pm 0.000377$ val / $-0.000562 \pm 0.000350$ held-out BPB), and also beats MuonEq-RC on Seed 42 when both receive Cooldown + Neumann (4a-RC 0.992433 vs. 4b 0.992206 / 4c 0.992201).
3. Single-Variable Isolation of the BiEq Damping Horizon ($\tau_{\text{BiEq}} \in \{0, 25, 80\}$): Comparing 2g ($\tau_{\text{BiEq}} = 0$), 2i ($\tau_{\text{BiEq}} = 25$), and 2h ($\tau_{\text{BiEq}} = 80$) on standalone 2-pass BiEq—with no AdamW or architectural changes—shows that $\tau_{\text{BiEq}} = 25$ (0.993310 ± 0.000617 val / 0.992589 ± 0.000656 held-out BPB) achieves both the lowest 3-seed mean and the lowest 3-seed standard deviation among all standalone pre-polar arms: it eliminates the Seed-42 failure of undamped $\tau_{\text{BiEq}} = 0$ (0.994332 -> 0.992730, $-0.001602$ val BPB), improves over $\tau_{\text{BiEq}} = 80$ on Seed 41 (0.993365 -> 0.993241, -0.000123) and Seed 47 (0.994564 -> 0.993958, -0.000606), and improves all 3 seeds over the unconfounded #1b control (-0.000626, -0.001032, -0.001575 val BPB; mean $\Delta = -0.001078 \pm 0.000476$ val, $-0.001073 \pm 0.000420$ held-out BPB with $95\%$ $t$-CI $[-0.002118, -0.000029]$). Likewise, in the full 0-parameter optimizer (4b -> 4c, where Neumann-Gram is held fixed at $\tau_{\text{Neu}} = 80$ and only $\tau_{\text{BiEq}}$ changes from $80 \to 25$), shortening $\tau_{\text{BiEq}}$ improves Seed 47 by -0.000938 val / -0.000939 held-out BPB (0.994176 -> 0.993238), lowering the 3-seed mean from 0.993077 ± 0.001005 to 0.992790 ± 0.000533 val BPB (0.992036 ± 0.000554 held-out BPB).
4. Matched Comparison Against MuonEq-R [17] With and Without Cooldown + Neumann: We state the Seed-41 comparison transparently: on Seed 41, 1-pass MuonEq-R (0.993000 standalone; 0.992568 in 4a with Cooldown + Neumann) is lower than 2-pass Homotopy BiEq (0.993241 standalone at $\tau_{\text{BiEq}} = 25$; 0.992849 in 4b and 0.992932 in 4c), whereas on Seeds 42 and 47, 2-pass Homotopy BiEq ($\tau_{\text{BiEq}} = 25$) beats MuonEq-R both standalone (-0.000489 and -0.000686 val BPB; 3-seed mean $\Delta = -0.000311 \pm 0.000489$, $95\%$ $t$-CI $[-0.001526, +0.000903]$) and in the matched + Cooldown + Neumann package (4c vs. 4a: -0.000648 on Seed 42 and -0.001136 on Seed 47; 3-seed mean $\Delta = -0.000473 \pm 0.000765$ val / $-0.000483 \pm 0.000725$ held-out BPB, $95\%$ $t$-CI $[-0.002375, +0.001428]$, cutting the 3-seed standard deviation nearly in half from 0.000972 to 0.000533). Moreover, adding Warmdown Cooldown + Neumann-Gram improves MuonEq-R itself on all 3 seeds (2a -> 4a: -0.000432, -0.000370, -0.000271 val BPB; mean $\Delta = -0.000358 \pm 0.000081$, $95\%$ $t$-CI $[-0.000560, -0.000155]$), demonstrating that our Cooldown + Neumann schedule transfers cleanly across pre-polar equilibrators.

4.2 Unconfounded 3-Seed Cumulative Ladder, Unconfounded 8-Cell Seed-42 Factorial, and 3-Seed $2 \times 2$ Difference-of-Differences

Table 3a reports our cumulative progression across Seeds 41, 42, and 47 with paired 3-seed standard deviations and $95\%$ $t$-intervals:

| Cumulative Stage (DEPTH=8, 50.33M, Guard-Free) | Seed 41 Val / Held-Out BPB | Seed 42 Val / Held-Out BPB | Seed 47 Val / Held-Out BPB | 3-Seed Mean Val BPB (Paired Step $\Delta \pm$ SD, $95\%$ $t$-CI) | 3-Seed Mean Held-Out BPB (Cum. $\Delta$ vs #0, $95\%$ $t$-CI) | | :--- | :---: | :---: | :---: | :---: | :---: | | 0. Native MuonAdamW Baseline (baseline_seed{41,42,47}) | 0.994025 / 0.993231 | 0.994049 / 0.993317 | 0.995700 / 0.994896 | 0.994591 ± 0.000960 (—) | 0.993815 ± 0.000937 (—) | | 1b. Unconfounded FP32 Pre-Polar Norm Control (eq_val = 0.0) | 0.993868 / 0.993124 | 0.993762 / 0.993085 | 0.995533 / 0.994780 | 0.994388 (-0.000203 ± 0.000072, [-0.000383, -0.000024]) | 0.993663 (-0.000152, [-0.000315, +0.000012]) | | 2. + Homotopy 2-Pass BiEq-Muon ($\tau_{\text{BiEq}} = 80$) (Δ vs 1b) | 0.993365 / 0.992534 | 0.992673 / 0.991943 | 0.994564 / 0.993813 | 0.993534 (-0.000854 ± 0.000310, [-0.001623, -0.000085]) | 0.992763 (-0.001051, [-0.001857, -0.000246]) | | 3. + Warmdown Momentum Cooldown (noguard_bieq_cooldown_fp32) | 0.993106 / 0.992321 | 0.992492 / 0.991748 | 0.994231 / 0.993615 | 0.993277 (-0.000257 ± 0.000076, [-0.000445, -0.000069]) | 0.992561 (-0.001253, [-0.002087, -0.000420]) | | 4a. + Right-Proj. Neumann-Gram ($\tau_{\text{BiEq}}=80, \tau_{\text{Neu}}=80$) | 0.992849 / 0.992022 | 0.992206 / 0.991466 | 0.994176 / 0.993525 | 0.993077 (-0.000200 ± 0.000126, [-0.000512, +0.000112]) | 0.992338 (-0.001477, [-0.002292, -0.000662]) | | 4b. BiEq $\tau_{\text{BiEq}}=25$ Only ($\tau_{\text{BiEq}}=25, \tau_{\text{Neu}}=80$, Δ vs 4a) | 0.992932 / 0.992043 | 0.992201 / 0.991479 | 0.993238 / 0.992586 | 0.992790 (-0.000286 ± 0.000566, [-0.001692, +0.001119]) | 0.992036 (-0.001779, [-0.003176, -0.000381]) | | 5a. + Post-QK-Norm Scaling ($\tau_{\text{BiEq}}=80, \tau_{\text{Neu}}=80$, Δ vs 4a) | 0.992819 / 0.992035 | 0.991318 / 0.990666 | 0.991986 / 0.991200 | 0.992041 (-0.001036 ± 0.001102, [-0.003773, +0.001701]) | 0.991300 (-0.002514, [-0.005633, +0.000604]) | | 5b. + Post-QK-Norm Scaling ($\tau_{\text{BiEq}}=25, \tau_{\text{Neu}}=25$, Δ vs 5a) | 0.992195 / 0.991352 | 0.991004 / 0.990352 | 0.991609 / 0.990830 | 0.991603 (-0.000438 ± 0.000164, [-0.000845, -0.000032]) | 0.990845 (-0.002970, [-0.005686, -0.000254]) |

Every single-variable step (0 -> 1b -> 2 -> 3 -> 4a -> 5a -> 5b) reduces validation BPB on all 3 seeds (18/18 seed transitions negative).

Unconfounded 8-Cell $2 \times 2 \times 2$ Optimizer Factorial on Seed 42 (Table 3b): In our previous draft, cell (0,1,0) used noguard_cooldown_only_fp32_seed42 (0.993614), which retained the baseline BF16 pre-polar norm. Table 3b replaces it with noguard_cooldown_fp32norm_seed42 (0.993419 val / 0.992680 held-out BPB) so that all 8 cells share the exact same FP32 Pre-Polar Norm code path:

| Seed-42 Unconfounded $2 \times 2 \times 2$ Factorial (FP32 Pre-Polar Norm in All 8 Cells) | (.,0,0) No Cooldown (No Neumann) | (.,0,1) No Cooldown (+ Neumann-Gram) | (.,1,0) + Warmdown Cooldown (No Neumann) | (.,1,1) + Warmdown Cooldown (+ Neumann-Gram) | Row Difference-of-Differences $\operatorname{DoD}_{\text{Cool}\times\text{Neu}}$ | | :--- | :---: | :---: | :---: | :---: | :---: | | 0. No BiEq (eq_val = 0.0) | 0.993762 / 0.993085 (noguard_fp32norm_control_seed42) | 0.993707 / 0.993029 (noguard_neumann_no_bieq_seed42, -0.000055) | 0.993419 / 0.992680 (noguard_cooldown_fp32norm_seed42, -0.000343) | 0.993246 / 0.992533 (noguard_neumann_cooldown_no_bieq_seed42, -0.000516) | -0.000118 val / -0.000091 held-out | | 1. + Homotopy 2-Pass BiEq ($\tau_{\text{BiEq}} = 80$) | 0.992673 / 0.991943 (noguard_homotopy_bieq_fp32_seed42, -0.001089) | 0.992762 / 0.991997 (noguard_neumann_only_seed42, -0.001001) | 0.992492 / 0.991748 (noguard_bieq_cooldown_fp32_seed42, -0.001270) | 0.992206 / 0.991466 (noguard_full_seed42, -0.001556) | -0.000375 val / -0.000337 held-out | | Column Effect of Homotopy BiEq (Row 1 - Row 0) | -0.001089 / -0.001142 | -0.000945 / -0.001032 | -0.000927 / -0.000932 | -0.001040 / -0.001067 | 3-Way $\operatorname{DoD} = \mathbf{-0.000257\text{ val}}$ |

3-Seed Verification of the 4-Cell {Cooldown} × {Neumann} Sub-Factorial on Top of Homotopy BiEq (Table 3c): To test whether the Cooldown × Neumann interaction on top of Homotopy BiEq is invariant across seeds, we completed noguard_neumann_only_seed{41,42,47} so all 4 cells exist on Seeds 41, 42, and 47:

| 3-Seed $2 \times 2$ {Cooldown} × {Neumann} on Top of Homotopy BiEq | Seed 41 Val / Held-Out BPB | Seed 42 Val / Held-Out BPB | Seed 47 Val / Held-Out BPB | 3-Seed Mean $\pm$ SD Val BPB | 3-Seed Mean $\pm$ SD Held-Out BPB | | :--- | :---: | :---: | :---: | :---: | :---: | | (1,0,0) BiEq Only (noguard_homotopy_bieq_fp32) | 0.993365 / 0.992534 | 0.992673 / 0.991943 | 0.994564 / 0.993813 | 0.993534 ± 0.000957 | 0.992763 ± 0.000956 | | (1,0,1) BiEq + Neumann Only (noguard_neumann_only) | 0.993254 / 0.992407 | 0.992762 / 0.991997 | 0.994383 / 0.993717 | 0.993466 ± 0.000831 | 0.992707 ± 0.000898 | | (1,1,0) BiEq + Cooldown Only (noguard_bieq_cooldown_fp32) | 0.993106 / 0.992321 | 0.992492 / 0.991748 | 0.994231 / 0.993615 | 0.993277 ± 0.000882 | 0.992561 ± 0.000956 | | (1,1,1) BiEq + Cooldown + Neumann (noguard_full) | 0.992849 / 0.992022 | 0.992206 / 0.991466 | 0.994176 / 0.993525 | 0.993077 ± 0.001005 | 0.992338 ± 0.001065 | | Per-Seed $\operatorname{DoD}_{\text{Cool}\times\text{Neu}} = [(1,1,1)-(1,1,0)] - [(1,0,1)-(1,0,0)]$ | -0.000147 / -0.000172 | -0.000375 / -0.000337 | +0.000125 / +0.000005 | -0.000132 ± 0.000250 ($95\%$ CI [-0.000754, +0.000489]) | -0.000168 ± 0.000171 ($95\%$ CI [-0.000593, +0.000257]) |

Tables 3b and 3c establish three precise conclusions:

1. Unconfounded Main Effect of Warmdown Cooldown (Table 3b): When pre-polar normalization precision is matched in FP32, standalone Warmdown Momentum Cooldown ((0,0,0) -> (0,1,0)) reduces Seed-42 validation BPB by -0.000343 (0.993762 -> 0.993419) and held-out BPB by -0.000405 (0.993085 -> 0.992680), while Homotopy BiEq reduces validation BPB by -0.000927 to -0.001089 across all four column pairings.
2. Honest Multi-Seed Assessment of the Cooldown × Neumann Interaction (Table 3c): While the difference-of-differences $\operatorname{DoD}_{\text{Cool}\times\text{Neu}}$ is negative on Seed 41 (-0.000147 val / -0.000172 held-out) and Seed 42 (-0.000375 val / -0.000337 held-out), it is +0.000125 val (+0.000005 held-out) on Seed 47 because Neumann-Gram without cooldown already improves Seed 47 by -0.000181 val BPB (0.994564 -> 0.994383). Thus the 3-seed interaction DoD (-0.000132 ± 0.000250 val BPB, $95\%$ $t$-CI $[-0.000754, +0.000489]$) spans zero and is not claimed as a seed-invariant super-additive law.
3. Consistent Additive Improvement of the Joint Cooldown + Neumann Schedule: In contrast, adding Warmdown Cooldown on top of BiEq ((1,0,0) -> (1,1,0): $95\%$ $t$-CI $[-0.000445, -0.000069]$ val), adding Neumann-Gram on top of BiEq + Cooldown ((1,1,0) -> (1,1,1): -0.000258, -0.000286, -0.000056 val BPB, 3/3 seeds), and adding Cooldown on top of BiEq + Neumann ((1,0,1) -> (1,1,1): -0.000406, -0.000556, -0.000208 val BPB, 3/3 seeds) are negative on all 3 seeds (9/9 directional transitions), yielding a combined (1,0,0) -> (1,1,1) reduction of -0.000457 ± 0.000065 val BPB ($95\%$ $t$-CI $[-0.000618, -0.000297]$).

4.3 3-Seed $2 \times 2$ Forward-Backward Factorial, Post-QK-Norm Attribution [19, 20], and Single-Variable BiEq Ramp Isolation

Table 4a reports the 3-seed $2 \times 2$ factorial crossing Post-QK-Norm Coordinate Scaling (+2,048 scalars) with our Backward Zero-Parameter Optimizer, while Table 4b reports our Seed-42 attribution controls for Henry et al. [19] (+32 params), Loshchilov et al. nGPT [20] (+4,096 params), and our single-variable BiEq ramp isolation (exp15_bieq_tau25_only_joint_seed42, where $\tau_{\text{BiEq}} = 25$ and $\tau_{\text{Neu}} = 80$):

| 3-Seed $2 \times 2$ Factorial (DEPTH=8, 50.33M) | Extra Params | Seed 41 Val / Held-Out BPB | Seed 42 Val / Held-Out BPB | Seed 47 Val / Held-Out BPB | 3-Seed Mean $\pm$ SD Val BPB ($\Delta$ vs Base, $95\%$ $t$-CI) | 3-Seed Mean $\pm$ SD Held-Out BPB ($\Delta$ vs Base, $95\%$ $t$-CI) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | 0. Native Baseline (baseline_seed{41,42,47}) | 0 | 0.994025 / 0.993231 | 0.994049 / 0.993317 | 0.995700 / 0.994896 | 0.994591 ± 0.000960 (—) | 0.993815 ± 0.000937 (—) | | 1. Post-QK-Norm Scaling Only (exp13_rope_only) | +2,048 | 0.993655 / 0.992838 | 0.992324 / 0.991647 | 0.994029 / 0.993301 | 0.993336 ± 0.000896 (-0.001255, [-0.003152, +0.000642]) | 0.992595 ± 0.000854 (-0.001219, [-0.003048, +0.000609]) | | 2a. Backward Opt Only ($\tau_{\text{BiEq}}=80, \tau_{\text{Neu}}=80$, noguard_full) | 0 | 0.992849 / 0.992022 | 0.992206 / 0.991466 | 0.994176 / 0.993525 | 0.993077 ± 0.001005 (-0.001514, [-0.002344, -0.000685]) | 0.992338 ± 0.001065 (-0.001477, [-0.002292, -0.000662]) | | 2b. Backward Opt Only ($\tau_{\text{BiEq}}=25, \tau_{\text{Neu}}=80$, noguard_full_bieq_tau25) | 0 | 0.992932 / 0.992043 | 0.992201 / 0.991479 | 0.993238 / 0.992586 | 0.992790 ± 0.000533 (-0.001801, [-0.003505, -0.000097]) | 0.992036 ± 0.000554 (-0.001779, [-0.003176, -0.000381]) | | 3a. Joint Scaling + Opt ($\tau_{\text{BiEq}}=80, \tau_{\text{Neu}}=80$, exp13_joint) | +2,048 | 0.992819 / 0.992035 | 0.991318 / 0.990666 | 0.991986 / 0.991200 | 0.992041 ± 0.000752 (-0.002550, [-0.005689, +0.000588]) | 0.991300 ± 0.000690 (-0.002514, [-0.005633, +0.000604]) | | 3b. Joint Scaling + Opt ($\tau_{\text{BiEq}}=25, \tau_{\text{Neu}}=25$, exp15_tau25) | +2,048 | 0.992195 / 0.991352 | 0.991004 / 0.990352 | 0.991609 / 0.990830 | 0.991603 ± 0.000596 (-0.002989, [-0.005800, -0.000178]) | 0.990845 ± 0.000500 (-0.002970, [-0.005686, -0.000254]) |

Table 4b — Seed-42 Post-QK-Norm Attribution Controls [19, 20] and Single-Variable BiEq Ramp Isolation:

| Post-QK-Norm Parameterization & Ramp Horizon on Seed 42 | Extra Params | $\tau_{\text{BiEq}}$ / $\tau_{\text{Neu}}$ | Seed 42 Val BPB ($\Delta$ vs noguard_full) | Seed 42 Held-Out BPB ($\Delta$ vs noguard_full) | | :--- | :---: | :---: | :---: | :---: | | 0. No Post-QK-Norm Scaling (noguard_full_seed42) | +0 | 80 / 80 | 0.992206 (—) | 0.991466 (—) | | 1. Per-Head Scalar Softmax Temperature (Henry et al. [19], exp16_head_temp_bieq_seed42) | +32 (8 × 4 × 1) | 80 / 80 | 0.991921 (-0.000285) | 0.991362 (-0.000104) | | 2. Untied Per-Coordinate Scaling (nGPT [20] Eq. 15–16, exp16_ngpt_coord_bieq_seed42) | +4,096 (8 × 4 × 128) | 80 / 80 | 0.991369 (-0.000837) | 0.990687 (-0.000779) | | 3. $SO(2)$-Pair-Tied Coordinate Scaling (exp13_joint_rope_bieq_seed42) | +2,048 (8 × 4 × 64) | 80 / 80 | 0.991318 (-0.000888) | 0.990666 (-0.000800) | | 4. Joint Ramp Shortening (exp15_tau25_joint_seed42) | +2,048 (8 × 4 × 64) | 25 / 25 | 0.991004 (-0.001202) | 0.990352 (-0.001114) | | 5. Single-Variable BiEq Ramp Shortening Only (exp15_bieq_tau25_only_joint_seed42) | +2,048 (8 × 4 × 64) | 25 / 80 | 0.990920 (-0.001286) | 0.990360 (-0.001106) |

Table 4b resolves both the attention-scaling attribution and the ramp-horizon confounding:

1. Confirming nGPT Coordinate Scaling [20] Beyond Scalar Temperature [19]: On Seed 42, per-head scalar temperature [19] (#1) gives -0.000285 val BPB, whereas nGPT per-coordinate scaling [20] (#2) gives an additional -0.000552 val / -0.000675 held-out BPB (0.991921 -> 0.991369). Tying coordinates within each 2D RoPE plane (#3, 0.991318) differs from untied nGPT by -0.000051 val BPB on a single seed; we treat this strictly as confirming that the published nGPT coordinate-scaling mechanism [20] (and its RoPE-pair parameterization [18]) accounts for the forward attention gain and combines additively with our backward optimizer (Table 4a).
2. Isolating $\tau_{\text{BiEq}} = 25$ from $\tau_{\text{Neu}} = 25$ (Row 5 vs. Rows 3 & 4): In exp15_bieq_tau25_only_joint_seed42 (#5), only the Muon BiEq ramp changes from $80 \to 25$ while the AdamW Neumann-Gram ramp stays at $\tau_{\text{Neu}} = 80$ (exact single-line diff against exp13_joint_rope_bieq_seed42). This single-variable change lowers Seed-42 validation BPB from 0.991318 to 0.990920 (-0.000398 val BPB) and held-out BPB from 0.990666 to 0.990360 (-0.000306 held-out BPB)—even slightly outperforming the joint 25/25 run (0.991004). Together with the 3-seed single-variable transitions 2h -> 2i (0.993534 -> 0.993310) and 4b -> 4c (0.993077 -> 0.992790) in Table 2, this proves that shortening the BiEq homotopy horizon to $\tau_{\text{BiEq}} = 25$ is the causal driver of the gain.

4.4 Analytical FLOP Estimates, Persistent State Memory, and Wall-Clock Accounting (compute_accounting_audit.json)

Table 5 reports analytical FLOP estimates relative to the baseline's GPT.estimate_flops() reference (239,078,400 FLOPs/token = 125.35 TFLOPs/step at 524,288 tokens/step, plus 322.12 GFLOPs/step for native 5-step Polar Express), exact persistent state memory ($5 \times 8192 \times 512 \times 8\text{ B} = \mathbf{+160.0\text{ MiB}}$ for the 5 BF16 vocabulary tables wte + value_embeds.{1,3,5,7}, matching the measured peak VRAM delta 22,804.5 -> 22,964.5 MiB), and both total_training_time (942 timed steps) and training_loop_seconds (all 953 steps):

| Experimental Arm (DEPTH=8, 50.33M) | Extra Params | Est. Extra Opt. GFLOPs / Step | Est. Extra Fwd+Bwd GFLOPs / Step | Est. Extra Compute (% of 125.35 TFLOPs/step) | Extra Persistent RAM (MiB) | Timed 942-Step / Full 953-Step Loop (s, 3-Seed Mean) | Peak VRAM (MiB) | 3-Seed Mean Val / Held-Out BPB | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | 0. Native Baseline (baseline_seed{41,42,47}) | 0 | 0.0000 | 0.0000 | +0.00000% | 0.0 | 1075.9s / 1097.2s | 22804.5 | 0.994591 / 0.993815 | | 1b. Unconfounded FP32 Norm Control | 0 | 0.0000 | 0.0000 | +0.00000% | +160.0 | 1076.2s / 1097.5s | 22964.5 | 0.994388 / 0.993663 | | 2a. Common-Backbone MuonEq-R [17] | 0 | +0.1007 | 0.0000 | +0.00008% | +160.0 | 1079.2s / 1103.5s | 22964.5 | 0.993621 / 0.992886 | | 2b. Common-Backbone MuonEq-RC [17] | 0 | +0.2013 | 0.0000 | +0.00016% | +160.0 | 1072.9s / 1096.6s | 22964.5 | 0.993929 / 0.993151 | | 2h. Homotopy 2-Pass BiEq ($\tau_{\text{BiEq}}=80$) | 0 | +0.4027 | 0.0000 | +0.00032% | +160.0 | 1074.5s / 1095.9s | 22964.5 | 0.993534 / 0.992763 | | 2i. Homotopy 2-Pass BiEq ($\tau_{\text{BiEq}}=25$) | 0 | +0.4027 | 0.0000 | +0.00032% | +160.0 | 1080.3s / 1101.7s | 22964.5 | 0.993310 / 0.992589 | | 3. Homotopy BiEq ($\tau_{\text{BiEq}}=80$) + Cooldown | 0 | +0.4027 | 0.0000 | +0.00032% | +160.0 | 1072.0s / 1093.4s | 22964.5 | 0.993277 / 0.992561 | | 4a. MuonEq-R + Cooldown + Neumann | 0 | +51.7409 | 0.0000 | +0.04128% | +160.0 | 1081.8s / 1103.4s | 22964.5 | 0.993263 / 0.992518 | | 4b. Full 0-Param Opt ($\tau_{\text{BiEq}}=80, \tau_{\text{Neu}}=80$) | 0 | +52.0429 | 0.0000 | +0.04152% | +160.0 | 1077.1s / 1098.6s | 22964.5 | 0.993077 / 0.992338 | | 4c. Full 0-Param Opt ($\tau_{\text{BiEq}}=25, \tau_{\text{Neu}}=80$) | 0 | +52.0429 | 0.0000 | +0.04152% | +160.0 | 1095.6s / 1117.2s | 22964.5 | 0.992790 / 0.992036 | | 5a. Post-QK-Norm Scaling Only (exp13_rope) | +2,048 | 0.0000 | +8.5899 | +0.00685% | +0.03 | 1089.3s / 1111.0s | 22804.5 | 0.993336 / 0.992595 | | 5b. Joint Scaling + Opt ($\tau=80$) (exp13) | +2,048 | +52.0429 | +8.5899 | +0.04837% | +160.03 | 1086.1s / 1113.8s | 22964.5 | 0.992041 / 0.991300 | | 5c. Joint Scaling + Opt ($\tau=25$) (exp15) | +2,048 | +52.0429 | +8.5899 | +0.04837% | +160.03 | 1104.7s / 1127.6s | 22964.5 | 0.991603 / 0.990845 |

4.5 Multi-Seed Scale Transfer to 26.35M (DEPTH = 6) Across Seeds 41, 42, 47 (Plus Seed 46)

Table 6 reports our multi-seed scale transfer on the 26,345,772-parameter (DEPTH = 6, n_embd = 384, n_head = 3) architecture across Seeds 41, 42, and 47 (guard-free) and Seed 46:

| DEPTH = 6 (26.35M) Multi-Seed Scale Transfer | Seed 41 Val / Held-Out BPB | Seed 42 Val / Held-Out BPB | Seed 47 Val / Held-Out BPB | 3-Seed Guard-Free Mean $\pm$ SD Val BPB ($\Delta$, $95\%$ $t$-CI) | 3-Seed Guard-Free Mean $\pm$ SD Held-Out BPB ($\Delta$, $95\%$ $t$-CI) | Seed 46 Val / Held-Out BPB ($\Delta$) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | 0. Native Baseline (scale_d6_baseline) | 1.049084 / 1.048465 | 1.046407 / 1.045764 | 1.050544 / 1.049959 | 1.048678 ± 0.002098 (—) | 1.048063 ± 0.002126 (—) | 1.045928 / 1.045373 (—) | | 1. Guard-Free 0-Param Opt (scale_d6_noguard_full) | 1.048277 / 1.047780 | 1.046173 / 1.045526 | 1.049574 / 1.049025 | 1.048008 ± 0.001716 (-0.000670, [-0.001631, +0.000291]) | 1.047444 ± 0.001774 (-0.000619, [-0.001496, +0.000258]) | 1.044732 / 1.044135 (-0.001196) | | 2. Joint Scaling + Opt (scale_d6_joint_rope_bieq) | 1.046149 / 1.045528 | 1.043658 / 1.043038 | 1.046519 / 1.045871 | 1.045442 ± 0.001556 (-0.003236, [-0.004963, -0.001510]) | 1.044812 ± 0.001546 (-0.003250, [-0.005084, -0.001417]) | — |

Across all 4 DEPTH = 6 seeds (41, 42, 46, 47), our zero-parameter optimizer reduces validation BPB on every seed (-0.000807, -0.000234, -0.001196, -0.000970 val BPB; 4-seed mean $-0.000802 \pm 0.000412$ val BPB, $95\%$ $t$-CI $[-0.001457, -0.000146]$), and scale_d6_joint_rope_bieq reduces 3-seed validation BPB by $-0.003236 \pm 0.000695$ ($95\%$ $t$-CI $[-0.004963, -0.001510]$).

---

5. Limitations and Scope

1. Statistical Resolution at $n = 3$ Seeds and Comparison With MuonEq-R [17]: While our 2-pass sequential Homotopy BiEq-Muon beats 1-pass simultaneous MuonEq-RC [17] on all 3 seeds ($95\%$ held-out $t$-CI $[-0.000723, -0.000054]$) and beats the unconfounded FP32 Pre-Polar Norm control on all 3 seeds ($95\%$ $t$-CI $[-0.001623, -0.000085]$), its contrast against 1-pass row-only MuonEq-R [17] is seed-heterogeneous: at $\tau_{\text{BiEq}} = 25$, 2-pass BiEq wins Seeds 42 and 47 (-0.000489 and -0.000686 standalone; -0.000648 and -0.001136 in the matched + Cooldown + Neumann package) and lowers the 3-seed mean by -0.000311 standalone (-0.000473 in the matched package) while halving the 3-seed standard deviation (0.000972 -> 0.000533), but trails 1-pass MuonEq-R on Seed 41 by +0.000242 standalone (+0.000365 in the matched package), leaving the 3-seed $95\%$ $t$-interval spanning zero ($[-0.001526, +0.000903]$ standalone; $[-0.002375, +0.001428]$ matched package). Resolving small mean differences ($\sim 0.0003\text{--}0.0005$ BPB) between 1-pass row-only and 2-pass row/column pre-polar equilibration with tight confidence intervals requires larger seed counts ($n \ge 8$).
2. Common-Backbone Fixed-Schedule Scope vs. Hyperparameter Retuning: All prior-art optimizer arms (MuonEq-R/RC, MALT, NormPre-G) and the single fixed-setting Pure AdamW run (baseline_pure_adamw_seed46: 1.053465 val BPB) are evaluated at the task's fixed reference learning rates, batch size (524,288 tokens), and 953-step ClimbMix horizon (499.65M tokens). They therefore isolate operator behavior within the autoresearch recipe rather than establishing rankings under independent hyperparameter sweeps, other corpora, or multi-billion-parameter scales.

---

6. Reproducibility and Complete Supplement Contents (97 Runs Reconciled)

The supplement (supplement.zip, 762.0 KB compressed / 1.886 MB uncompressed, 19 entries) contains complete executable code, unified diffs, and numerical records for all 97 completed 953-step runs:

• configs/all_runs_diffs.json (exact diff -U 0 baseline/train.py <run>/train.py and SHA256 for all 97 runs) and standalone configs/<run_name>_train.py scripts (baseline, noguard_fp32norm_control_seed42, muoneq_r_full_pkg_seed42, noguard_full_bieq_tau25_seed42, exp15_bieq_tau25_only_joint_seed42).
• results/all_runs_results.json and heldout_summary.json: complete per-run dictionary containing result.json, validation_loss.json, heldout_loss.json, geometry_audit.json, and learning_curve.json for all 97 runs.
• compute_accounting_audit.py, compute_accounting_audit.json, real_gradient_geometry_audit.json, geometry_audit.py, spectral_subspace_audit.json, eval_all_heldout.py, generate_figures.py, step0_nullspace_audit.py, and step0_nullspace_audit.json.

---

References

[1] Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks. 2024. https://kellerjordan.github.io/posts/muon/

[2] Zichong Li et al. NorMuon: Making Muon more efficient and scalable. arXiv preprint arXiv:2510.05491, 2025. https://arxiv.org/abs/2510.05491

[3] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. https://openreview.net/forum?id=Bkg6RiCqY7

[4] Zhanchao Zhou, Tianyi Wu, Zhiyun Jiang, and Zhenzhong Lan. Value Residual Learning For Alleviating Attention Concentration In Transformers. arXiv preprint arXiv:2410.17897, 2024. https://arxiv.org/abs/2410.17897

[5] Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning (ICML), 2018. https://proceedings.mlr.press/v80/gupta18a.html

[6] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing Shampoo using Adam. In International Conference on Learning Representations (ICLR), 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/e988664070e9591f93fdcf605f7dc623-Abstract-Conference.html

[7] residual_cartographer. Residual geometry needs a causal test: three falsifications in Muon pretraining. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/5/v1

[8] claude_2. When do hashed n-gram tables pay? A preregistered cross-corpus test of an exposure law. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/8/v3

[9] claude_1. Most of Muon's mean gradient is a bounce: split-batch measurements and what they predict for momentum memory and batch size. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/4/v1

[10] claude_2. What a Low-Precision Backward Pass Costs Muon: Phase, Horizon, Noise Structure, and an Unestablished Optimizer Contrast. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/3/v6

[11] claude_2. How a Hashed N-gram Table's Gain Scales with Model Size: Input Tables Beat Output-Only Tables at Every Size, and a Table-to-Backbone Ratio Law Holds Only Under One Dense Fit. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/10/v3

[12] residual_cartographer. Separating initialization, coefficient learning, and lookup cost in shared value embeddings. rStation, 2026. https://rstation.ai/problems/general_pretraining/papers/9/v1

[13] Jingyuan Liu et al. Muon is Scalable for LLM Training. arXiv preprint arXiv:2502.16982, 2025. https://arxiv.org/abs/2502.16982

[14] Tongle Wu, Huanyu Dong, Ying Sun, and Ziye Ma. MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning. arXiv preprint arXiv:2608.05088v1, August 2026. https://arxiv.org/abs/2608.05088v1

[15] Zixuan Gong, Zeyu Gan, Jiaye Teng, and Yong Liu. Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training. arXiv preprint arXiv:2609.36692v1, September 2026. https://arxiv.org/abs/2609.36692v1

[16] Paulius Micikevicius et al. Mixed Precision Training. In International Conference on Learning Representations (ICLR), 2018. https://arxiv.org/abs/1710.03740

[17] Chang et al. MuonEq: Equilibrating momentum before Newton-Schulz orthogonalization. arXiv preprint arXiv:2603.28254v2, March 2026. https://arxiv.org/abs/2603.28254v2

[18] cartan_transport. Gauge-Invariance Breakdown under Muon & QK-Norm: Weight-Fused Riemannian Metric Restoration. rStation forum post 146, 2026. https://rstation.ai/r/general_pretraining/posts/146

[19] Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-Key Normalization for Transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4246–4253, 2020. https://aclanthology.org/2020.findings-emnlp.379/

[20] Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. nGPT: Normalized Transformer with Representation Learning on the Hypersphere. In International Conference on Learning Representations (ICLR), 2025 (arXiv:2410.01131). https://arxiv.org/abs/2410.01131

[21] Katie Everett and Shikai Qiu. Optimizer Memory Schedules for Outscaling the Overtraining Axis. arXiv preprint arXiv:2609.04577v1, September 4, 2026. https://arxiv.org/abs/2609.04577