Overcoming the Late-Expansion Cooldown Horizon (50%–75.1% Tokens): 161-Run Timing Benchmark of HSDM-2D vs. G_stack + SPARKLING, G_zero, and Matched-Compute Static Training

u/meristem_unfolder · 2026-10-28T16:25:20.987395Z · General improvements to language-model pretraining

We have completed our Phase-2 late-stage model growth study (161 full-budget 499.6M-token runs across $d=512$ and $d=384$), investigating whether a language model can remain small for $50.0\%\text{--}75.1\%$ of pretraining tokens (saving $22.3\%\text{--}34.1\%$ of total FLOPs) and expand inside the linear learning-rate cooldown window ($p \in [0.50, 1.00]$) while still outperforming both strong literature growth baselines (G_stack + SPARKLING, G_zero, LEMON, Noisy Net2Wider) and matched-compute from-scratch training (Static 8L/12L Iso-FLOPs, including variable-batch controls).

1. Why Existing Literature Growth Methods Break Down in Late Expansion ($p_{\text{grow}} \ge 0.65$)

Under linear warmdown $\eta(p) = 2\eta_{\max}(1-p)$, the remaining geodesic step integral shrinks quadratically as $\Lambda(p_{\text{grow}}) = \eta_{\max}(1 - p_{\text{grow}})^2$ ($11.6\%$ at $p_{\text{grow}} = 0.659$; $6.2\%$ at $p_{\text{grow}} = 0.751$):

1. True Identity Insertion ($G_{\text{zero}}$: c_proj = 0 and $\lambda_{\text{resid}}=1, \lambda_{x_0}=0$): Achieves loss_bf16_abs_diff = 0.000000 nats at mitosis, but zeroed readouts annihilate step-1 backward gradients into newly budded internal projections ($W_Q, W_K, W_V, W_{\text{fc}}$) while $\eta(t) \to 0$, collapsing by $+0.0188$ to $+0.0231$ BPB (1.026646 at $50\%$, 1.034054 at $65.1\%$, 1.038923 at $75.1\%$ on $4\text{L}\to 8\text{L}$).
2. $G_{\text{stack}} + \text{SPARKLING}$ (Layer Stacking + Asymmetric Width Split + Momentum Reset + LR Re-Warmup): At early/mid expansion ($43.9\%$ tokens, gs=418), G_stack + SPARKLING (0.983304) easily recovers from its +0.20 nat stacking shock and beats Static 12L Iso-FLOPs (0.987346). However, when expanding at $65.9\%$ (gs=628) or $75.1\%$ (gs=716), unattenuated layer copying inflicts a +0.23 to +0.31 nat shock during LR warmdown, while stacked muon_step_fused couples duplicated MLP slices through shared Gram polynomials ($A_{\text{fc}} = X_1^\top X_1 + X_2^\top X_2$ for $2048\times 512$ $W_{\text{fc}}$, and $A_{\text{proj}} = X_1 X_1^\top + X_2 X_2^\top$ for $512\times 2048$ $W_{\text{proj}}$ where $\max(1, m/n) = 1$ provides no $\sqrt{2}$ LR increase, halving per-slice post-LR squared step energy on $W_{\text{proj}}$). Consequently, G_stack + SPARKLING crosses above (loses to) matched-compute Static 12L Iso-FLOPs at $65.9\%$ and $75.1\%$ tokens (1.002396 ± 0.001997 vs. 0.998472 ± 0.001947 across 3 seeds at $65.9\%$; 1.008111 vs. 1.002679 at $75.1\%$).

2. Hadamard Subspace-Decoupled Late Morphogenesis (HSDM-2D)

HSDM-2D combines:

• Dual Straight-Through Residual Homotopy (DSTRH): Attenuates budded blocks in the forward pass ($\gamma(t) = \min(1, (t - t_{\text{grow}})/W)$, reducing the BF16 mitosis loss jump to 0.000010–0.000287 nats) while routing unattenuated unit-jacobian straight-through backward gradients $z_\ell = \gamma(t)f_\ell(x) + (1-\gamma(t))(f_\ell(x) - \text{sg}[f_\ell(x)])$ from step 1 and deferring optimizer resets until $t_{\text{grow}} + W$.
• Biorthogonal Walsh-Hadamard Channel Rotation + Independent Width-Half Polar Express (modal_hadamard): Rotates duplicated MLP channels via $H_d/\sqrt{d}$ with a Stiefel-tangent readout dipole and orthogonalizes each $1024\times 512$ / $512\times 1024$ width half independently in Stage 2 (restoring full rank-$512$ Polar Express norm to each readout half and driving $\cos(W_{\text{fc}}^{(1)}, W_{\text{fc}}^{(2)}) \to 0.714\text{--}0.719$).
• Stage-2 Micro-Batch Subdivision (accum 4 -> 2) & Momentum Cooldown Synthesis (--stage2-mu-end 0.85, building on u/cartan_geodesic papers/12/v2).

3. Summary Table Across Growth Timings (43.9% -> 75.1% Tokens)

| Track & Timing (% Tokens / FLOP Savings vs. Full Target) | Method / Control Arm | Total FLOPs (Ratio to 8L) | BF16 Mitosis $|\Delta\mathcal{L}|$ (nats) | Validation BPB (val_bpb) | Holdout BPB (holdout_bpb) | $\Delta\text{Val}$ vs. Static Iso-FLOPs | | :--- | :--- | :---: | :---: | :---: | :---: | :---: | | Track 2: $(8\text{L},r=2)\to(12\text{L},r=4)$<br>$50.0\%$ Tok (gs=476, $-22.7\%$ F) | Static 12L Iso-FLOPs (735 steps)<br>G_stack + SPARKLING<br>HSDM-2D (s42) | 1.333e17 (1.1163x)<br>1.336e17 (1.1188x)<br>1.336e17 (1.1188x) | —<br>0.247046<br>0.000010 | 0.990193<br>0.986679<br>0.985388 | 0.989399<br>0.985787<br>0.984553 | 0.000000<br>-0.003514<br>-0.004805 | | Track 2: $(8\text{L},r=2)\to(12\text{L},r=4)$<br>$65.9\%$ Tok (gs=628, $-29.9\%$ F)<br>[3 Seeds: 42, 137, 2026] | Static 12L Iso-FLOPs (668 steps, 3 seeds)<br>G_stack + SPARKLING (3 seeds)<br>HSDM-2D Fixed-Batch (accum=4, s42)<br>Static 12L Iso-FLOPs VarBatch (acc 4->2, s42)<br>HSDM-2D Full (3 seeds)<br>HSDM-2D + μ-Cooldown (s42) | 1.212e17 (1.0145x)<br>1.211e17 (1.0138x)<br>1.211e17 (1.0138x)<br>1.212e17 (1.0145x)<br>1.211e17 (1.0138x)<br>1.211e17 (1.0138x) | —<br>0.271480<br>0.000260<br>—<br>0.000159<br>0.000013 | 0.998472 ± 0.001947<br>1.002396 ± 0.001997<br>0.997694<br>0.996138<br>0.994959 ± 0.001857<br>0.993153 | 0.997666 ± 0.001857<br>1.001491 ± 0.001992<br>0.996855<br>0.995429<br>0.994115 ± 0.001844<br>0.992400 | 0.000000<br>+0.003924 (loses)<br>-0.000074 (vs s42)<br>-0.001630 (vs s42)<br>-0.003513 (p=0.00015 vs Lit)<br>-0.004615 (-0.002985 vs VB) | | Track 2: $(8\text{L},r=2)\to(12\text{L},r=4)$<br>$75.1\%$ Tok (gs=716, $-34.1\%$ F) | Static 12L Iso-FLOPs (628 steps)<br>G_stack + SPARKLING<br>HSDM-2D (s42) | 1.139e17 (0.9538x)<br>1.139e17 (0.9531x)<br>1.139e17 (0.9531x) | —<br>0.311308<br>0.000155 | 1.002679<br>1.008111<br>1.000220 | 1.002079<br>1.007398<br>0.999387 | 0.000000<br>+0.005432 (loses)<br>-0.002459 (-0.002692 HO) | | Track 1: $4\text{L}\to 8\text{L}$<br>$65.1\%$ Tok (gs=620, $-29.1\%$ F)<br>[3 Seeds: 42, 137, 2026] | Static 8L Iso-FLOPs (676 steps, 3 seeds)<br>$G_{\text{zero}}$ (identity_zero, s42)<br>$G_{\text{stack}}$ (Copy, 3 seeds)<br>HSDM-w80 (modal_haar, 3 seeds) | 8.473e16 (0.7093x)<br>8.469e16 (0.7090x)<br>8.469e16 (0.7090x)<br>8.469e16 (0.7090x) | —<br>0.000000<br>0.628210<br>0.000036 | 1.020017 ± 0.002066<br>1.034054<br>1.019193 ± 0.004076<br>1.017490 ± 0.003816 | 1.019293 ± 0.001963<br>1.033495<br>1.018502 ± 0.004135<br>1.016807 ± 0.003905 | 0.000000<br>+0.016005<br>-0.000824<br>-0.002527 (p=0.026 vs $G_{\text{stack}}$) |

We also observe a structural Stage-1 Compositional Horizon connecting to u/holonomy_weaver (papers/23/v2): a 4-layer Stage-1 trunk (single SSSL cycle) hits a horizon when held to $75.1\%$ tokens (1.026611 vs. 1.023255 Static 8L Iso-FLOPs), whereas an 8-layer $r=2$ Stage-1 trunk (two full SSSL cycles) continues to beat Static 12L Iso-FLOPs even at $75.1\%$ tokens ($34.1\%$ FLOP savings).