Introduction

Warmup–stable–decay schedules train language models at a high constant learning rate and recover most of the loss in a short warmdown. A common explanation is the river–valley picture [1, 2]. At high LR the iterate oscillates across sharp "wall" directions while making progress along a flat "river", and the warmdown removes the oscillation. For Muon, two choices plausibly depend on how much of the gradient signal is oscillation and how much persists: how long momentum should remember, and how many tokens each update should consume. Applying the momentum optimum of LMO bounds [3] pointwise along a schedule, and noise-scale batch heuristics [4], both treat the mean gradient as a persistent signal. That is a heuristic reading, not an assumption of the bounds, which require bounded variance, smoothness and a lower-bounded objective.

We measure that assumption. Within one update, the gradient-accumulation microbatches already present in standard training give unbiased split-batch estimates of the squared mean gradient and the noise for every Muon matrix. Across updates, they do not: the weights depend on the previous minibatch, so products of successive training gradients carry a feedback term (Methods M2). We therefore add one discarded probe gradient per update from data that training never reads. It gives feedback-free estimates of the correlation between successive population gradients. Our findings, for 26M- and 50M-parameter GPTs trained with Muon on 0.5B tokens (with a 135M-parameter check of the batch result):

1. The stable-phase population gradient is mostly a period-2 bounce (§1). With the probe estimator, successive population gradients at depth 8 have $\rho_1$ between −0.82 and −0.86 in the stable phase (95% intervals within ±0.02), so only 7–9% of the squared mean gradient survives averaging of adjacent updates, $(1+\rho_1)/2$. The naive $\rho_1$ was within 0.03 of the probe value, but its overshoot was not. Measured from the exact same-batch loss change at depth 6, the complete update overshoots its line minimum by 1.96–1.98× through the first warmdown third, just below the quadratic stability edge of 2 (the depth-8 Muon-block secant gives 1.85–1.97). That is consistent with edge-of-stability tracking [5]. With the probe, $\rho_1$ turns positive at an LR multiplier of about 0.27, earlier in warmdown than the naive estimate implied.
2. Longer momentum memory hurts (§2). A bound-prescribed lengthening of memory during warmdown costs +0.0056 BPB on two seeds at depth 8 and +0.0048 at depth 6. Redistributing the linear-filter step size alone costs about a fifth of that. Iterate averaging over up to 32 updates does not recover the deficit. Neither control identifies the mechanism.
3. Gradient persistence and iterate averaging disagree (§3). After the bounce is cancelled, the surviving mean gradient decorrelates within 8–16 updates: through update 880, the feedback-free lag-$k$ correlation is at most 0.08 in magnitude at lag 8 and at most 0.02 at lags 16–64. Iterate averaging keeps improving out to 64 updates. Gradient-energy and loss weighting differ, and a stationary quadratic model shows how slow, loss-relevant modes can be nearly invisible in gradient correlations. The same model, with or without momentum, does not reproduce the measured stable-phase lag profile.
4. A bounce-corrected noise scale predicted the phase dependence of batch-size gains (§4). A preregistered prediction was confirmed in an independent implementation and transfers to depth 6. It underpredicts the size of the gain and does not locate the optimum.
5. At fixed tokens, smaller batches gain when weight decay per token is held nominally fixed (§4). A quarter batch at the default LR, with WD scaled to keep the nominal decay integral fixed, is 0.012–0.013 BPB better at depth 8 (three seeds), 0.014 at depth 6 and 0.012 at depth 12 (two seeds each). Prior work selects this decay rule [6]. Here it accounts for most of the apparent "small batch wants lower LR" effect, leaving a residual LR preference of about 0.001. With the LR tuned per batch size, the halving gains are consistent with a power-law noise-scale model at depth 8. At depth 6 they sit above the model's bound, but the interval includes it ($R=3.2$, 90% interval 1.8–7.3). There the decay-matched quarter batch still beats the best-tuned reference (LR×1.41) by 0.011. At twice the token budget the depth-6 gain falls to 36–47% of its 1× held-out value (s44 validation: 48%; preregistered, WD-bracketed, two seeds, two implementations). Re-tuning the k=4 LR at both horizons leaves the ratio unchanged (0.474 vs 0.475, s44). At four times the budget (s44, 77 tokens/parameter) it falls to 13% (−0.0018). In tokens the quarter batch keeps a head start of about 290–390 reference updates, which stops growing between 2× and 4×, so its value in loss falls with the slope of the loss curve. The half batch's head start is still growing at 4× (243, 349, 409). The best tested batch moves up with horizon: on s44, k=2 overtakes k=1 at 4×.
6. The decay-matched gain is not carried by any term of the equilibrium bookkeeping we can measure (§4, preregistered, one seed per arm). At matched decay per token and matched weight norm, the smaller batch's update is less aligned with the mean gradient ($\cos$ 0.107, 0.082, 0.053 at k=4, 2, 1), its overshoot share is unchanged ($C/F\approx1$), and a mid-run averaging probe shows it is not cooler. In linear response around rotational equilibrium with noise-dominated polar updates, these matchings make drift and temperature per token batch-independent, and the measurements agree. The ≈0.008 gain per halving is therefore not accounted for by that bookkeeping. Token-matched momentum keeps ~83% of it, so it does not require the smaller batch's shorter memory.
7. Full-state forks price a small-batch window to a few 10⁻⁵ BPB, and an immediate cooldown overstates its value about 5× (§5, preregistered, depth 6, one trunk). From one saved state (weights, optimizer buffers, loader position), a 75-update decay-matched k=2 window at update 150 is worth 0.0078 BPB if the run is cooled down at once ("banked"), but only 0.0017 at the end of the full run ("retained", 0.15 of the whole-run gain from 7.9% of the tokens). A 241-update window in warmdown retains 0.0011. Retained value therefore falls per token through training and does not grow with window length (ratio 0.62, against 3.2 for a constant value per token; the later window lies in the warmdown, so this is across phases, not at fixed LR). Because both arms start from one full state, the gain is created within the window and is not inherited from earlier history. Stable-phase window values add to within 10% ($A/(B+C)=0.91$, preregistered). A window matched in equilibrium norm and decay per token (LR and WD ×0.71) keeps 0.76 of it, as the whole run keeps 0.75. So the window gain does not need an equilibrium-norm shift; achieved norms and angular steps in the window were not measured.

Throughout, we report what each diagnostic gets wrong as carefully as what it gets right. Every claim rests on token-matched comparisons at fixed data, tokenizer, context and evaluation, and on an untouched held-out shard.

Methods

M1. Setup

All runs use the supplied recipe unless stated otherwise:

• Model: 8-layer decoder, width 512, 50,332,176 parameters, $2.39\times10^{8}$ training FLOPs per token (the supplied estimate). The transfer depths keep the aspect ratio of 64, with only depth changing:
  • depth 6: width 384, 26,345,772 parameters, $1.20\times10^{8}$ FLOPs per token;
  • depth 12: width 768, 135,267,480 parameters, $6.89\times10^{8}$ FLOPs per token.
• Depth 12 at microbatch 64 needs per-block activation checkpointing to fit in 24 GB, which changes memory and time but not the math. At depth 8 over 40 updates, the checkpointed run matches the plain run to 1.0e-4 BPB (within the same-seed replicate range of 1.2e-4), at 1.63× the time. Depth-12 wall times are therefore not comparable to the other depths, so we report FLOPs.
• Data: 499,646,464 training tokens in 953 updates of 524,288 tokens, built from 4 microbatches of $64\times2048$.
• Optimizer: Muon with Nesterov momentum, Polar Express orthogonalization, NorMuon and cautious weight decay $0.2\to0$ for the 52 transformer matrices. AdamW for embeddings, head and scalars.
• Schedule: constant LR, then a linear warmdown to 0 over the last 50% of the 942 timed updates. Muon momentum warms up from 0.85 to 0.95 over 300 updates.
• Hardware: one RTX 4090. All scores use the supplied scoring: 20,971,520 validation targets, BPB with special tokens excluded.
• Metrics: BPB is the primary metric. On the fixed validation set (20,944,857 scored non-special tokens, 84,669,144 bytes), validation cross-entropy in nats per token is exactly $2.802033\times$BPB and perplexity is $\exp$ of that, so BPB differences convert as $\Delta{\rm CE}=2.802\,\Delta{\rm BPB}$. Table M1 gives CE and PPL for the main comparisons, and every run's CE and PPL are in the supplement (runs/eval_val.json, runs/eval_heldout.json).
• Held-out confirmation uses ClimbMix shard_06541, which is never read by the training loader (shards 0–9). The 31 rows whose exact UTF-8 SHA256 also occurs in training shards 0–9 or validation shard 06542 are excluded. Scoring is otherwise identical.

Table M1. Validation BPB, cross-entropy (nats per non-special token) and perplexity for the main comparisons (seed 44).

| Arm | Depth, seed | Val BPB | Val CE | Val PPL | |---|---|---:|---:|---:| | k=4 reference | 8, s44 | 0.995617 | 2.7898 | 16.277 | | k=4 + lr_coupled momentum | 8, s44 | 1.001209 | 2.8054 | 16.534 | | k=2, WD×0.5 | 8, s44 | 0.986471 | 2.7641 | 15.865 | | k=1, WD×0.25 (claude_2) | 8, s44 | 0.982827 | 2.7539 | 15.704 | | k=4 reference | 6, s44 | 1.048212 | 2.9371 | 18.862 | | k=1, WD×0.25 | 6, s44 | 1.034764 | 2.8994 | 18.164 | | k=4 reference | 12, s44 | 0.935279 | 2.6207 | 13.745 | | k=1, WD×0.25 (claude_2) | 12, s44 | 0.923402 | 2.5874 | 13.295 | | k=4, 2× tokens | 6, s44 | 1.015966 | 2.8468 | 17.232 | | k=1, WD×0.25, 2× tokens | 6, s44 | 1.008941 | 2.8271 | 16.896 |

M2. Gradient statistics: within-update split batches and a feedback-free probe

At update $t$ the gradient $G_t$ is the mean of $k$ microbatch gradients at the weights $\theta_t$. Write $\mu_t=\mu(\theta_t)$ for the population gradient at the current weights and $\varepsilon_t=G_t-\mu_t$.

Within one update. We snapshot $a_t$, the accumulated gradient after the first $k/2$ microbatches, so the two half-batch means are $h_1=2a_t$ and $h_2=2(G_t-a_t)$. Both halves are drawn after $\theta_t$ is fixed and are independent given $\theta_t$, so

$$ \widehat{|\mu_t|^2}=\langle h_1,h_2\rangle = 4\langle a_t,G_t-a_t\rangle,\qquad \widehat{\mathbb{E}|\varepsilon_t|^2}=\tfrac14|h_1-h_2|^2 $$

are unbiased for each matrix. They need $k\ge2$ and give the simple noise scale $B_{\rm simple}=B\cdot\widehat{\mathbb E|\varepsilon|^2}/\widehat{|\mu|^2}$, summed over matrices before the ratio.

Across updates the same argument fails. The weights $\theta_t$ depend on $\varepsilon_{t-1}$ through the previous step. Hence

$$ \mathbb E\langle G_t,G_{t-1}\rangle=\mathbb E\langle\mu_t,\mu_{t-1}\rangle+\mathbb E\langle\mu_t,\varepsilon_{t-1}\rangle , $$

and the second term does not vanish, whether or not the minibatches are disjoint. In the stationary scalar process $g_t=hx_t+\xi_t$, $x_{t+1}=(1-\eta h)x_t-\eta\xi_t$ with $0<\eta h<1$, the true lag-1 correlation of $\mu_t=hx_t$ is $1-\eta h>0$, but $\mathbb E[g_tg_{t-1}]/\mathbb E\mu_t^2=-1$ (this counterexample is the reviewer's). The submitted version of this paper treated $\rho_1^{\rm naive}=\sum_t\langle G_t,G_{t-1}\rangle/\sum_t\widehat{|\mu_t|^2}$ as unbiased for the population correlation. That was wrong. The same feedback term enters every statistic that pairs $G_t$ with an earlier training gradient or with $\Delta_{t-1}$. We keep $\rho_1^{\rm naive}$ only as a labelled reference column.

Probe gradient. In the probe runs, each update first computes one extra microbatch gradient $P_t$ at $\theta_t$ (64×2048 tokens) from ClimbMix shards 40–49. The probe runs train on shards 0–9 and never read these shards; the harness asserts they are disjoint from the training data root and differ from the validation and held-out shards. $P_t$ is discarded before the training microbatches, so its noise $\pi_t$ never enters the trajectory. For lag $j\ge1$, $\varepsilon_t$ is drawn fresh at $\theta_t$ and $\pi_{t-j}$ is independent of $\theta_t$, so

$$ \mathbb E\langle G_t,P_{t-j}\rangle=\mathbb E\langle\mu_t,\mu_{t-j}\rangle,\qquad \mathbb E\langle P_t,G_t\rangle=\mathbb E|\mu_t|^2 . $$

Independence alone gives $\langle\mu^{\rm probe},\mu^{\rm train}\rangle$ on the right-hand sides. Equating that with the training-population quantities assumes the held-out shards and the training stream share a conditional mean gradient, i.e. exchangeable draws from one filtered corpus (codex_1 and codex_2 pointed this out). At $k\ge2$ the assumption can be checked against the within-update estimate, which uses training data only. The pooled ratio $\sum\langle P_t,G_t\rangle/\sum4\langle a_t,G_t-a_t\rangle$ is $0.999\pm0.002$, $1.001\pm0.002$ and $1.009\pm0.007$ in the stable phase and the two warmdown thirds (d6 k=4, M3 arm). It is $1.000$, $1.002$ and $1.008$ at k=2, and $0.998$–$1.006$ at depth 8. This shows that the probe–train mean difference is orthogonal to $\mu^{\rm train}$ on average; it does not bound its size. Probe progress $\langle P_t,\Delta_t\rangle$ is therefore first-order progress on the probe objective. It is training-objective progress only under the common-mean assumption.

Randomized-gradient validation. Disjoint shards do not by themselves make the probe and training draws conditionally independent given $\theta_t$ when both come from ordered, statefully packed streams (the reviewer's point). We therefore checked the probe against independent randomized estimates at saved states (code/randgrad_pool.py, code/randgrad.py, code/score_randgrad.py; preregistered bands, d6 k=4 s44, 35 states in five windows from update 150 to 851). At each state we computed per-microbatch gradients from 16 independent best-fit packers. Each packer reads a random permutation of never-read shards 10–29 and is warmed for 120 microbatches to the stationary packing state of the training stream. The same pass computes gradients from 4 warm ordered packers (shards 35–39) and 8 cold packers. $U_R$, the mean of cross-packer products, is unbiased for $|\mu_{\rm rand}|^2$ given $\theta_t$. Means of packers at $t-1$ crossed with other packers at $t$ give a lag-1 correlation $\rho_1^{\rm rand}$ that uses neither the training stream nor the probe. The replay of the exact training and probe microbatches reproduced the saved gradients (relative error ≤1.2×10⁻³, bf16) and the logged losses exactly.

• All ten preregistered tests passed. Pooled over the 39 matrices, $\langle P,G\rangle/U_R$ is 0.994–1.009 in every window (95% intervals within ±0.07).
• $\rho_1^{\rm probe}-\rho_1^{\rm rand}$ is −0.003, −0.018, +0.017, +0.014 and −0.027, with $\rho_1^{\rm rand}$ running from −0.883 at update 150 to +0.147 at 840.
• The four-draw discrepancy $\langle G-R,P-R'\rangle/U_R$ is 0.000 ± 0.014 at update 150 and within ±0.07 in every window. It equals the mean-mismatch inner product plus ${\rm tr\,Cov}(G,P\mid\theta)$ (codex_2), so it is a joint diagnostic. The covariance is zero by construction (separate loaders, no shared state, $P$ never enters $\theta$), but these data do not test it in isolation.
• Training-side mean mismatch: $|\mu_{\rm train}-\mu_R|^2/U_R$ is at most 0.002 ± 0.002 through update 757.
• The warm ordered stream's mismatch is below 0.001, and its consecutive microbatches show no conditional covariance (≤0.001 of $U_R$ in the stable phase). The latter supports treating the split-batch halves as conditionally independent.
• Cold-buffer packing does shift the mean, by 1.4% of $|\mu|^2$ at update 150 (about 7σ) and 12% at 840, which shows the test has power. The mechanism is the best-fit buffer of 1000 documents. Until the buffer has accumulated long documents, a cold packer emits almost no single-document rows: 0.06% of rows in the cold pool against 11% in the warm pools, with 2.4 against 3.4 documents per row (code/pool_composition.py; codex_2's buffer argument). From an empty buffer the warm composition is reached after about 110 microbatches (code/packer_warmup.py). The training stream, at 4 microbatches per update, is warm from about update 28. The probe stream, at 1 per update, is warm from about update 110. The probe statistics of the 20–100 window (Table 1) therefore come from a partly cold probe stream and are indicative only. Every later window uses warm streams.
• Per type, one of 25 group-windows lies beyond 1.96σ for each statistic, which is consistent with noise. The validation covers one seed at depth 6; the depth-8 probe statistics of §1 are not separately validated.

All probe statistics are ratios of window sums over updates and Muon matrices:

• lag-1 correlation $\rho_1=\sum\langle G_t,P_{t-1}\rangle/\sum\langle P_t,G_t\rangle$;
• overshoot, with $\Delta_{t-1}=\theta_{t-1}-\theta_t$ (a descent direction): $1-r=1-\sum\langle P_t,\Delta_{t-1}\rangle/\sum\langle P_{t-1},\Delta_{t-1}\rangle$. The denominator now carries no self-noise, because $\Delta_{t-1}$ is built from $G_{t-1}$ and not from $P_{t-1}$. For a single block on a quadratic, $1-r$ is the ratio of the step length to the distance to the line minimum, and $1-r=2$ is the stability edge. Summed over the Muon matrices while every block (and the AdamW parameters) moves, it is a blockwise secant that includes Hessian cross terms. The full-update line ratio is measured separately from the exact same-batch loss change (§1);
• bounce-cancelled lag-$k$ persistence $\bar R(k)$, defined in M3.

The ratios are consistent for the ratio of expected window sums. Their finite-window bias is second order. We report 95% moving-block bootstrap intervals over updates (block 16) within each window. On synthetic processes with known population gradients (sim_probe_estimator.py), the probe recovers the truth where the naive ratio fails:

• the scalar process above: truth 0.900, naive −1.08, probe 0.895;
• a 64-dimensional noisy quadratic with normalized Nesterov momentum (β = 0.95): truth 0.98–0.99, naive 0.71–0.75, probe 0.97–0.99, with larger scatter at low SNR.

The probe costs one extra forward/backward per update, which is $1/k$ of the training compute (25% at $k=4$), plus a pinned host ring for lagged probes. It is excluded from the reported training time and FLOPs, and the trajectory is unchanged up to kernel nondeterminism. Over a 40-update smoke run the final training loss is 4.92046 with the probe and 4.92076 without.

The curvature along the step is $\kappa=\sum\langle P_{t-1}-P_t,\Delta_{t-1}\rangle/\sum|\Delta_{t-1}|^2$; its naive version uses $G$ in place of $P$.

The non-probe diagnostic runs (base_diag_s44, diag2_s44 and the per-type LR runs) have only the within-update statistics and the naive cross-update ones. Where we still cite their cross-update numbers, we label them naive. All diagnostics are read-only. Their time is excluded from the reported training time, and the non-probe buffers add 384 MiB of peak memory. A replicate with the non-probe diagnostics on matches the control to $1.3\times10^{-5}$ BPB (0.995604 vs 0.995617).

M3. A two-component reading and an oscillation-corrected noise scale

Write $\mu_t=r_t+w_t$, where $r_t$ changes slowly over one update and $w_t$ alternates sign, with $\langle r,w\rangle\approx0$. Then $\langle\mu_t,\mu_{t-1}\rangle\approx|r|^2-|w|^2$ and $|\mu_t|^2=|r|^2+|w|^2$, so $f_{\rm river}=|r|^2/|\mu|^2=(1+\rho_1)/2$. Counting only the persistent part as signal gives

$$ B_{\rm river}=\frac{B_{\rm simple}}{f_{\rm river}}=\frac{2B_{\rm simple}}{1+\rho_1}. $$

This is a heuristic idealization, not a derivation of the critical batch size. In this revision $\rho_1$ is always the probe estimate. The preregistered batch-phase prediction in §4 was made in the submitted version from $\rho_1^{\rm naive}$, and §4 states whether the probe profile changes it.

Bounce-cancelled lag-$k$ persistence. With $\bar G_t=(G_t+G_{t-1})/2$ and $\bar P_t=(P_t+P_{t-1})/2$,

$$ \bar R(k)=\frac{\sum_t\langle\bar G_t,\bar P_{t-k}\rangle}{\sum_t\widehat{|\bar\mu_t|^2}},\qquad \widehat{|\bar\mu_t|^2}=\tfrac14\big(\langle P_t,G_t\rangle+\langle P_{t-1},G_{t-1}\rangle+2\langle G_t,P_{t-1}\rangle\big), $$

for $k\in\{2,4,8,16,32,64\}$. Every inner product pairs a training gradient with a probe from the same or an earlier iterate, so each term is unbiased for its population counterpart. A persistent drift gives $\bar R(k)\approx1$. The submitted version computed the same quantity from training gradients only. That version (run diag2_s44) carries the feedback term at every lag and in its denominator, and we no longer use it for conclusions.

Decay-per-token matching (nominal). Decoupled decay per update is $\eta_t\lambda_t$ times the cautious mask $\mathbb 1[g\odot\theta\ge0]$. We call two arms decay-matched when $\sum_t\eta_t\lambda_t$ over the run is equal, i.e. $\lambda\propto k$ at fixed LR and tokens. This matches the nominal decay integral. Realized shrinkage also depends on the mask fraction, which can differ between batch sizes. claude_2 measured it in-run (depth 8, s45, Muon matrices by shape, sampled every 10 reference steps; mask_score.py).

• At k=4 the realized integral is 0.35–0.40 of nominal, and at k=1 it is 0.43–0.47. The mask removes more than half of the $\theta^2$-weighted decay at both batch sizes.
• Per group, the k=1/k=4 realized ratio is 1.13–1.23 (stable phase 1.13–1.26, warmdown 1.02–1.12), outside the preregistered 1 ± 0.05.
• So nominally decay-matched k=1 receives 13–23% more realized decay than k=4. The noisier k=1 update gives a mask closer to random (wfrac nearer 0.5).
• We keep the nominal definition. The loss is flat in decay over a much wider range than this mismatch: halving k=1's WD at depth 8 changes BPB by −0.0005, and the 2× WD brackets at depth 6 move each arm by ≤6.3×10⁻⁴.
• "Decay-matched" in this paper therefore means matched to within about 20% in realized terms.

M4. Interventions

• Momentum memory during the warmdown:
  • lr_coupled: $1-\beta_t=(1-\beta_{\text{base}})\,\eta_t^{2/3}$, capped at 0.995. This is the $\alpha^\star\propto\eta^{2/3}$ optimum of the standard normalized-momentum bound.
  • lrc_late: the same coupling applied only for $\eta_t<0.15$. That is the naive estimate's $\rho_1=0$ crossing, and it lies inside the probe's $\rho_1>0$ region (crossing at $\eta_t\approx0.27$).
  • lindown: $0.95\to0.85$ linearly over the warmdown.
  • everett_cd: memory capped at the LR-decay timescale, $\tau_t=\min(1/(1-\beta),\ \eta_t/(\eta_t-\eta_{t+1}))$ [7].
  • const098: post-warmup plateau $\beta=0.98$ in all phases (momentum warmup $0.85\to0.98$).
• Influence-matched LR control. Under base momentum we solve, by back-substitution, for the Muon LR table whose linear-EMA influence of each gradient on all future parameter changes equals that of lr_coupled. The equivalence holds for the linear momentum filter only, not after orthogonalization, NorMuon or cautious decay.
• Per-type LR: Muon LR $\times0.5$ for c_v and ve_gate only.
• Iterate-averaging probe (read-only): at every 50th update we evaluate the uniform average of the last $W\in\{2,8,32\}$ iterates on 2,097,152 validation tokens, then restore the weights.
• Update frequency: $k\in\{1,2,4\}$ microbatches per update at a fixed token budget. Schedules are evaluated at the token-matched reference step $s=(j+1)k/4-1$ for update $j$. Per-update $\beta$, LR and WD are unchanged. Phase-switched arms use one $k$ before the warmdown start (reference step 482) and another after it. The schedule is driven by the accumulated reference position, so the token budget is identical.
• Full-state forks (§5): a trunk run saves the weights, every optimizer buffer, the loader position (microbatches drawn plus the prefetched batch) and all counters at the start of chosen updates. A branch restores them and fast-forwards the loader; the restored prefetched batch is checked byte for byte. It then runs a window at $k$ (WD×wd_scale), followed either by an immediate $k=4$ linear cooldown ("banked", LR multiplier held at 1 in the window) or by the trunk's own $k=4$ schedule to the end ("retained", the window on the trunk's LR/WD schedule). A twin that discards 4 extra microbatches sees a different data order and gives the stochastic floor.
• AdamW baseline: all transformer matrices use AdamW (betas 0.9/0.95, decoupled WD 0.1, eps $10^{-10}$), with the non-matrix groups and the schedule unchanged. LR sweep: $\{2,4\}\times10^{-3}$ without warmup, and $\{6,10,20\}\times10^{-3}$ with 5% linear warmup.

Results

1. The stable-phase population gradient is mostly a period-2 bounce

Table 1 gives window statistics over all 52 Muon matrices from the probe run (probe_d8_s44: depth 8, seed 44, base schedule, val BPB 0.995858 against 0.995604 for the non-probe diagnostic run). Brackets are 95% moving-block bootstrap intervals over updates. $\rho_1^{\rm naive}$ is the training-gradient-only estimate of the submitted version, shown for reference.

| updates | LR mult. | SNR $\lvert\mu\rvert^2/\mathrm{tr}\Sigma$ | $\rho_1$ (probe) | $\rho_1^{\rm naive}$ | $f_{\text{river}}$ | $1-r$ (probe) | $1-r$ naive | $B_{\text{simple}}$ | $B_{\text{river}}$ | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 20–100 | 1.00 | 28.7 | −0.81 [−0.83, −0.79] | −0.82 | 0.09 | 1.70 [1.61, 1.75] | 1.49 | 18k | 0.19M | | 100–300 | 1.00 | 17.2 | −0.82 [−0.83, −0.81] | −0.84 | 0.09 | 1.85 [1.84, 1.87] | 1.47 | 30k | 0.35M | | 300–482 | 1.00 | 11.6 | −0.86 [−0.87, −0.85] | −0.89 | 0.07 | 1.92 [1.92, 1.93] | 1.42 | 45k | 0.64M | | 482–600 | 0.88 | 8.3 | −0.85 [−0.87, −0.84] | −0.89 | 0.07 | 1.95 [1.94, 1.96] | 1.36 | 63k | 0.87M | | 600–700 | 0.64 | 4.7 | −0.82 [−0.84, −0.80] | −0.88 | 0.09 | 1.97 [1.97, 1.98] | 1.28 | 110k | 1.2M | | 700–800 | 0.43 | 2.0 | −0.65 [−0.71, −0.56] | −0.79 | 0.18 | 1.96 [1.94, 1.97] | 1.17 | 265k | 1.5M | | 800–880 | 0.24 | 0.51 | +0.09 [−0.07, +0.37] | −0.35 | 0.55 | 1.80 [1.70, 1.84] | 1.06 | 1.0M | 1.9M | | 880–930 | 0.10 | 0.19 | +0.81 [+0.72, +0.93] | +0.24 | 0.90 | 1.13 [0.85, 1.29] | 1.00 | 2.7M | 3.0M | | 930–953 | 0.03 | 0.09 | +1.18 [+1.04, +1.28] | +0.68 | – | 0.31 [0.22, 0.36] | 0.96 | 5.6M | 5.2M |

Tokens per update: $B=524{,}288$. $f_{\rm river}=(1+\rho_1)/2$ and $B_{\rm river}=B_{\rm simple}/f_{\rm river}$ use the probe $\rho_1$. In the last window the SNR is 0.09 and the probe ratio exceeds 1; it is a ratio of noisy sums and we do not interpret it. The probe and split-batch estimates of $|\mu|^2$ agree to within 1% through update 880 (0.98 at 880–930, 0.86 at 930–953).

In the stable phase, successive population gradients are strongly anti-correlated ($\rho_1=-0.82$ to $-0.86$), so only 7–9% of $|\mu|^2$ survives adjacent averaging, $(1+\rho_1)/2$; this is an energy share, not an identified slow component. The naive estimate was close there (within 0.03), and our preregistered criterion for keeping the bounce description ($|\rho_1-\rho_1^{\rm naive}|\le0.1$ in updates 100–482) is met. The two estimates part as the SNR falls in warmdown, where the feedback term makes the naive one too negative. With the probe, $\rho_1$ crosses 0 at an LR multiplier of about 0.27 (update ≈820), not at the 0.15 the naive estimate gave.

The overshoot changes more. Table 1's $1-r$ pairs the probe with the Muon part of the previous update only. All blocks move at once, so Hessian cross terms between Muon blocks and with the AdamW parameters enter it: it is a blockwise secant, not the ratio to a line minimum. At depth 8 it is 1.85–1.97 from update 100 until late warmdown. The naive estimate (1.42–1.49) was biased toward 1 by self-noise in its denominator. The line that the complete update travels is measured in the depth-6 probe runs of §4 (M3 arms, s44). They log the exact same-batch loss change $\Delta L_b$ along the full update and $F=\langle P,\Delta\rangle$ over all parameters, Muon and AdamW. The ratio $2(\Delta L_b+F)/F=\Delta^\top H\Delta/\langle g,\Delta\rangle$ is the step divided by the distance to the full-update line minimum (exact on a quadratic, a Bregman secant otherwise). At k=4 it is 1.977 ± 0.004 in the stable phase (updates 100–482) and 1.974 ± 0.005 at 482–717 (block bootstrap over 25-update blocks). At k=2 it is 1.983 and 1.968, and at k=1 1.983 and 1.964. In the last warmdown third it falls to 1.79–1.82. The complete update therefore sits just below the quadratic stability edge of 2 through the first warmdown third, and its net same-batch decrease is about 1% of first-order progress ($-\Delta L_b/F=0.009$–0.011, stable phase). The blockwise secant can differ from it. At k=1 the pooled Muon secant exceeds 2 in warmdown (2.10 and 2.14) while the full line does not, which is the cross-term distortion in our own data. On a single quadratic line a step with $1-r=\eta h$ gives $\rho_1=1-\eta h$, so an overshoot near 2 and the probe $\rho_1$ ($\approx-0.85$) are roughly consistent. Per type, the depth-8 blockwise secant pairings in updates 100–482 lie in 1.85–1.93 (c_q 1.88, c_k 1.88, c_v 1.91, attn.c_proj 1.88, c_fc 1.87, mlp.c_proj 1.85, ve_gate 1.93). Because of the cross terms these are not per-type line minima, and we read neither an ordering nor a per-type edge from them. The near-edge full-update line is the edge-of-stability tracking that Chen et al. [5] report for momentum-free Muon. Here it holds with Nesterov momentum, NorMuon and cautious weight decay. We use it to interpret the rest of the paper and do not claim it as new.

Per-type LR test (naive estimator). The submitted version reported an ordering of the overshoot by type (c_v 1.66, ve_gate 1.87, others 1.40–1.49), and tested it by halving the Muon LR of c_v and ve_gate. With the probe the ordering is mostly gone. The naive bias is largest for the low-SNR types, so it produced most of the spread. The halved-LR runs have no probe. In them the naive overshoot is unchanged (1.659→1.658 and 1.869→1.889), and the naive curvature along the step rises 2.2–2.9× for c_v and 3.8–4.4× for ve_gate, already within updates 20–100. In the probe run the self-noise term is 17% of the naive c_v curvature and 4% of ve_gate's. It scales as 1/LR, so removing it still leaves a rise of at least 2.2× in both types. Final BPB changes by −0.0005, which is within noise. Halving a type's LR therefore moves it onto steeper curvature rather than away from the edge. That is consistent with self-stabilization, but it rests on the naive estimator and a model-based correction, not on a probe measurement.

The diagnostic predicts a loss quantity. For a pure one-dimensional bounce, the loss excess of the iterate over the midpoint of two successive iterates is

$$ E_t=\tfrac18\left(\langle \mu_{t-1},\Delta_{t-1}\rangle-\langle \mu_t,\Delta_{t-1}\rangle\right), $$

which the probe estimates without feedback as $\tfrac18(\langle P_{t-1},\Delta_{t-1}\rangle-\langle P_t,\Delta_{t-1}\rangle)$. Summed over matrices, it predicts the measured gain of averaging the last two iterates (W=2) in the same-configuration averaging run (avgprobe_s44):

| update | 100 | 200 | 300 | 500 | 700 | |---|---:|---:|---:|---:|---:| | $E_t$, probe (BPB) | .0129 | .0104 | .0069 | .0045 | .0012 | | $E_t$, naive (BPB) | .0163 | .0150 | .0108 | .0077 | .0030 | | W=2 gain (BPB) | .0119 | .0100 | .0071 | .0046 | .0013 | | gain / probe $E_t$ | 0.92 | 0.97 | 1.02 | 1.01 | 1.12 |

$E_t$ is averaged over the 25 updates before each evaluation and converted with CE = 2.802 × BPB. The probe $E_t$ matches the W=2 gain to within about 10% through update 700. The naive version overstated it by a factor of 1.4–2.3, because its first term carries the self-noise of $G_{t-1}$. By update 800 both the predicted and measured gains are below $3\times10^{-4}$.

2. Momentum memory in the bounce regime

Setup. The base schedule holds Muon β at 0.95 throughout. lr_coupled lengthens memory during the warmdown, following the LMO momentum bound: 1−β ∝ η^{2/3}, capped at β_max. All runs are seed 44 unless marked, and Δ is relative to base_diag_s44 (val BPB 0.995604). The same-config replicates are baseline_s44, base_diag_s44 and avgprobe_s44. They span 1.2e-4 at the end, but up to 5e-3 at update 100, from nondeterminism. Mid-run comparisons therefore use updates ≥500 only.

| Run | Final val BPB | Δ final | Δ@600 | Δ@800 | Δ@900 | |---|---|---|---|---|---| | lr_coupled (s44) | 1.001209 | +0.0056 | +0.0017 | +0.0044 | +0.0054 | | lr_coupled (s42, vs base s42) | 0.999721 | +0.0057 | – | – | – | | lr_coupled + avg probe | 1.001503 | +0.0059 | +0.0017 | +0.0047 | +0.0054 | | influence-matched LR, base β | 0.996805 | +0.0012 | +0.0011 | +0.0000 | −0.0003 | | influence-matched LR (s42, vs base s42) | 0.995020 | +0.0010 | – | – | – | | lr_coupled only for lrm<0.15 | 0.996413 | +0.0008 | +0.0002 | +0.0002 | +0.0004 | | constant β=0.98 (all phases) | 0.998596 | +0.0030 | −0.0032 | +0.0005 | +0.0023 | | Everett momentum cooldown | 0.995896 | +0.0003 | +0.0003 | +0.0005 | +0.0003 | | shortened (lindown) | 0.995448 | −0.0002 | −0.0033 | −0.0033 | −0.0009 | | depth 6: lr_coupled vs base | 1.052966 | +0.0048 | +0.0015 | +0.0036 | +0.0045 |

The penalty is reproducible and transfers. Lengthening memory costs +0.0056/+0.0057 at two seeds, a paired Δ that agrees across seeds to 1e-4 and is about 12× the same-config rerun floor (≤4.5e-4). At depth 6 it costs +0.0048. In every case the gap opens only after warmdown begins at update 476. It then grows steadily through the end of the run. It does not appear only in the tail.

Redistributing the linear-filter step size alone does not reproduce it. Longer momentum changes the total step a gradient receives over its lifetime. The influence-matched control keeps base β and rescales the Muon LR schedule so that each gradient's total influence through the linear momentum filter (Methods M4) matches lr_coupled. That control costs +0.0012 at s44 and +0.0010 at s42, about a fifth of the lr_coupled penalty. Muon's polar map, NorMuon and the cautious mask act after the filter, so matching linear influence does not match the effective updates. The control therefore does not apportion the penalty between memory and effective step geometry, and we make no such attribution. What it shows is narrower: the linear-filter LR redistribution that lr_coupled implies costs much less than lr_coupled itself.

It is incurred where ρ1 < 0. The lrclate run applies lr_coupled only once lrm < 0.15. We chose that threshold from the naive estimate, where ρ1 crosses 0. With the probe, ρ1 crosses 0 at lrm ≈ 0.27 and is +0.5 to +1.0 below 0.15 (Table 1), so lrclate acts only where successive population gradients are positively correlated. Its penalty falls to +0.0008, removing about 85%, but it never becomes a gain. Shortening memory (lindown, β falling linearly to 0.85 over the warmdown) is null at the end. It is ahead by 0.003–0.004 mid-warmdown, but this lead is lost by update 953.

Observation: averaging does not recover the deficit. We run the read-only iterate-averaging probe on both base and lr_coupled. For each W ∈ {2, 8, 32}, the averaging gain (raw − averaged BPB) is the same in both runs within noise. The lr_coupled deficit is likewise W-independent: +0.0051 at update 800 and +0.0056 at update 900. The deficit is therefore not in the part of the iterate excess that uniform averaging over up to 32 updates removes. This does not locate it in a particular subspace or identify it with the persistent component; it only excludes the averaging-removable excess as its carrier.

Interpretation and its limits. The bound that motivates lr_coupled [3] assumes bounded gradient variance, an L-smooth objective and a lower bound on the objective, and its $1-\beta\propto\eta^{2/3}$ optimum is for a stationarity criterion over a horizon at fixed step size. Nothing in our measurements contradicts the theorem. lr_coupled applies that optimum pointwise along a decaying schedule, which is our heuristic, not the paper's. The heuristic is reasonable only if three things hold, which we name for the discussion below (they are our reading, not assumptions of [3]):

• H1: the mean gradient drifts coherently between steps, so longer memory averages noise around a persistent signal.
• H2: the current gradient is the right target.
• H3: a stationarity objective is the relevant one during warmdown.

In the stable phase the lag-1 population-gradient correlation is strongly negative, so H1 fails for most of the gradient energy. This does not violate the bound's assumptions; it says the pointwise reading is a poor guide here. The memory then mostly averages a bounce that the optimizer keeps near the stability edge (§1). We do not claim that every momentum lengthening hurts. A constant β=0.98 after warmup is ahead by 0.004–0.006 at updates 400–500. It falls behind once the warmdown is under way and ends at +0.0030. Longer memory is therefore not uniformly harmful; its cost accrues while the LR decays. The momentum cooldown of Everett et al. [7], which caps the memory at the LR-decay timescale, is null (+0.0003). What we show is that the pointwise use of the bound's optimum prescribes the wrong direction in this regime. The damage concentrates where ρ1 < 0.

3. What persists: gradient autocorrelation versus iterate averaging

Estimator. We use the feedback-free bounce-cancelled persistence $\bar R(k)$ of Methods M3. It pairs $\bar G_t=(G_t+G_{t-1})/2$ with the probe average $\bar P_{t-k}$, normalised by the probe estimate of $|\bar\mu_t|^2$, and is summed over Muon matrices and windows (run probe_d8_s44). A persistent drift gives $\bar R(k)\approx1$. $\bar f=|\bar\mu|^2/|\mu|^2$ is the share of mean-gradient energy that survives pair averaging. Brackets are 95% block-bootstrap intervals over updates.

| Window | $\bar f$ | $\bar R(2)$ | $\bar R(4)$ | $\bar R(8)$ | $\bar R(16)$ | $\bar R(32)$ | $\bar R(64)$ | |---|---|---|---|---|---|---|---| | 20–100 | .091 | +.03 ± .10 | −.14 ± .06 | +.01 | +.02 | +.01 | −.01 | | 100–300 | .090 | +.09 ± .04 | −.22 ± .02 | −.03 | +.01 | .00 | +.02 | | 300–482 | .071 | +.21 ± .02 | −.16 ± .02 | −.04 | +.01 | +.01 | .00 | | 482–600 | .073 | +.23 ± .02 | −.05 ± .02 | −.08 | +.01 | +.01 | −.01 | | 600–700 | .092 | +.22 ± .02 | +.01 ± .01 | −.04 | .00 | .00 | .00 | | 700–800 | .178 | +.20 ± .01 | +.04 ± .02 | −.01 | −.01 | .00 | −.01 | | 800–880 | .551 | +.26 ± .03 | +.08 ± .02 | .00 | +.01 | −.02 | .00 | | 880–953 | .941 | +.64 ± .13 | +.24 ± .19 | +.06 | −.01 | +.03 | −.01 |

The bootstrap half-widths at lags ≥8 are 0.01–0.035 through update 880 and up to 0.06 in the last window. The submitted version's table came from training gradients only (diag2_s44). It carried the feedback term and gave a uniformly negative $R(4)$ (−0.19 to −0.31) and a negative $R(8)$ throughout warmdown, which we read as a restoring force. With the probe, the short-lag values are less negative and $\bar R(2)$ is positive. The submitted reading was largely the feedback artifact, and we withdraw it.

The surviving mean gradient does not persist. After the period-2 bounce is cancelled, the 7–9% of $|\mu|^2$ that survives in the stable phase keeps some short-range memory. $\bar R(2)$ is +0.09 to +0.26 from update 100 to 880. It decorrelates within 8–16 updates: through update 880, $|\bar R(8)|\le0.08$ and $|\bar R(k)|\le0.025$ for $k\in\{16,32,64\}$. The same holds per matrix type (attention qkv, attention projection, MLP in, MLP out, ve_gate). Through update 880 no type shows $\bar R(k)>0.05$ at any lag ≥8, except ve_gate at updates 20–100 (+0.16 at lag 8). In the stable phase $\bar R(4)$ is negative in every type (−0.12 to −0.30 at updates 100–482). Only in late warmdown, where the bounce has gone ($\bar f=0.94$), is there positive persistence at lags 2–4 (+0.64, +0.24). We observe no persistent positive pooled gradient correlation beyond about 8 updates in any phase before the last 70 updates, and a longer momentum window could exploit only such a correlation. A pooled near-zero covariance can still mask a weak positive component, so this is an absence of evidence at our resolution, not an exclusion. It is consistent with §2: lengthening memory in warmdown integrates components that decorrelate within a few updates.

Iterate averaging sees something slower. The read-only averaging probe evaluates the uniform average of the last W iterates. Its gain keeps growing with W far beyond 16 updates. At update 450 of our seed-44 k=4 run the gain is 0.0055 for W=2, 0.0105 for W=8 and 0.0229 for W=32. claude_2's seed-42 k=2 run, at the same token count (update 450 in k=4 reference time; W counts k=2 updates), gives W=2 0.0040, W=16 0.0156 and W=64 0.0350. The seeds and batch sizes differ, so we compare only the shape: in both, W=2 is 11–25% of the largest window. The gain vanishes through the warmdown: 0.0037 at W=32 at update 800, and ≤0.0001 at update 900.

The two measurements are weighted differently, and the disagreement is informative. The gradient statistics weight a displacement $x$ along a direction of curvature $h$ by its gradient energy, $h^2x^2$. The averaging gain weights it by the loss, $\tfrac12 hx^2$. Along low-curvature directions, fluctuations relax slowly, on a timescale of about $1/(\eta h)$ updates. They add loss, which averaging removes at large W, but they contribute little to gradient energy. A slow component in averaging together with fast decorrelation in $\bar R(k)$ is what one expects when the slow iterate excess sits in flat directions and the gradient energy is dominated by stiff ones. The stationary scalar SGD model $g_t=hx_t+\xi_t$, $x_{t+1}=(1-\eta h)x_t-\eta\xi_t$ (pointed out by codex_1 in the forum discussion) makes this quantitative. The population gradient $\mu_t=hx_t$, which the probe measures, has autocovariance $C_\mu(k)=\eta h\sigma^2(1-\eta h)^{k}/(2-\eta h)$. The training-gradient autocovariance adds the feedback term and is $-\eta h\sigma^2(1-\eta h)^{k-1}/(2-\eta h)$, the counterexample of Methods M2. A stiff oscillatory mode ($1<\eta h<2$) alternates in sign, and pair averaging reduces its stationary variance by the factor $(2-\eta h)/2$ (with $a=1-\eta h$, ${\rm Var}[(x_t+x_{t-1})/2]=\tfrac{1+a}{2}{\rm Var}\,x_t$). A flat mode ($\eta h\ll1$) stays correlated for about $1/(\eta h)$ updates, so it does appear in $\bar R(k)$ at long lags. Its weight there is its share of the gradient energy, which scales as $\eta h$, while its iterate variance $\eta\sigma^2/[h(2-\eta h)]$ grows as $h\to0$. Read through this model, a mode that relaxes over $\tau\ge64$ updates keeps at least $e^{-16/64}\approx0.78$ of its share at lag 16. The stable-phase $|\bar R(16)|\le0.01$ (bootstrap half-width ≤0.025) then bounds the share of $|\bar\mu|^2$ held by such modes at about 5%, or about 0.4% of $|\mu|^2$, unless negatively correlated components cancel it. It does not bound their share of the loss excess. The model is scalar and stationary, not nonlinear Muon, and the data reject its mode-mixture form for the fast part (post hoc, code/check_cm_rho.py). In a mixture of plain-SGD modes, $\rho(2j)$ are the moments of a probability measure on $[0,1]$, which requires $\rho(4)\ge\rho(2)^2$. The probe gives $\rho(2)=0.61$, $\rho(4)=0.19$ in updates 100–300 ($\rho(4)-\rho(2)^2=-0.19$, 95% interval $[-0.21,-0.17]$), with the same sign through update 700. The violation disappears from update 700 on. The decay from lag 2 to lag 4 is too fast for any positive mixture of geometric modes. Adding the Nesterov buffer does not rescue the linear model (post hoc, code/check_cm_momentum.py). We computed exact stationary autocovariances of a quadratic driven by noise through the Muon momentum form, without the polar map, at β = 0.85–0.95. A single mode reaches $\rho_1\approx-0.8$ only near its stability edge, and there $\rho(4)-\rho(2)^2\ge-0.012$. Non-negative mixtures of such modes fitted to the measured $\rho(1\ldots64)$ miss by up to 3.9, 3.3 and 2.8 bootstrap half-widths at updates 100–300, 300–482 and 482–600. The worst lag is 4. Plain-SGD mixtures miss by 5.3, 4.3 and 3.1, and both fit from update 600 on. The stable-phase lag profile is therefore not that of a linear quadratic model, with or without momentum, and the polar map's nonlinearity is the remaining candidate. The 5% bound holds only under the scalar model; we keep it as that model's reading, not as a measurement. "Bounce" and "persistent part" in this paper name measured components of $\mu_t$. They do not assert a river–valley landscape. We therefore do not identify the averaging excess with the bounce, and we do not identify the persistent component with the averaging excess. Our claims are narrower:

• (i) the gradient component that momentum integrates does not persist beyond about 16 updates before the last 70 updates of the run;
• (ii) the loss that averaging recovers lives on longer timescales and must be weakly expressed in gradient energy.

An Ornstein–Uhlenbeck check (claude_2, k=2, s45, same seed across LR arms) refines this. An equilibrated OU process has a stationary excess linear in η. Raising the LR 1.41× raises the averaging gain by 1.23–1.35× at W=2 and only 1.10–1.13× at W=64 (updates 300/450/600). Lowering it to ×0.71 gives 0.67–0.92× at W=2 and about 0.90× at W=64. The excess is sub-linear in η at every window, so no window is in OU equilibrium. The raw BPB gap between LR arms is about 2.5× the W=64 gap, so it is mostly lost progress, not removable excess.

$B_{\rm river}$ (§4) is computed from gradient energy and inherits this weighting. We do not know in which direction that biases it as a noise scale for loss, and we draw no conclusion about the size of the batch-size gain from it.

Caveats. One seed and one scale for the probe statistics. $\bar R(k)$ is pooled over the matrices of a group, so per-matrix heterogeneity is averaged out. The probe reads shards 40–49, which the training stream never reaches. Agreement of the probe and split-batch $|\mu|^2$ (within 1% through update 880) shows only that the probe–train mean difference is orthogonal to $\mu$ on average; it does not bound its size. The randomized-gradient validation (Methods M2) bounds it directly at 35 saved states of one d6 run: the training stream's conditional mean matches independent random-packer gradients to 0.1–0.2% of $|\mu|^2$, and the probe-based $\rho_1$ matches a fully random-data estimate within 0.02 through update 757.

4. Batch size: what the noise scale predicts and where it fails

Prediction. The split-batch simple noise scale $B_{\rm simple}={\rm tr}\,\Sigma/|\mu|^2$ treats the whole mean gradient as useful signal. Most of that mean is the bounce (§1). Attributing progress only to the persistent fraction gives $B_{\rm river}=2B_{\rm simple}/(1+\rho_1)$. The preregistered phase prediction below was made from the naive profile: at depth 8, 0.2–0.75M tokens in the stable phase and 2–7M during warmdown, against the reference batch of 0.52M tokens. The probe profile (Table 1) agrees in the stable phase (0.19–0.64M). It is lower in early warmdown: 0.87M at updates 482–600, 1.2–1.9M at 600–880, and 3–5M only in the last 70 updates. The reference batch therefore sits near $B_{\rm river}$ in the stable phase, so halving it should help there. It sits below $B_{\rm river}$ through the warmdown, by 1.7× at its start and by 2–10× from update 600 on, so halving it should help less there. With the probe profile this second half of the prediction is weaker than the naive profile implied, because the first ~120 warmdown updates are only modestly below $B_{\rm river}$. The depth-6 profile (naive estimator, base_d6_diag_s44) is about 0.88× the depth-8 one. $B_{\rm river}$ is weighted by gradient energy (§3), and we do not know whether that makes it too high or too low as a noise scale for loss.

Batch-size arms. k is the number of 64×2048-token microbatches per update (reference k=4). Every arm keeps the token budget fixed (499.6M), with LR, momentum warmup and WD schedules evaluated in token time. Per-update β, LR and WD are unchanged, so halving k doubles the number of updates. The depth-8 arms are claude_2's and use their independent implementation, except the s44 reference, the s44 avg-probe pair and the s44 k=2 WD×0.5 arm, which are ours. The depth-6 arms are ours. Δ is val BPB against the k=4 run of the same seed.

| Arm | Depth, seed | Updates | Val BPB | Δ | Held-out BPB | |---|---|---|---|---|---| | k=2 all phases | 8, s45 | 1906 | 0.986982 | −0.0081 | – | | k=2 all phases | 8, s42 | 1906 | 0.986598 | −0.0074 | 0.985853 | | k=2 stable only | 8, s45 | 1435 | 0.988186 | −0.0069 | 0.987311 | | k=2 warmdown only | 8, s45 | 1424 | 0.993707 | −0.0014 | 0.992787 | | k=2, LR×0.71 | 8, s45 | 1906 | 0.985514 | −0.0096 | 0.984718 | | k=2, LR×0.71 | 8, s42 | 1906 | 0.985780 | −0.0082 | 0.984922 | | k=2, LR×1.41 | 8, s45 | 1906 | 0.988569 | −0.0065 | 0.987678 | | k=2, LR×0.5 | 8, s45 | 1906 | 0.989081 | −0.0060 | 0.988189 | | k=2 all phases, rerun | 8, s45 | 1906 | 0.987436 | −0.0077 | 0.986552 | | k=2 all phases (avg-probe run) | 8, s44 | 1906 | 0.987992 | −0.0075 | – | | k=2, all LR×0.71 (avg-probe run) | 8, s44 | 1906 | 0.988829 | −0.0067 | – | | k=2, betas$^{1/2}$ (k=4 token horizon) | 8, s45 | 1906 | 0.988367 | −0.0067 | 0.987553 | | k=4, betas$^{4}$ (k=1 token horizon) | 8, s45 | 953 | 1.009935 | +0.0148 | 1.009224 | | k=2, LR×1, WD×0.5 | 8, s45 | 1906 | 0.985427 | −0.0097 | 0.984588 | | k=2, LR×1, WD×0.5 | 8, s44 | 1906 | 0.986471 | −0.0091 | 0.985611 | | k=2, LR×0.71, WD×1.41 (k=2 lr·wd per update) | 8, s45 | 1906 | 0.987118 | −0.0080 | 0.986241 | | k=1, LR×1 | 8, s45 | 3812 | 0.989633 | −0.0055 | 0.988753 | | k=1, LR×0.5 | 8, s45 | 3812 | 0.985050 | −0.0101 | 0.984261 | | k=1, LR×0.71 | 8, s45 | 3812 | 0.984928 | −0.0102 | 0.984082 | | k=1, LR×0.35 | 8, s45 | 3812 | 0.985223 | −0.0099 | 0.984399 | | k=1, LR×1, WD×0.25 | 8, s45 | 3812 | 0.982919 | −0.0122 | 0.982165 | | k=1, LR×1, WD×0.25 | 8, s42 | 3812 | 0.980854 | −0.0132 | 0.979952 | | k=1, LR×1, WD×0.25 | 8, s44 | 3812 | 0.982827 | −0.0128 | 0.981954 | | k=1, LR×1, WD×0.25 | 12, s45 | 3812 | 0.924050 | −0.0115 | 0.923150 | | k=1, LR×1, WD×0.25 | 12, s44 | 3812 | 0.923402 | −0.0119 | 0.922479 | | k=2, LR×1, WD×0.5 | 12, s45 | 1906 | 0.926570 | −0.0089 | 0.925688 | | k=1 stable, k=4 warmdown, decay matched | 8, s45 | 2399 | 0.985801 | −0.0093 | 0.984846 | | k=4 stable, k=1 warmdown, decay matched | 8, s45 | 2366 | 0.992843 | −0.0023 | 0.992081 | | k=4, all LR×1.41 | 8, s45 | 953 | 0.994715 | −0.0004 | 0.993784 | | k=4, all LR×2 | 8, s45 | 953 | 0.997215 | +0.0021 | 0.996406 | | k=4, WD×2 | 8, s45 | 953 | 0.996737 | +0.0016 | 0.995918 | | k=2 all phases | 6, s44 | 1906 | 1.037287 | −0.0109 | 1.036444 | | k=2 stable, k=4 warmdown | 6, s44 | 1435 | 1.038268 | −0.0099 | 1.037418 | | k=4 stable, k=2 warmdown | 6, s44 | 1424 | 1.046236 | −0.0020 | 1.045370 | | k=2 to update 250, then k=4 | 6, s44 | 1203 | 1.040754 | −0.0075 | 1.039890 |

Held-out BPB is on the deduplicated shard_06541 (M1); k=4 references are 0.994436 (s45 rerun) and 0.993203 (s42); val references are 0.995113 (s45), 0.994052 (s42), 0.995617 (s44; held-out 0.994801) and 0.995496 (s44 avg-probe run, scored without the held-out pass). The s44 k=1 decay-matched arm is claude_2's and is paired with our s44 reference. Where both are available, held-out Δ agrees with val Δ to within 3e-4. The depth-12 references (ours) are 0.935506 val / 0.934572 held-out (s45) and 0.935279 / 0.934425 (s44); the depth-12 k=1 arms and both decay-matched phase-split arms are claude_2's, and the depth-12 held-out Δs are −0.0114 (s45) and −0.0119 (s44). A depth-12 k=4 reference split into 16 microbatches of 32 (claude_2) gives 0.935719, so the microbatch split at fixed batch per update moves BPB by 2e-4. The depth-6 reference is 1.047405 held-out; all four depth-6 held-out Δs (−0.0110, −0.0100, −0.0020, −0.0075) match their val Δs to within 1e-4.

What holds. The gain is not a disguised LR or WD change: at k=4, raising LR by 1.41× or 2×, or doubling WD, does not reproduce it. At depth 8 the k=4 reference is LR-locally optimal (×1.41 −0.0004, ×2 +0.0021 at default WD). The decay-matched ×1.41 control improves it by only 0.0010, 8% of the gain, on both seeds (s45 claude_2, s44 −0.0010). At depth 6 the reference is not LR-optimal. Decay-matched k=4 arms at ×1.41 and ×2 improve it by 0.0036 and 0.0031 (s44/s45 means), and ×2.83 (s44) by 0.0026, so the k=4 curve is flat from ×1.41 up. Against the best k=4 arm the depth-6 quarter-batch gain is −0.0111 instead of −0.0147 (cell means, best LR per k), and 76% of it survives. The batch gain is therefore not an artefact of an under-tuned baseline at either depth, though at depth 6 a quarter of the untuned figure is. An exact rerun of k=2 at s45 differs by 4.5e-4, which sets the rerun floor for these comparisons. At depth 8 the phase split matches the preregistered prediction (log, 18:01Z). Almost all of the gain comes from the stable phase (−0.0069), and the warmdown-only arm gives −0.0014, close to seed noise. The k=2 gain is reproduced on a second seed and in a second implementation, and it transfers to depth 6. The phase split also transfers: at depth 6 the stable-only arm keeps 91% of the full gain (−0.0099 of −0.0109), and the warmdown-only arm gives −0.0020. The two parts add roughly (−0.0119 versus −0.0109). These splits used default WD, so decay per token also changed at the switch. A decay-matched quarter-batch split at depth 8 (s45, preregistered ≥75%) removes that confound. k=1 in the stable phase and k=4 in warmdown gives −0.0093, which is 76% of the full decay-matched k=1 gain (−0.0122). The reverse arm (k=4 stable, k=1 warmdown) gives −0.0023 (19%), inside its preregistered ≤25%, and the two parts add (−0.0116 versus −0.0122).

What fails.

1. Magnitude. For depth 6 we preregistered Δ ∈ [−0.009, −0.004]. The observed Δ was −0.0109, larger than predicted. In the noise-scale model the gain from halving the batch shrinks as the noise scale grows, so a larger gain would need a smaller noise scale than $B_{\rm river}$, not a larger one. We have no mechanism for the magnitude.

2. Location of the optimum. The k=2 LR optimum lies near ×0.71–×1 (s45: ×0.5 −0.0060, ×0.71 −0.0096, ×1 −0.0081, ×1.41 −0.0065). ×0.71 beats ×1 at s45 and s42 by 0.0008–0.0015 and loses at s44 by 0.0008 (0.988829 vs 0.987992), so the curve is flat over this range and the sign of the retune gain is within seed noise. Averaged over the three seeds, k=2 at ×0.71 is −0.0082 against k=4. The k=1 optimum is bracketed and flat: ×0.35, ×0.5 and ×0.71 span 3e-4 (best ×0.71, −0.0102), and ×1 is +0.0046 worse. Once the LR is retuned, k=1 and k=2 (0.13M and 0.26M tokens) differ by 0.0006 at s45 (best arms, −0.0102 vs −0.0096), close to the 4.5e-4 rerun floor, so under LR retuning we call them tied. With decay per token held fixed instead (Reading), k=1 beats k=2 by 0.0025. The optimum sits at or below the low end of the stable-phase $B_{\rm river}$ (0.2–0.75M tokens). $B_{\rm river}$ gets the direction and the phase dependence right, but it does not locate the optimum. The LR–batch dependence that survives once decay per token is fixed is quantified in the Reading below.

3. Depth-6 warmdown. In the depth-6 run, the k=2 advantage shrinks to its minimum at update 600 (−0.0032) and then reopens through warmdown, reaching −0.0097 at update 900. On its face this contradicts the depth-8 warmdown-only null. The two phase-switched depth-6 arms separate a warmdown effect from carry-over of a better stable-phase iterate. The predictions were preregistered before any result: stable-only Δ ∈ [−0.009, −0.003], warmdown-only Δ ∈ [−0.005, +0.001]. The warmdown-only arm landed inside its interval (−0.0020). The stable-only arm missed on magnitude in the same direction as point 1 (−0.0099). So the reopening is carry-over, not a warmdown benefit. The mid-training curves show why. Switching from k=2 to k=4 at the start of warmdown lowers the loss by about 0.01 within 100 updates (stable-only arm, Δ −0.0060 → −0.0168 between updates 400 and 600). Switching the other way raises it by +0.017. The noisier k=2 iterate hides a better-placed iterate until the LR, or the noise, falls. The full-k2 curve's minimum at 600 marks where the stable-phase advantage is most hidden, not where it is smallest. A third preregistered arm uses k=2 only up to update 250, the first half of the stable phase (Δ ∈ [−0.008, −0.002], point −0.005). It lands inside the interval at −0.0075, near the front-loaded edge. The first half of the stable phase gives 76% of the stable-only gain. Its lead appears as soon as it switches to k=4 (−0.025 at update 300) and then erodes under k=4 to −0.0075. The placement advantage is bought early and partly decays when the noisier, shorter-memory regime ends.

4. Saturation (post hoc), restricted to one model family. Consider McCandlish et al.'s fixed-token trade-off with a noise scale that may vary by phase $p$: effective tokens $D_{\rm eff}(x)=\sum_p D_p/(1+c_px)$, where $x\in\{1,2,4\}$ is the batch in units of $k=1$ and $c_p=B_1/B_{{\rm noise},p}$. Take loss $L=A\,D_{\rm eff}^{-\alpha}+C$ with $A>0$ and $0<\alpha\le1$, with each batch at its own optimal LR, as the model assumes. Then the halving ratio $R=g_1/g_2=[L(4)-L(2)]/[L(2)-L(1)]$ is at most 2. Proof. Write $h=D_{\rm eff}$. By Cauchy–Schwarz, $h'^2=\big(\sum_pD_pc_p(1+c_px)^{-2}\big)^2\le h\cdot\sum_pD_pc_p^2(1+c_px)^{-3}=h\,h''/2$. Hence $(\alpha+1)h'^2\le h\,h''$, so $L(x)$ is concave in $x$, and concavity on $\{1,2,4\}$ gives $g_1\le2g_2$. (noise_scale_bound.py checks this on 200,000 random phase mixtures; the maximum is 1.99999998.) The statement covers only this family. It says nothing about stochastic-optimization models outside it, such as conditioning, oscillation or LR-dependent noise effects, which McCandlish et al. themselves list as limitations [4].

   The submitted version compared this bound with ratios from arms tuned unevenly across $k$ and argued that further tuning could not lower $R$. That argument was wrong: improving $k=4$ lowers the numerator, and improving $k=1$ raises the denominator. At depth 8, s45, replacing the default-LR $k=1$ arm with its matched-decay LR×0.71 arm (0.981040) lowers $R$ from 3.86 to 2.21. We now score $R$ only on decay-matched arms, take the best LR per $k$ on the s44/s45 cell means, and propagate per-run noise by a parametric bootstrap. The bootstrap uses a residual sd of 1.1×10⁻³ from an additive seed + cell fit over all 47 decay-matched runs, and it repeats the per-$k$ LR selection in every draw (score_halving.py):

   | depth | LR cells per $k$ (k4 / k2 / k1) | best LR (k4, k2, k1) | $R$ | 90% interval | $R$ at LR×1 only | |---|---|---|---:|---|---:| | 6 | ×0.71, ×1, ×1.41, ×2 (×2.83 s44 only) / ×0.71, ×1, ×1.41 / ×0.71, ×1, ×1.41 | ×1.41, ×1, ×0.71 | 3.23 | 1.76–7.29 | 6.66 | | 8 | ×1, ×1.41 / ×0.71, ×1 / ×0.71, ×1 (×1.41 s45 only) | ×1.41, ×1, ×0.71 | 1.89 | 1.25–3.51 | 3.06 | | 12 | ×1 / ×1 / ×1 | untuned | 3.06 | 1.64–8.05 | 3.06 |

   With the LR tuned per $k$, depth 8 is consistent with the bound ($R=1.89$; the decay-matched k=4 ×1.41 cell, both seeds, is now its best k=4 cell). Depth 12 has no LR tuning at any $k$ and its interval includes 2. Depth 6 has the densest grid. Tuning matters most there for $k=4$, whose optimum lies at ×1.41–×2 rather than ×1 and closes a quarter of the gain. The point estimate ($R=3.23$) sits above the bound, but its interval includes 2. Reaching $R=2$ would take $L_1$ 0.0016 lower, i.e. 1.5 per-run σ. The best $k=1$ cell (×0.71) is the lowest LR tested at depth 6. At depth 8, $k=1$ is flat within 3×10⁻⁴ from ×0.35 to ×0.71, so a lower cell is unlikely to supply that 0.0016. We therefore do not claim that the gain saturates faster than this model family allows. The submitted version's claim that it did "at every depth" is withdrawn.

   A second check fits McCandlish's per-token rate with the measured two-batch noise scale (claude_2). It overpredicts the stable-phase lead of $k=1$ over $k=4$ by 2–5× (Supplement figure lead_vs_mccandlish.png; its numerical inputs and the plotting script are in impl_claude2/). The observed lead is an early offset of 70–110 reference steps that stops growing after about 20% of training, in line with the depth-6 arm that used $k=2$ only up to update 250 (point 3). That comparison uses raw training loss at constant LR, so it is a check on the shape of the lead, not a test of the bound.

Reading. These controls constrain explanations of the batch gain. They do not identify its mechanism, because changing the batch changes the gradients, the nonlinear optimizer states and the later trajectory together. Rescaling the optimizer betas to match token horizons neither removes the gain at k=2 (betas$^{1/2}$ keeps ~83% of it, −0.0067 vs −0.0081) nor reproduces it at k=4 (betas$^4$ is +0.0148 worse; this also takes β2 to 0.81, so it mixes noisier second moments with shorter averaging). An end-of-training average over token-matched windows leaves the s44 gap unchanged at LR×1 (raw −0.00743, averaged −0.00738 to −0.00742) and at all LRs ×0.71 (raw −0.00657, averaged −0.00655 to −0.00656; see Limitations), against the prediction of a lower-temperature reading. The memory result (§2) agrees only in the narrower sense that longer per-update momentum hurts. Two further controls are informative but not identifying.

• Noise injection (claude_2). Adding k=1-level per-update noise to the k=4 Muon gradients costs +0.055 (s45). The extra noise is c·(g₀−g₁) with c = √(3/8), so the window mean is unchanged and per-update variance rises to k=1 level, at the k=4 step count, LR and WD. So added per-update noise alone, at the k=4 step count, is harmful. This argues against reading k=1's gain as a benefit of its noise, but the injected noise is not distributed like k=1's own gradient noise along the trajectory. A related measurement uses quantization noise [32]. At k=1, an all-FP4 backward pass costs +0.0088 BPB against the k=1 BF16 reference, which is 1.6× its cost at k=4 (+0.0055). The 2×2 interaction is +0.0033, at a 0.75× lower noise dose. The small-batch gain survives: k=1 with FP4 still beats k=4 in BF16 by 0.0055 (claude_2, one seed, descriptive: the dose check failed, .1253 against a floor of .129).
• Anchored forward (codex_2 [8]). One arm keeps all 3,812 sequential optimizer-state transitions of k=1, but commits forward parameters only every fourth step. It does not preserve the live k=1 improvement: it is +0.0188 against live k=1 and +0.0066 against k=4 (seed 43). Transition count alone is therefore insufficient under this intervention. Changing the commit interval also changes later gradients, states and cautious masks, so this does not apportion the benefit causally. A period-2 bounce that cancels only under live feedback is consistent with the result, but the result does not identify it.

At fixed tokens, the LR retune acts mostly through nominal decay per token, not step size. Decoupled decay per update is lr·wd, and halving k doubles the number of updates, so at default settings k=2 silently doubles decay per token. k=2 with WD×0.5 at LR×1 (−0.0097) matches k=2 with LR×0.71 (−0.0096) at s45. A paired s44 replicate gives −0.0091 (held-out −0.0092), 0.0015 better than default k=2 and 0.0024 better than all LRs ×0.71 at the same seed. The two-seed mean of the WD route is −0.0094, against −0.0082 for the three-seed LR route. So halving WD is at least as good as lowering the LR, and at s44 it is clearly better. (The preregistered s44 interval, ±0.0015 around default k=2, was missed by 3e-5 on the favourable side.) k=2 with LR×0.71 and WD×1.41 keeps the default k=2 lr·wd per update and loses the whole retune gain (0.987118, against 0.986982 and a rerun at 0.987436; both within the 4.5e-4 floor). The k=4 WD×2 arm (+0.0016) points the same way. The rule also carries to the quarter batch at depth 8. k=1 at LR×1 with WD×0.25 gives 0.982919 (s45), 0.980854 (s42) and 0.982827 (s44), Δ −0.0122, −0.0132 and −0.0128 against k=4 (held-out −0.0123, −0.0133, −0.0128; mean −0.0127). That is 0.0020 better than the best LR-retuned k=1 arm (×0.71, 0.984928) and 0.0025 better than decay-matched k=2 at s45 (0.985427). Lowering the LR was therefore a worse proxy for lowering decay per token. The working rule is to hold lr·wd·(number of updates), the nominal decay integral, fixed when changing batch at fixed tokens. With cautious decay the realized shrinkage also depends on the mask fraction (M3). This is consistent with normalized weight decay [9], which modern coupled-decay recipes omit, and with the fixed-timescale view of AdamW decay [10, 11]. Dang et al. [6] (§3.2) select the same rule (both LRs fixed, matrix WD scaled linearly with batch) from 216 candidates for MuonW-AdamW. Our additions are narrower. In a recipe with cautious (mask-dependent) WD and NorMuon, the rule accounts for the k=2 retune at fixed tokens. A naive LR sweep at default WD exaggerates the LR shift with batch, because it confounds step size with decay per token. A smaller shift remains once decay is fixed (see below). A preregistered transfer test at depth 6 (two seeds, s44/s45) uses Muon WD scaled only, at LR×1. With decay per token held fixed, k=2 (WD×0.5) gives 1.036551/1.037227 against k=4 at 1.048212/1.049672, a mean of −0.0121. k=1 (WD×0.25) gives 1.034764/1.035396, a mean of −0.0139. The paired k1−k2 difference is −0.0018 in both seeds, against −0.0025 at depth 8 (s45). So the decay-matched quarter batch still gains at both depths, with diminishing returns, and we see no crossover down to the microbatch-64 floor. At depth 12 (135M parameters, same 499.6M tokens), the decay-matched quarter batch gives Δ −0.0115 (s45; held-out −0.0114), inside a preregistered interval of [−0.012, −0.006], and −0.0119 on a second seed (s44; held-out −0.0119). Across depth the gain shrinks at most mildly. It is −0.0139 at depth 6 (two seeds), −0.0127 at depth 8 (three seeds) and −0.0117 at depth 12 (two seeds), or 1.33%, 1.28% and 1.25% of the reference BPB. Each step is about one cross-seed sd. The reference WD is not mistuned. At depth 6 (s44, preregistered), k=4 with WD×0.5 gives +0.0004 and with WD×2 gives +0.0011 against WD×1. At depth 8 (s45), the corresponding arms give +0.0013 and +0.0016. The batch gain is therefore measured against a reference whose WD is tuned at its own batch size.

Residual LR preference at fixed decay. Fixing decay per token removes most of the LR preference. Only the sign of what remains is consistent. All arms below were preregistered, and each ×0.71 arm keeps lr·wd per update fixed.

• Reference batch (k=4). Lowering LR ×0.71 costs +0.0010 and +0.0006 at depth 6 (s44/s45) and +0.0013 and +0.0016 at depth 8 (s44/s42).
• Quarter batch (k=1). The same cut gives −0.0001 ± 0.0010 at depth 6 and −0.0007 ± 0.0008 at depth 8 (SE, three seeds each). Neither is distinguishable from zero. At default WD, by contrast, the k=1 arms at depth 8 differ by 0.0047.
• k=2. At depth 8, the cut gives +0.0035 (s44) and −0.0006 (s45) against decay-matched k=2. The s44 anchor beat its own preregistered interval. At this noise level, two seeds cannot place k=2 relative to k=1 and k=4. A preregistered test of a peak in LR sensitivity near $B_{\rm river}$ (Δ ≥ +0.002 at s45) failed.

An exploratory response-surface fit to 21 depth-8 s45 arms suggested an optimum shifting roughly as √batch. It failed its preregistered interpolation check at k=2 (+0.0007 predicted, +0.0035 observed), so we do not claim a scaling form. A refit with seed intercepts over 37 arms and three seeds (residual sd 0.0009) still excludes a zero shift between k=4 and k=1 (log2 optimum shift −0.68, 90% CI −1.00 to −0.26). It does not resolve k=2 (CI on its shift from k=4 spans −0.52 to +0.42). The supported statement is narrower. At fixed decay per token, the reference batch prefers LR×1 over LR×0.71 by about 0.001, while the quarter batch is indifferent within ±0.001. At default WD the naive sweep shows a shift several times larger.

At depth 6, k=2 with WD×0.5 is also only −0.0008 better than k=2 with WD×1 (s44), so the decay correction matters less there than at depth 8. This is what one expects when the gradient signal does not persist beyond about 16 updates (§3). A diagnostic based on gradient energy predicts the sign and the phase. It underpredicts the size, and we do not explain the size.

Token horizon (preregistered, d6). All results above use one budget, 499.6M tokens (19 tokens/parameter at d6, 10 at d8). We therefore asked whether the decay-matched small-batch gain survives a longer horizon. Setup:

• Model: d6 (26.3M parameters), trained at 1× (953 reference updates) and 2× (1906 updates, 999.3M tokens), and on s44 also at 4× (3812 updates, 1998.6M tokens, 77 tokens/parameter), with every schedule on the token clock.
• Data: the longer corpus appends ClimbMix shards 10–49, with exact duplicates of the validation and held-out shards removed. A CPU replay of the loader confirms that the 2× stream ends inside shard 19 without repeating data. Every 2× and 4× run reports final_epoch = 1, so no document is repeated.
• WD brackets: at 2× each batch size also gets a WD×0.5 arm, so a horizon-dependent shift of the optimal integrated decay cannot masquerade as a change in the batch gain.
• Classes, with $G^*=\min(k1\ \text{arms})-\min(k4\ \text{arms})$, fixed before any 2× endpoint was scored:
  • preserved: $G^*(2\times)\le 0.8\,G(1\times)$
  • short-budget: $G^*(2\times)\ge 0.5\,G(1\times)$
  • ambiguous: between the two
  • reversal: $G^*>0$
• At 4× (s44), fixed with the 2× classes: shrinkage continues if $|G(4\times)|<|G(2\times)|$. A later prediction (H4), written after the 2× results and the 4× k=4 endpoint but before the 4× small-batch endpoints, assumed the token lead grows as $D^{1/2}$ and gave $G(4\times)\approx-0.0031$ (band −0.0020 to −0.0045).

One author predicted preservation at 55%. The other predicted shrinkage.

| d6, seed | horizon | k=4 | k=4, WD×0.5 | k=1 (decay/token) | k=1, WD×0.5 | $G^*$ | ratio to 1× | |---|---|---|---|---|---|---|---| | s45 | 1× | 1.048841 | – | 1.034507 | – | −0.01433 | – | | s45 | 2× | 1.015611 | 1.015837 | 1.010652 | 1.010410 | −0.00520 | 0.36 | | s44 | 1× | 1.048212ᵛ | – | 1.034764ᵛ | – | −0.01345 | – | | s44 | 2× | 1.015966ᵛ | 1.015332ᵛ | 1.008941ᵛ | 1.009517ᵛ | −0.00639 | 0.475 | | s44 | 4× | 0.993610ᵛ | – | 0.991825ᵛ | – | −0.00179 | 0.13 |

Held-out BPB for s45. ᵛ marks s44 validation BPB, the primary metric of our harness. The s44 held-out scores give the same result: $G^*$ is −0.01348, −0.00630 and −0.00175 at 1×, 2× and 4× (ratios 0.467 and 0.130). Matching CE/PPL are in the supplement.

Halving the integrated decay changes each s45 2× arm by ≤2.4×10⁻⁴, which is within the same-seed rerun floor. The bracket therefore does not move $G^*$; the one-sided version (k=4 bracket only) is −0.00496. On s44 the k=4 WD×0.5 arm is the better k=4 arm by 6.3×10⁻⁴, and it enters $G^*$ as the minimum.

Reading. At twice the tokens the gain shrinks to 0.36 (s45) and 0.475 (s44) of its 1× value. Both seeds fall in the short-budget class (s44 is 3.4×10⁻⁴ inside the 0.5 boundary), and the preserved-gain prediction failed. At four times the tokens (s44) it is 0.13 of its 1× value and 0.28 of its 2× value, so the shrinkage continues as predicted. H4 failed on the small-gain side: −0.00179 is outside −0.0020 to −0.0045. The 4× horizon has no k=4 WD×0.5 arm. At 2× that bracket improved k=4 by 6×10⁻⁴, so adding it could only make the 4× gain smaller.

• Val-subset lead in the 2× run (k4 − k1, at steps 250/500/…/1750): 0.040, 0.015, 0.011, 0.007, 0.006, 0.004, 0.005. It declines through the stable phase rather than plateauing.

• In tokens, k=1 does not lose ground on trend. The number of extra reference steps k=4 needs to reach k=1's val-subset loss is:

  • 2×: 73, 91, 108, 82, 113, 119, 131, 159, 150 at steps 50, 100, 200, 300, 400, 500, 600, 700, 750 (8-batch subset, so the series is noisy)
  • 1×: 71, 91, 116, 92 at steps 50–300

  The implied local step-efficiency of k=1 falls from ≈1.2× (steps 50–200) to ≈1.1× (steps 500–800). As the loss curve flattens, a slowly growing token head start buys less loss.

• The 4× runs evaluate a fixed 2.1M-token val subset every 200 updates (every 50, 100 and 200 reference updates at k=1, 2 and 4), which gives a longer within-run series. We read the lead only where the k=4 reference is still at constant LR (reference evaluations up to step 1800 of its 1906-step stable phase); beyond that the interpolation reaches into the reference's warmdown and biases the lead short. The k=1 lead is 114, 137, 191, 209, 246, 223, 216 and 176 reference updates at steps 200, 400, …, 1600 (k=2: 76, 111, 171, 222, 226, 208, 260, 188). It rises over the first ~1000 reference updates and then stops growing. The preregistered power-law fit (lead ∝ $t^\gamma$) is not readable by its own rule, because the residuals exceed twice the evaluation noise: the shape is a rise and then a plateau, not a power law. The same-step BPB gap needs no interpolation and falls monotonically: −0.048, −0.0195, −0.0088 and −0.0040 at steps 200, 400, 1000 and 1800, and −0.0018 at the end.

• Re-tuning k=4's LR at both horizons (LR×1.41, decay-matched; s44: 1.044964 at 1×, 1.013785 at 2×) gives a tuned ratio of 0.474 (best k1 − best tested k4: −0.00484 / −0.01020), against 0.475 untuned. The shrinkage is therefore not k4 LR mistuning (single seed).

• At d8 the decay-matched k4 LR×√2 control improves the reference by only 0.0010 (8% of G; s44 and s45 alike). At d6 (s44) it recovers 24%, and there the k4 LR curve is flat over ×1.41–2.83. So the small-batch gain is not an LR-area effect at either depth.

• In token-equivalent terms, k=1 at 1× matches k=4 at ≈1.3× the tokens. The log-linear interpolation between the k=4 1× and 2× endpoints gives 1.34–1.35× on two seeds; a quadratic in log-tokens through the s44 k=4 1×/2×/4× endpoints (1.048212 / 1.015966 / 0.993610 val) gives 1.30×. At 2×, k=1 matches k=4 at ≈1.2× (s44; 1.21× quadratic, 1.24× log-linear), and at 4× at ≈1.08× (quadratic, extrapolated slightly past the last reference point). The k=4 curve is concave in log-tokens (−0.0322, then −0.0224 per doubling), so the log-linear figures are biased up by ≈0.03. Against the LR-tuned k=4 (×1.41), the quadratic multipliers are ≈1.25× and ≈1.18× at 1× and 2×.

  As an absolute head start, the endpoint lead is 288, 393 and 290 reference updates at 1×, 2× and 4×. So the token-equivalent advantage stops growing between 2× and 4× while the loss curve keeps flattening. Equivalently, the BPB gain falls faster at the second doubling (0.47, then 0.28) than at the first. k=2's endpoint lead keeps growing (243, 349, 409).

• The earlier dose-response (each doubling of the k=1 phase adds ≈0.0025 at 1×) is a loss-unit measurement at a fixed horizon. It does not transfer across horizons, and we do not use it to extrapolate.

• The preferred batch moves up with horizon (s45, post hoc; s44 preregistered on post 33).

  • k=2 (2^18 tokens, decay per token) scores 1.036590 at 1× and 1.009227 at 2×.
  • The paired contrast $D_h=\mathrm{BPB}(k2,h)-\mathrm{BPB}(k1,h)$ is +0.0021 at 1× and −0.0014 at 2×. That gives $D_{2\times}-D_{1\times}=-0.0035$, beyond the ±0.001 threshold.
  • So at 2× the best tested batch is k=2, not k=1, and the best small-batch gain is −0.0064 (ratio 0.45 to the 1× k1 gain).
  • This is the direction predicted by critical-batch growth with tokens (Merrill et al. [15]).
  • s44 (preregistered on post 33: REPLICATED if $D_{2\times}-D_{1\times}<-0.001$): k=2 scores 1.036551 at 1× and 1.009635 at 2× (val). $D_{1\times}=+0.00179$ and $D_{2\times}=+0.00069$, so $D_{2\times}-D_{1\times}=-0.00109$. This counts as REPLICATED, but only by 9×10⁻⁵ past the threshold.
  • On s44, k=2 does not overtake k=1 at 2× ($D_{2\times}>0$). The replicated claim is therefore the direction: the advantage of k=1 over k=2 shrinks with horizon.
  • s44 at 4× (preregistered on post 33: the drift CONTINUES if $D_{4\times}<D_{2\times}-0.001$): k=2 scores 0.991160 and k=1 0.991825, so $D_{4\times}=-0.00067$ (held-out −0.00063), and $D_{4\times}-D_{2\times}=-0.00136$. The drift CONTINUES. On s44, k=2 overtakes k=1 at 4× rather than 2×. The horizon at which the best tested batch changes is seed-dependent; the direction is not.

Scope. These are two horizons at d6 on two seeds (one per implementation) and a third, 4×, on one seed. We claim that the gain is concentrated at short horizons in this regime: at 77 tokens/parameter about an eighth of it remains. We do not claim that it vanishes at any particular budget. Whether it is worth using at compute-optimal or overtrained budgets for larger models is untested.

Where the decay-matched gain is not (preregistered, depth 6, s44). Muon with decoupled decay settles into rotational equilibrium [29]. There $\lVert W\rVert\propto\sqrt{\eta/\lambda}$ and the angular step per update is $a\approx\sqrt{2\eta\lambda}$. Decay matching ($\lambda\propto k$ at fixed $\eta$) therefore halves $a^2$ per halving of $k$, doubles the update count $n$, and keeps the heat per token $n a^2$ fixed. This is the square-root rule of Malladi et al. [30], written in angular units. At fixed heat, first-order progress per token scales as $\sqrt n\cdot c$, where $c=\cos(\delta,\mu)$ is the alignment of the update with the exact mean gradient. So the batch gain can come from three places:

• drift, if $c$ falls less than $1/\sqrt2$ per halving;
• overshoot cost, if the second-order share $C/F$ falls at small batch (Marek et al. [23]);
• neither, leaving heat or terms outside the matrix ledger.

We measured $c$ and $C/F$ per update. The probe is a gradient at the current weights on a batch from shards never trained on (M2, probe=true in the harness), and it is unbiased under feedback. Arms are matched in nominal decay per token and in the rotational-equilibrium norm $\sqrt{\eta/\lambda}$ (k4 LR×1.41 WD×0.707, k2 WD×0.5, k1 LR×0.71 WD×0.354), and each lands within 1×10⁻⁴ of its uninstrumented twin. The realized norms $\sqrt{\sum\lVert W\rVert^2}$ over the Muon matrices, logged per update, are 352/345/342 (k4/k2/k1, stable phase), 545/531/529 (first warmdown third) and 575/559/557 (last third), so they match to within 2–3% (k2/k4 0.98, k1/k4 0.97). The estimator was calibrated on a river-and-walls quadratic before the runs; it is within 0.005 of exact inside the measured domain, and a signal-free probe gives 0.

| phase (ref. updates) | $c$, k=4 | $c$, k=2 | $c$, k=1 | $r_{21}=c_2/c_4$ | $r_{12}=c_1/c_2$ | $C/F$ k4 / k2 / k1 | |---|---|---|---|---|---|---| | stable 100–482 | 0.107 | 0.082 | 0.053 | 0.76 ± 0.05 | 0.65 ± 0.05 | 0.94 / 0.97 / 0.99 | | warmdown 482–717 | 0.077 | 0.056 | 0.033 | 0.73 ± 0.03 | 0.58 ± 0.03 | 0.98 / 1.00 / 1.05 | | warmdown 717–953 | 0.055 | 0.035 | 0.019 | 0.64 ± 0.10 | 0.54 ± 0.08 | 0.96 / 1.00 / 1.07 |

• The preregistered bands were: $r\ge0.9$ signal-dominated, $r\le0.8$ noise-dominated. $r_{21}=0.76$ is noise-dominated (we had predicted 0.95).
• Halving the batch at matched decay therefore buys only $\sqrt2\,r_{21}=1.08\times$ drift per token. Because successive mean gradients are strongly anticorrelated (§1, $\rho_1\approx-0.84$), about 90% of $\lVert\mu\rVert^2$ is the oscillating component, so $c$ mostly measures alignment with the alternating component. The null therefore rules out a gain carried by alignment with the full mean gradient, not one carried by alignment with its slow part, which these runs do not isolate. Most of the fall in $c$ is a property of the polar map at fixed state, not a change of state. In an offline replay at fixed weights (claude_2; d6, s45 and s44), adding real microbatch-difference noise equal to one extra k=4 minibatch of variance to the current gradient leaves $\lvert u\rvert$ unchanged and lowers $\langle\mathbb E[u],\mu\rangle$ by 7%, 8% and 12% in the three phases (s44 replicates s45 within 0.004). The persistent k4→k2 increment is that noise amplified by the stationary Nesterov buffer (β=0.95), a single-step dose of 3.2. On s44 it lowers $\langle\mathbb E[u],\mu\rangle$ by 17%, 20% and 29%, with the stable window starting at update 300, where β has reached 0.95. That is 71%, 74% and 81% of the measured fall in $c$ (in $1-c$ terms), with the same phase ordering. The replay is at fixed state, so it does not capture the k=2 trajectory's own state differences or the remaining quarter. $\mu$ comes from one independent val-batch gradient per window, which is unbiased and pooled over 8–10 windows per phase. This is the noise-induced mean distortion of orthogonalized updates analysed for DP-Muon [31]. The replay script and its assay records are in the supplement (impl_claude2/code/assay_efall.py, impl_claude2/replay/).
• $C/F$ does not fall at small batch ($q_{21}=1.03$; preregistered ≥0.95 = adapted to the edge of stability). The overshoot share is pinned near 1 at every batch, so a reduction in overshoot does not carry the gain either.
• The AdamW groups (embeddings, scalars) take 14–18% of first-order movement in the stable phase (k4/k2/k1: 0.18/0.15/0.14) and 16–33% in warmdown, where the share rises at k=1 (last third: 0.23/0.26/0.33). In the stable phase, where the k2-over-k4 gain is measured, the share is slightly lower at small batch, so the matrix ledger is not missing a gain carried by the AdamW parameters.
• The quarter batch is less aligned still: $r_{12}=0.65\pm0.05$, below even the pure-noise value $1/\sqrt2$ for a fixed-norm update. So at equal heat k=1 has 8% less drift per token than k=2, consistent with the small k2→k1 step (−0.0015 here). The preregistered saturation test (a noise boundary between k2 and k1, i.e. $r_{12}\le0.8$ with $r_{21}\ge0.9$) is not applicable, because $r_{21}$ is already noise-dominated. The alignment statistic orders the arms ($|\Delta c|\approx0.025$–$0.03$), but in the opposite direction to BPB. Direction noise under the polar map is real, and the small batch wins despite it.
• Lower heat does not carry it either (depth 8, s44, preregistered). As a temperature proxy we use the averaging gap, i.e. BPB of a uniform average of the last W reference steps of iterates minus BPB of the raw iterate. At reference step 450, W=32, the decay-matched k=2 arm gives −0.0274 against −0.0229 at k=4 (1.20×). That is outside the equal-temperature band (−0.020 to −0.026) on the hot side, and far from the $a^2$ band (−0.0115 to −0.016). Only the fastest component (W=2) is cooler (0.73×); the slow components (W=8, 32) are 1.17–1.27× k=4 at steps 300–600. The non-matched k=2 arms are hotter still (WD×1: 1.43×), so decay per token orders the proxy but does not equalize it. The proxy is not a function of the angular step alone: relative to the matched arm, the gap's elasticity to $a^2$ is 0.25 when $a^2$ is raised through WD and 0.10 when it is raised through LR. The window's lag along the descending curve costs each arm about 3×10⁻³ and biases the k2−k4 difference by about 2×10⁻⁴. That is comparable to the 4×10⁻⁴ margin past the preregistered rejection line and to the same-seed spread between two k=4 arms (2×10⁻⁴), so we read P5 as at the boundary on the hot side, not as a decisive rejection. The robust statement is that the decay-matched small batch is not cooler.
• The decay-matched small batch is therefore ahead in where the averaged trajectory is, not in how much it fluctuates around it. Between W=32-averaged iterates the k2−k4 lead is −0.0147 at step 450, against −0.0102 between raw iterates. In token units the averaged iterate leads by 1.26×/1.19×/1.11× at steps 300/450/600. Per token, first-order Muon progress in the stable phase is 15% higher at k=2 (d6). Because $c$ is lower, this comes from larger $\lvert\delta\rvert\,\lvert\mu\rvert$, not from alignment. We report this as the location of the gain, not as a net-progress attribution. With $C/F\approx1$, overshoot cancels almost all first-order progress. The exact one-step change of the probe loss (the same probe before and after each update; code/ledger_blocks.py, post hoc) is only 0.9–1.1% of first-order movement in the stable phase (full-line ratio 1.98 at every $k$), 1.3–1.8% in the first warmdown third and 9–11% in the last. The split of that small net change between Muon and AdamW parameters is not identified, because the end-point gradient is recorded only for the Muon matrices.

Why matched bookkeeping predicts no gain (proposition). Let $B$ be the batch, so there are $n\propto1/B$ updates per token. Let the Muon decay per update be matched per token ($\lambda\propto B$ at fixed $\eta$), and let each matrix update be its polar factor. Assume:

• (A1) rotational equilibrium: $\lvert W\rvert$ sits at the decoupled-decay equilibrium, so the angular step is $a=\sqrt{2\eta\lambda}\propto\sqrt B$;
• (A2) noise-dominated polar updates: the alignment of the polar factor with the mean gradient is $c\propto\sqrt B/\sigma$;
• (A3) linear response at the current state: the per-token loss change is a drift term $n a c$ times a slope plus a heat term $n a^2$ times a curvature, where slope and curvature depend on the state but not on $B$;
• (A4) state locality: per-token rates evaluated along the trajectory integrate to the endpoint, with no memory beyond the state.

Under these assumptions drift per token $nac$ and heat per token $na^2$ are both independent of $B$. The two batches then follow the same trajectory per token, and the endpoint gain is zero. code/check_bookkeeping.py checks both scalings numerically:

• the equilibrium $a$ matches $\sqrt{2\eta\lambda}$ to 0.3%;
• $c(B)/c(2B)=0.68$–$0.73$ when noise dominates, rising to 0.98 when the mean dominates;
• drift and heat per token are flat in $B$ within Monte-Carlo noise.

Three of the assumptions have a measured proxy here, and each proxy behaves as the proposition requires:

• (A1): the realized norms of the matched arms agree within 3% (352/345/342 in the stable phase);
• (A2): $r_{21}=0.76\pm0.05$ is close to $1/\sqrt2$;
• (A3): $C/F$ is pinned near 1 at every batch, and the small batch is not cooler.

The ~0.008 gain per halving shows that at least one assumption fails anyway. Either the proxies miss the failure, or (A4) fails; no measurement in this section tests (A4). Forks from fixed saved states, comparing batches over the same token interval, probe it directly (§5). Two further controls in this section bear on other routes. Token-matched betas keep ~83% of the k=2 gain, so the gain does not require k=2's shorter memory in tokens. The norm-matched M3 arms equalize the early angular rate per token as well as the equilibrium, and the gain persists in them, so it does not require a difference in how fast the weight norm equilibrates. The gain is nevertheless front-loaded in those arms (post hoc): per token, first-order Muon movement at k=2 is 1.38× k=4 in reference updates 50–100 and 1.15× over the stable phase. At depth 6 the k=4 LR×1.41 benefit is largest at update 200 (−0.0055/−0.0058, s44/s45), about zero at 400 and −0.0025/−0.0034 at 800, which places the tunable quarter of the depth-6 gain in the transient and in warmdown. The ~0.008 gain per halving (d6/d8) is therefore not accounted for by linear-response bookkeeping around equilibrium. Two candidate failures of (A3) that the proxies would miss remain, and we test neither: (A) progress along the slow component, which $c$ cannot see, and (B) the nonlinearity of the polar map in a noisier momentum, since the polar factor of a noisy sum is not the polar factor of its mean.

Read post hoc, $r_{21}$ fits $c\propto(1+B_{\rm noise}/B)^{-1/2}$ with $B_{\rm noise}\approx1.4$M tokens in the stable phase (90% interval 0.6–9M), and $B_{\rm noise}$ rises into warmdown. That predicts a k4→k2 token efficiency of 1.16, against an observed endpoint token multiplier of 1.26. A noise scale that grows as the loss falls is also the standard explanation for the horizon shrinkage above. We do not claim this link from one seed.

5. Full-state forks: what a small-batch window is worth, and when

§4 does not say which assumption of the proposition fails, and no measurement there tests (A4). A fork compares two batch sizes over the same tokens from the same saved state (Methods M4), so it prices a window of small-batch training at a given point in the run. The trunk is the depth-6 k=4 configuration (s44). It ends at 1.048179; the uninstrumented run of the same configuration ends at 1.048212. Forks start at reference updates $T_0\in\{150,482\}$, with a window of $T_0/2$ reference updates at k=2 (decay-matched, WD×0.5) or k=4. All tests were preregistered on the forum before the arms ran (preregistrations.md). code/score_forks.py and code/score_retained.py score them from the bundled runs.

Precision. From state 150 the k=4 fork reproduces the trunk's update-150 loss bit for bit, and its loss stays within 4.7×10⁻⁴ over the next 20 updates. After a cooldown, a k=4 fork and its data-order twin differ by 4.2×10⁻⁵ ($T_0=150$) and 1.4×10⁻⁵ (482). At the end of the window, before the cooldown, they differ by 1.05×10⁻³ and 7.3×10⁻⁴. A paired fork therefore resolves a few 10⁻⁵ BPB once both arms are cooled. The reduction is 25× at $T_0=150$ and 51× at 482. Each is a single draw, and at 150 the 90% interval is about 2–320 (claude_2).

Banked value. Here each window holds LR×1 and is followed by an immediate 95-reference k=4 linear cooldown.

| $T_0$ | window (ref.) | banked $g$ = k4 − k2 | gap at window end | $g$, 190-ref cooldown | |---|---|---|---|---| | 150 | 150–225 | +0.00778 | +0.00515 | +0.00611 | | 482 | 482–723 | +0.00140 | −0.00232 | — |

• The banked value of a 75-reference window at $T_0=150$ is 0.67 of the whole-run decay-matched gain ($G$ = 0.01166 at depth 6, s44). The value falls 5.6× by $T_0=482$ while the window grows 3.2×. The ratio $R=0.18$ lies in neither preregistered band (log-uniform accrual 0.6–1.6, per-token 2.2–4.6).
• Doubling the cooldown removes 21% of the banked value at $T_0=150$.
• At $T_0=482$ the k=2 branch is behind at the end of the window and ahead after the cooldown. Its cooldown drop is larger by 0.0037, the pattern of a hotter iterate, which agrees with the mid-run averaging probe in §4. With one twin per $T_0$ this split is descriptive: $\Delta h$ is 5.0× its single-draw floor, which is $p\approx0.13$ (two-sided, $t_1$).

Banked values are not endpoint contributions. In claude_2's noisy-quadratic model (plain SGD, uncalibrated; forum post 63), later training erodes a banked lead. On this schedule, a lead banked at update 150 keeps 0.18–0.27 of its value to the end in that model (measured below: 0.22). The same model overstates retention at 482 (0.88–1.13 against a measured 0.75) and puts the retained ratio $R$ at 1.82–2.14 (measured 0.62), so it gets the banked/retained split right only for the early window. The prediction that banked exceeds retained passed at both $T_0$, against the switch-arm (log-uniform) retained estimate of 1.21×10⁻³ per window (6.4× and 1.15×).

Retained value. Here each window runs on the trunk's own LR/WD schedule, and the branch then continues on the trunk's k=4 schedule to the end. The k=4 retained fork from state 150 ends 9×10⁻⁶ from the trunk, against a preregistered gate of 2×10⁻⁴.

• At $T_0=150$ the k=2 window retains +0.00171 (k4 1.048170, k2 1.046465, data-order twin 1.048191). That is 0.15 of $G$ from 7.9% of the tokens, or 1.9× the whole-run average rate per token.
• Of the banked lead, a fraction $E=0.22$ survives to the end. It falls inside the preregistered band [0.08, 0.24] implied by the switch arms, and far from $E\approx1$. Because the trunk LR multiplier is 1 throughout this window, $E$ is a pure erosion ratio here.
• The window carries 0.62 of the unpaired switch-arm value of [100, 250] (2.74×10⁻³) while covering 0.44 of its log-doublings. This does not reject additivity, and with unpaired switch arms (cross-seed sd ≈10⁻³) it is weak in either direction.
• At $T_0=482$ the k=4 retained fork ends 7×10⁻⁷ from the trunk, and the k=2 window retains +0.00105 (k2 1.047130). That is 0.09 of $G$ from 25% of the tokens, or 0.35× the average rate per token. This window lies in the trunk's warmdown, whereas the banked window held LR×1, so the ratio to banked (0.75, preregistered band [0.43, 1.30]) mixes erosion with a different LR regime and is not an erosion ratio.
• Retained value does not grow with the window. The 482 window is 3.2× longer in tokens but retains 0.62× as much as the 150 window. Accrual per token (preregistered prediction 3.21×, band [2.2, 4.6]) is rejected. The ratio lies at the lower edge of the log-uniform band [0.6, 1.6]: both windows span the same 0.585 doublings, and the retained value per doubling falls from 2.9×10⁻³ to 1.8×10⁻³. Because the 482 window lies in the warmdown, this rejects a constant value per token across phases, not per-token accrual at fixed LR (one trunk, d6 s44).
• The two windows together cover 33% of the tokens and retain 0.24 of $G$.

Additivity (preregistered). Retained forks from state 100 price three stable-phase windows: A = [100, 250], B = [100, 150] and C = [150, 250]. The baseline for A and B is a k=4 fork from state 100, which ends 3.9×10⁻⁵ from the trunk; C uses the 150 baseline. The windows retain A = +0.00317, B = +0.00132 and C = +0.00217.

• $Q=A/(B+C)=0.91$ lies inside the preregistered additive band [0.8, 1.25]. The residual A − (B + C) = −0.00031 is −3.6 times the declared tolerance √3·5×10⁻⁵. Because the state-100 baseline cancels from the residual (codex_1), that tolerance is operational. Independent endpoints with per-difference sd 5×10⁻⁵ would give a residual sd of 7.1×10⁻⁵, or −4.4. So a k=2 window joined to an earlier one probably loses about 9% of its value, but window values add to within 10% in the stable phase.
• A from a k=4 history is 1.16× the unpaired switch-arm value of the same window from a k=2 history (2.74×10⁻³; descriptive, arm spread about 2×10⁻⁴).
• C/ret(150) = 1.27, against 1.26 for log-uniform and 1.33 for per-token accrual (descriptive). At constant LR, retained value accrues nearly uniformly per log-doubling.
• The remaining intervals (updates 0–100, 250–482 and 723–953) have not been priced, so the sum of window values has not been compared with $G$.

Significance in this paragraph uses the preregistered floor of 5×10⁻⁵, because each data-order twin differs from its k=4 fork by less than that (2.1×10⁻⁵ at 150 and 2.8×10⁻⁵ at 482). The retained values are 34× and 21× that floor. A second data-order twin at each $T_0$ (skipping 8 microbatches; 1.048157 and 1.048198) gives three k=4 draws per point, with sd 1.7×10⁻⁵ and 1.4×10⁻⁵ (2 df). The preregistered $t_2$ scale $\sigma_d=\sqrt2\,$sd, floored at 5×10⁻⁵, therefore equals the floor at both points, and both retained values pass the bar of $4.5\sigma_d$. The numerator k4₀ − k2 reuses one of the three draws that define sd, so this statistic is not exactly $t_2$. Under an iid Gaussian null it is conservative: its tail at 4.5 is 0.016, against 0.023 for $t_2$ (codex_1, codex_2). The exact-$t_2$ form [mean(three k4 draws) − k2]/(sd·√(4/3)) is a supplemental calculation made after the outcomes. It gives 86 and 66, or 42 and 26 with sd floored. Either way the draws come from one perturbation family (loader offset) on one trunk. These are ideal-null calculations, not an empirical calibration.

Reading.

• What the forks show about the proposition. Both batches start from one full state (weights and every optimizer buffer) and see the same tokens. The k=2 window nevertheless gains 21–185× its floor, banked or retained. So the gain is produced within 75–241 reference updates of a shared state; it is not inherited from differences in earlier history. The decay-matched k=2 branch (WD×0.5 at LR×1) starts at the k=4 weight norm but has a 1.41× larger equilibrium norm, so (A1) does not hold for it during the window. A preregistered norm-matched fork closes this route. In it, the window runs k=2 at LR×0.71 and WD×0.71, matching both the decay per token and the equilibrium norm of k=4. It retains +0.00129 (26× the floor), 0.76 of the decay-matched window and inside the preregistered band [0.55, 0.95]. The whole-run norm-matched arms keep the same fraction of the decay-matched gain (0.748 and 0.743, s44 and s45). This result holds for one joint intervention. It matches the equilibrium norm $\sqrt{\eta/\lambda}$ and the decay per token. It does not match the achieved norm or angular step over a finite 75-reference window. Its LR change also reaches the AdamW groups (embeddings, head, scalars), which lie outside the scale-invariant argument. Fork runs log no norm or angular trajectories, because diagnostics are off in branches. So the fork rules out one route: that the gain is the equilibrium-norm shift of the decay-matched arm. On its own it does not establish an equal-angular-step batch effect (codex_2's caution). The forks leave the failure in (A3) or in what the proxies of (A1)–(A3) miss: progress along slow directions, which $c$ cannot see, or the nonlinearity of the polar map under buffer noise (§4).
• Banked overstates retained. An immediate cooldown after an early small-batch window overstates its contribution to the final model about 5×; near the warmdown the two agree to within about a quarter. A fork protocol that priced interventions by banked value would therefore mis-rank windows taken at different times, here by a factor of about 3.
• Value per token falls through training. This is consistent with the horizon result in §4 (the gain is worth less once the loss curve flattens) and with the phase split in §4 (most of the gain is in the stable phase), now measured from a shared state rather than inferred from unpaired switch arms.

Related work

River–valley picture and Muon oscillation. WSD warmdown has been explained as high-LR oscillation across valley walls while progress happens along a river [1]. This has been extended to Muon, where orthogonalized updates overshoot sharp directions [2], and momentum's role has been analyzed in that picture [12]. AMUSE [13] combines the river–valley framing with Muon oscillation and schedule-free averaging, and LITE [14] accelerates flat directions for Muon/SOAP. Chen et al. [5] show that Muon tracks a loss-stability boundary. Our near-edge full-update overshoot (§1, 1.96–1.98 through the first warmdown third) is a consistency check on their picture under Nesterov momentum, NorMuon and cautious WD; we do not claim it as new. What we add is a per-update, per-matrix measurement of the bounce/persistent split: within-update split-batch statistics at no extra forward/backward cost, and feedback-free cross-update statistics at one extra forward/backward per update (the probe, Methods M2).

Gradient noise and batch size. We use McCandlish et al.'s simple noise scale [4]. Direct critical-batch measurements [15], WSD-specific batch-size theory and schedulers [16, 17], non-Euclidean noise scales for sign/spectral methods [18], and Muon critical-batch analyses [19] are the closest prior work. Our $B_{\rm river}$ is a heuristic correction for a strongly anticorrelated mean, and it underpredicts the gain (§4). For weight decay across batch sizes, Dang et al. [6] select scaling matrix WD linearly with batch at fixed LRs for MuonW-AdamW, which is the rule our batch arms arrive at (§4). CompleteP [20] tunes the equivalent EMA timescale τ = (η·λ·N)⁻¹ across parameterizations and notes that it ignores schedule shape. Our matched arms keep the schedule shape identical, so the nominal integral $\sum_t\eta_t\lambda_t$ per token is matched as well. Cautious WD applies decay only where the update and the weight agree in sign, so the realized integral can differ between batch sizes (Methods M3).

Momentum schedules. Bounds for LMO-type methods give $1-\beta\propto\eta^{2/3}$ at fixed step size under bounded variance and L-smoothness [3]; our lr_coupled arm applies it pointwise along the schedule, which is our heuristic. Momentum cooldown [7] caps the memory timescale by the LR decay rate. Cutkosky & Mehta [21] and Demon [22] study momentum variation. Marek et al. [23] hold optimizer half-lives fixed in tokens across batch sizes, which our batch arms deliberately do not do. VCMM [24] sets momentum from split-batch noise and successive-gradient drift. Products of successive training gradients include feedback from the optimizer's own step (Methods M2), so a drift estimate built from them is biased unless one factor comes from data the step did not see. IGT for LMO methods [25] corrects momentum staleness by extrapolation.

Iterate averaging. LAWA-type and EMA averaging [26, 27] and schedule-free methods [28] recover part of the cooldown gain. We use averaging only as a read-only probe of the excess loss of the iterates.

Limitations

• Scale and data. Two model sizes (26M and 50M parameters) for the diagnostics and most arms, plus a 135M-parameter check of the decay-matched batch result. One dataset, a 2048-token context, and a ~0.5B-token budget (about 10 tokens per parameter at depth 8 and 3.7 at depth 12). Only the depth-6 horizon arms go beyond it, to 1B (two seeds) and 2B tokens (one seed, no k=4 WD bracket).
• Seeds. Most of our arms use one seed. The same-seed replicate range is 1.2e-4 and the cross-seed sd is about 1e-3. The memory penalty is replicated across seeds and depths. Most depth-8 batch arms are from a second implementation (claude_2). The depth-6 phase-split arms are single runs; the depth-6 decay-matched arms have two or three seeds; the depth-12 decay-matched pair has two seeds; the depth-8 decay-matched phase split is one seed per direction.
• Estimators.
  • Within-update split-batch statistics are unbiased but need $k\ge2$.
  • Feedback-free cross-update statistics come only from runs with the probe (one depth-8 seed). Other runs report $\rho_1^{\rm naive}$, which mixes population drift with feedback from the previous update; we label it and draw no conclusion from it alone.
  • The probe estimates the population gradient on shards 40–49, which the probe runs never train on. If those shards differ in distribution from the training stream, the probe measures a slightly different population.
  • $(1+\rho_1)/2$ overestimates the persistent share under momentum (synthetic check, Supplement).
  • $\bar R(k)$ is pooled over matrices.
  • Influence matching holds only for the linear Nesterov filter, not after polar/NorMuon/cautious WD.
• Baseline. The AdamW-matrix comparator is tuned only over LR and warmup at fixed betas and WD. Its best point (LR $10^{-2}$, 5% warmup, 1.0487; bracketed by 1.0542 at $6\times10^{-3}$ and 1.0822 at $2\times10^{-2}$) trails Muon by 0.053 BPB, and this gap is not the subject of the paper.
• Microbatch floor. The protocol fixes the microbatch at 64 sequences (131k tokens), so k=1 is the smallest batch we can test without changing the forward pass. "The optimum is at or below k=1" is therefore a boundary result. A micro32 arm (claude_2, +0.002 vs k=1) changes the microbatch and is not a clean extension.
• Two-component model. The bounce/river split is an idealization. §3 shows that the loss-relevant slow component is not visible in gradient energy.
• Mechanism of the batch gain, one seed. One reading of the small-batch gain is more progress along the persistent component. The other is a lower effective temperature along flat modes, which predicts that averaging both arms shrinks the gap. We preregistered an end-of-training test (s44, LR×1, token-matched uniform windows of $2^{20}$–$2^{24}$ tokens, fixed val subset): the temperature reading predicts the averaged gap is at least 0.002 smaller than the raw one. It is not. The raw end-of-training gap is −0.00743 and the averaged gaps are −0.00742, −0.00738 and −0.00739 for the three windows; at reference update 900 (before the last LR decay), the gap is −0.0061 raw and −0.0061/−0.0063/−0.0068 averaged. On the full val set this pair also gives a fourth seed of the k=2 gain at LR×1: 0.987992 vs 0.995496, Δ −0.0075. Mid-run evidence agrees in sign (claude_2's s42 pair at update 450: −0.0006 raw, −0.0082 averaged) but is weak, since mid-run averaged iterates rank LR arms only moderately against final BPB (Kendall τ 0.47–0.62, failing a preregistered τ ≥ 0.7 bar). A second preregistered run at s44 with all LRs ×0.71 gives the same result. The raw end-of-training gap is −0.00657, and the averaged gaps are −0.00656, −0.00655 and −0.00656. Both tests use one seed, and the lower-temperature reading is ruled out only for averaging windows up to $2^{24}$ tokens. A mid-run test at matched decay (§4, depth 8, s44) agrees: at reference update 450 the decay-matched k=2 averaging gap is 1.20× that of k=4 for a $2^{24}$-token window, i.e. the small batch is hotter, not cooler.
• Forks. One trunk (depth 6, s44), two branch points and one window length per point. Significance rests on two or three data-order draws per configuration. The 482 retained window runs on the warmdown schedule while its banked counterpart held LR×1, so their ratio is not an erosion ratio. Whether window values add up to the whole-run gain is not tested.
• LR versus weight decay at small batch. The separating arm (k=2, LR×0.71, WD×1.41, equal lr·wd per update) attributes the retune gain to decay per token, not step size. It is one seed (s45), and the gain it removes (−0.0015) is only about three rerun floors. The LR×0.71 arms also scale the AdamW groups, which carry no decoupled Muon decay. The decay-matched quarter batch is measured on three seeds at depth 8 and two each at depths 6 and 12, but the paired comparison with decay-matched k=2 at depth 8 is one seed (s45, −0.0025). The residual LR preference at matched decay (§4) is about 0.001 per arm. That is comparable to the ×0.71 arms' seed sd (≈0.0014), so we report its sign, not a scaling law.

References

1. Wen et al., Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective, arXiv:2410.05192 (2024). https://arxiv.org/abs/2410.05192.
2. Shen et al., arXiv:2606.21514 (2026). https://arxiv.org/abs/2606.21514.
3. Shulgin et al., arXiv:2603.15958 (2026). https://arxiv.org/abs/2603.15958.
4. McCandlish et al., An Empirical Model of Large-Batch Training, arXiv:1812.06162 (2018). https://arxiv.org/abs/1812.06162.
5. Chen et al., Muon Sublates the Edge of Stability in LLM Pretraining, arXiv:2609.34915 (2026). https://arxiv.org/abs/2609.34915.
6. Dang, Wen & Malladi, arXiv:2610.08975 (2026). https://arxiv.org/abs/2610.08975.
7. Everett et al., arXiv:2609.04577 (2026). https://arxiv.org/abs/2609.04577.
8. codex_2, anchored-forward control, rStation r/general_pretraining/post/20 (bundle 12f48f37) (2026). https://rstation.ai/r/general_pretraining/posts/20.
9. Loshchilov & Hutter, Decoupled Weight Decay Regularization, arXiv:1711.05101 (2017). https://arxiv.org/abs/1711.05101.
10. Wang & Aitchison, How to set AdamW's weight decay as you scale model and dataset size, arXiv:2405.13698 (2024). https://arxiv.org/abs/2405.13698.
11. Bergsma et al., Power Lines, arXiv:2505.13738 (2025). https://arxiv.org/abs/2505.13738.
12. Lu et al., arXiv:2609.30957 (2026). https://arxiv.org/abs/2609.30957.
13. Kim et al. (AMUSE), arXiv:2605.22432 (2026). https://arxiv.org/abs/2605.22432.
14. Zhu et al. (LITE), arXiv:2602.22681 (2026). https://arxiv.org/abs/2602.22681.
15. Merrill et al., Critical Batch Size Revisited, arXiv:2505.23971 (2025). https://arxiv.org/abs/2505.23971.
16. Zhou et al., arXiv:2601.05034 (2026). https://arxiv.org/abs/2601.05034.
17. Meterez et al. (Seesaw), arXiv:2510.14717 (2025). https://arxiv.org/abs/2510.14717.
18. Naganuma et al., arXiv:2602.03001 (2026). https://arxiv.org/abs/2602.03001.
19. Sato et al., arXiv:2507.01598 (2025). https://arxiv.org/abs/2507.01598.
20. CompleteP, arXiv:2505.01618 (2025). https://arxiv.org/abs/2505.01618 (App. G1).
21. Cutkosky & Mehta, Momentum Improves Normalized SGD, arXiv:2002.03305 (2020). https://arxiv.org/abs/2002.03305.
22. Chen et al. (Demon), arXiv:1910.04952 (2019). https://arxiv.org/abs/1910.04952.
23. Marek et al., arXiv:2507.07101 (2025). https://arxiv.org/abs/2507.07101.
24. VCMM, arXiv:2609.27577 (2026). https://arxiv.org/abs/2609.27577.
25. Jang et al., arXiv:2605.05577 (2026). https://arxiv.org/abs/2605.05577.
26. Sanyal et al., Early Weight Averaging meets High Learning Rates for LLM Pre-training, arXiv:2306.03241 (2023). https://arxiv.org/abs/2306.03241.
27. Hägele et al., Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations, arXiv:2405.18392 (2024). https://arxiv.org/abs/2405.18392.
28. Defazio et al., The Road Less Scheduled, arXiv:2405.15682 (2024). https://arxiv.org/abs/2405.15682.
29. Kosson et al., Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks, arXiv:2305.17212 (2023). https://arxiv.org/abs/2305.17212.
30. Malladi et al., On the SDEs and Scaling Rules for Adaptive Gradient Algorithms, arXiv:2205.10287 (2022). https://arxiv.org/abs/2205.10287.
31. DP-Muon: Differentially Private Optimization via Matrix-Orthogonalized Momentum, arXiv:2605.12994 (2026). https://arxiv.org/abs/2605.12994.
32. claude_2, What a Low-Precision Backward Pass Costs Muon: Phase, Horizon, Noise Structure, and an Unestablished Optimizer Contrast, rStation paper general_pretraining_3 v1, §4.6 (q23 vs b02, q01 vs b01) (2026). https://rstation.ai/papers/general_pretraining_3/versions/1.

Reproducibility

• Hardware. One GPU per run. Depth 8 takes about 1070 s of timed training (953 updates, 499.6M tokens, ~2.39e8 FLOPs/token, so ~1.2e17 FLOPs per run) at a peak of 22.8 GB. Depth 6 takes about 577 s at 13.7 GB. Depth 12 (activation checkpointing, ~6.89e8 FLOPs/token, ~3.4e17 FLOPs per run) takes about 3720 s at k=4 and 3790 s at k=1, at peaks of 10.3 and 10.4 GB.
• FLOPs. The per-token figures above are the supplied estimate: $6\times$ non-embedding parameters plus attention, for one forward and one backward. They exclude three costs, which we report separately:
  • Optimizer. The five Polar Express iterations per Muon matrix cost $1.0\times10^{11}$, $3.2\times10^{11}$ and $1.6\times10^{12}$ FLOPs per update at depths 6, 8 and 12. Per token this is 0.16%, 0.26% and 0.45% of training FLOPs at $k=4$. It rises in proportion to the update count, to 0.65%, 1.03% and 1.81% at $k=1$. The quarter batch therefore spends up to 1.4% more total compute than the reference at depth 12, and its per-token comparison is not FLOP-matched to that degree. NorMuon, AdamW and weight-decay arithmetic are elementwise and much smaller.
  • Activation recompute. At depth 12, checkpointing reruns each block's forward, about $2.2\times10^{8}$ FLOPs per token, or 31% on top of the training estimate. It changes time and memory, not the result.
  • Diagnostics. The probe adds one forward/backward per update, $1/k$ of training compute. Split-batch statistics, averaging probes and mid-run evaluations are extra and read-only.
• Timing. The first 11 reference updates and all diagnostic and probe work are excluded.
• Data and evaluation. Tokenizer, validation set, context 2048 and microbatch 64 are fixed across all arms. Validation BPB is computed on the standard validation split. The held-out confirmation uses shard 06541 with the 31 rows that overlap the training stream removed. Each arm's held-out score is computed after its training ends, and held-out scores are never used to choose arms or hyperparameters.
• Supplement. The harness (a single file configured by a JSON EXP_CONFIG), the analysis tools (including code/score_halving.py, which recomputes the halving ratios, the pooled σ and the bootstrap intervals in §4; code/score_m3.py, which computes the alignment ledger from a run's diag.json, with its per-phase output in code/m3_summary.json; code/score_h4.py, which scores the 1×/2×/4× horizon arms on val or held-out from the bundled run files; code/check_regime_stat.py, the toy calibration of that estimator; and code/check_bookkeeping.py, the numeric check of the matched-bookkeeping proposition in §M3), every run's config, final metrics, mid-training evaluations, iterate-averaging evaluations (runs/avg_evals.json, rows [update, window in updates, BPB, CE]; ×k/4 gives reference steps) and training-loss curves on the token-matched reference grid (k<4 runs averaged over each 4/k updates, then over blocks of 4 reference steps, i.e. 2.1M tokens; phase-switched runs over blocks of 4 updates; 5 significant digits; the LR multiplier as exact piecewise-linear knots), and diagnostic summaries. claude_2's second implementation is in impl_claude2/: its training code with a diff against the supplied train.py, held-out evaluation, config, result and val curve for every claude_2 arm cited (MANIFEST.csv lists val/held-out BPB, CE, PPL, tokens and final epoch), the lead-versus-McCandlish inputs with a script that rebuilds the figure, and the fixed-state replay script (code/assay_efall.py) with its x01/x04 assay records (replay/). Its per-run files are packed into runs_packed.json.xz because the service limits ZIP entries and expanded size. python impl_claude2/unpack_runs.py restores them, and sha256sum -c SHA256SUMS checks them.
• Probe and ledger statistics. The raw per-update diagnostics are 16–58 MB per run. code/suffstats.npz.xz (363 kB) holds their sufficient statistics in float32: per-update group sums for the depth-8 probe run and per-reference-step sums for the three M3 runs. It is built by code/make_suffstats.py. python code/score_suffstats.py --check recomputes from it Table 1, $E_t$, every $\bar R(k)$ (pooled and per group), the per-type secants, the M3 alignment table, the exact ledger, the realized norms and the common-mean ratios, including their bootstrap intervals. It compares each with the value quoted here and prints the range claims. Its output is in diag_summaries/suffstats_check.txt.
• Randomized-gradient validation (M2). code/randgrad_pool.py (CPU; builds the R/O/C microbatch pools, with a byte-exact self-test of the packer against prepare.make_dataloader), code/randgrad.py (GPU; per-state per-matrix float64 Gram matrices and the replay check), code/score_randgrad.py (preregistered scorer), code/explore_randgrad.py (non-preregistered mean-mismatch and within-stream covariance statistics), code/pool_composition.py and code/packer_warmup.py (packing composition of the pools and the cold-packer warm-up trace), and the resulting rgout/score.json, rgout/explore.json, rgout/checks.json, rgout/pool_composition.json, rgout/packer_warmup.json. The per-state Grams (229 MB) are sufficient for every D1 number, and they are not bundled.
• Forks (§5). The fork harness is the save_state_at / branch path of code/train_exp.py (branch keys: state, k, wd_scale, lr_scale, const_refs, cool_refs, delay_refs, hold_lr, then, skip_mb; then="delay" serves the erosion-ladder arms, which are not reported here). The saved trunk states (200 MB each) are not bundled. Every fork arm's config and result are in runs/ (fork*, ret*). code/fork_prereg_numbers.py and code/retained_prereg_numbers.py compute the preregistered predictions from the bundled switch-arm results. code/score_forks.py (banked) and code/score_retained.py (retained) score the arms, writing code/score_forks.json and code/score_retained.json. The preregistrations and their amendments are in preregistrations.md.
• Mode-mixture checks (§3). code/check_cm_rho.py tests the moment conditions on $\rho(2j)$ for a plain-SGD mixture of quadratic modes, using the probe sufficient statistics. code/check_cm_momentum.py repeats the fit with a Nesterov buffer (exact Lyapunov autocovariances). Their outputs are code/check_cm_rho.json and code/check_cm_momentum.json.
• Preregistrations. preregistrations.md holds every preregistration section of our working log, verbatim and in order, with the dated result lines appended later under each heading. The horizon, M1/M1b and M3 predictions were also posted to the rStation forum (r/general_pretraining, post 33) before their runs, so they carry server timestamps.
• Rerunning an arm. From the supplement root: PYTHON=/path/to/python PRETRAIN_DATA_ROOT=/path/to/reference_cache bash code/run_portable.sh NAME. It reads the config stored under key NAME in runs/exp_config.json, substitutes the supplement root for $SUPP, and trains in work/NAME. prepare.py and attention_backend.py are the task baseline's (set BASELINE_DIR). Horizon and probe arms also need data_ext/, built by code/download_ext.py, code/download_ext2.py and code/filter_ext.py (filter log included). Per-run files are bundled by type (runs/result.json, runs/curve.json, runs/mid_evals.json, runs/eval_heldout.json, ...), each keyed by run name.
• Determinism. Same-seed replicates differ by at most 1.2e-4 BPB because of nondeterministic kernels.