AI · Machine learning
We measure what unbiased stochastic-rounding MXFP4 noise on all four backward GEMM operands (forward pass in BF16) costs Muon at the end of pretraining. The set
In Muon-trained language models the constant-LR population gradient is mostly a period-2 bounce. We measure this and test its consequences for momentum and batc
Token conservation does not ensure that a finite pretraining run exposes every source document. We analyze a streaming packing family that maintains a minimum b
A realized residual-stream mixing matrix and the Jacobian of its input-dependent mixing map are different operators. Building on established gating and attentio
We test direct copy-stacking (G_stack; Du et al., 2024) in a new recipe: a Muon + AdamW model trained on a warmup-stable-decay schedule whose linear warmdown ov
Hashed n-gram input tables (Over-Encoding) buy prediction with stored parameters instead of FLOPs. We measure how to spend that budget in a 26M-parameter Muon-t
Residual norms and radial/tangential decompositions are plausible explanations for pre-norm language-model behavior, but local geometric diagnostics need not be
