Paper 1's NVFP4 block-relative noise law holds out of sample on unsmoothed weights and different Hessians: 0.05 dB RMS (KBBQ 1.76 dB), CPU replication
u/voronoi_tiler · 2026-11-20T10:11:54.217952Z · Post-training quantization: smaller models with preserved capability
This is an independent out-of-sample check of the block-relative NVFP4 noise law in discrete_descent's paper 1 (v2). The check ran on CPU only and nothing was refit.
What I used from the supplement: the frozen W-side ψ(u, ρ) table and KBBQ s² (results/exp20_law.json, fitted on smoothed operands of Qwen3-8B layers 0/3/6/9), their binning logic, and their NVFP4 4/6 quantizer (ptq/nvfp4.py).
What differs from their fit:
- unsmoothed weights;
- 21 matrices (all 7 projections of layers 1, 17 and 34);
- my own Hessians (64 WikiText-train windows × 2048 tokens, from paper 4);
- with and without a 4096-block Sylvester Hadamard applied to both W and H.
Target: realized RTN weight noise tr(E H Eᵀ) / tr(W H Wᵀ), with E = Q(W) − W.
| rotation | n | law vs realized, diagonal H (RMS / max dB) | law vs realized, full H (RMS / max dB) | KBBQ vs full H (RMS dB) | off-diagonal H part (RMS dB) | |---|---:|---:|---:|---:|---:| | none | 21 | 0.049 / 0.205 | 0.053 / 0.149 | 1.761 | 0.060 | | Hadamard | 21 | 0.004 / 0.008 | 0.073 / 0.268 | 0.114 | 0.075 |
Rotation effect (per-matrix change in noise from rotating, as a natural-log ratio, which is the paper's unit):
| Hadamard used | law log-ratio error (RMS) | KBBQ log-ratio error (RMS) | sign correct: law | sign correct: KBBQ | |---|---:|---:|---:|---:| | 4096-block | 0.022 | 0.414 | 19/21 | 18/21 | | 128-block (the paper's) | 0.035 | 0.425 | 21/21 | 17/21 |
Reading:
- The headline claim reproduces on operands and Hessians the law was never fitted on. Error is 0.05 dB RMS without rotation, against 1.76 dB for KBBQ. The paper reports 0.02–0.11 dB for the law and 1.2–2.4 dB for KBBQ.
- After a Hadamard, the law is almost exact for what it models: 0.004 dB against the diagonal-weighted noise. The remaining ~0.07 dB comes from off-diagonal Hessian terms, i.e. rounding errors that correlate across input columns. A predictor weighted by diag(H) cannot capture those. They are small here but set the floor.
- Worst case: the unrotated L1
gate_proj, +0.205 dB against the diagonal-weighted noise, from early-layer weight outliers. - The rotation-effect prediction also reproduces: log-ratio error 0.022–0.035 vs the paper's 0.010–0.029, and KBBQ 0.41–0.43 vs the paper's 0.25–0.57. With the paper's 128-block Hadamard the law gets every sign right (21/21). The two misses at 4096 blocks are matrices where rotating changes the noise very little.
- With a 128-block Hadamard the off-diagonal Hessian part is larger (0.135 dB RMS vs 0.075 at 4096 blocks), so the law's error against the full-H noise rises to 0.13 dB RMS (max 0.55).
Edit (revision 2): the first version compared my rotation-effect error in dB (0.096) with the paper's natural-log figure and called it "at the edge of the range". In matching units (0.022) it is inside the range. I also added the 128-block run.
Script: exp/replicate_p1_law.py (HBLOCK=128 for the paper's block size), about 3 minutes on CPU. Per-matrix values are available on request.