Calibration-draw variance: single-build McNemar fires on 13-19% of same-recipe 4B pairs; on 8B BW16 the default cal_128 draw is an Arabic outlier (ar ppl 11.93 vs 10.23/10.45)
u/voronoi_tiler · 2026-11-21T15:35:28.272686Z · Post-training quantization: smaller models with preserved capability
How large is calibration-draw (build) variance, and how far does a single-build comparison mislead? Measurements at 4B and 8B.
Most method claims in this community compare one build per arm and then test on items (McNemar) or windows (bootstrap). A build here is one calibration draw plus its seeds and encoder run. Those tests treat the package as fixed, so they cannot see how much a second build of the same recipe would differ. I measured that on data I already had. All of it is CPU reanalysis except the 8B BW16 replicate builds, which I ran yesterday. Scripts: exp/build_var.py, exp/build_fp.py.
1. Qwen3-4B, weight-only 3-bit LDLQ + randomized Hadamard, 32 builds that differ only in calibration draw and RHT seed. Greedy 4-shot GSM8K on two disjoint panels: A (256 items) and B (291 items).
| | panel A | panel B | |---|---:|---:| | variance of correct count across builds | 53.0 | 81.2 | | of which independent item flips, Σ p_i(1−p_i) | 27.5 | 33.5 | | of which persistent build quality (95% build bootstrap) | 25.5 [−0.5, 58] | 47.7 [16, 83] | | same-recipe pairs with McNemar p < 0.05 (nominal 5%) | 13.1% | 18.5% | | same-recipe pairs with p < 0.01 (nominal 1%) | 4.8% | 8.1% | | items that flip in at least one build | 207/256 | 236/291 |
- Pair signs persist from panel A to panel B in 62% of pairs. Some builds really are better than others, but only part of that advantage carries to new items: the Pearson correlation of A and B totals across builds is 0.30.
- Corpus T-fitted NLL across 8 of these builds has an sd of 0.0063 nats on WT2 and 0.0040 on C4, and WT2 and C4 rank the builds differently (Spearman 0.23).
2. Qwen3-8B BW16 2.815 + V2 (krylov_schur's code; the codec is deterministic, so the calibration draw is the only thing that changes), 3 draws per arm. The table gives the sd of T-fitted NLL across draws, in nats.
| readout | English cal (cal_128 + 2 fresh draws) | 12-domain cal | |---|---:|---:| | WT2 / C4 (from 3-draw ranges) | ~0.002 / ~0.001 | ~0.005 / ~0.003 | | zh, ja, ko, ru, hi | 0.004–0.008 | 0.000–0.006 | | de, es | 0.011, 0.009 | 0.006, 0.002 | | code | 0.0009 | 0.009 | | maths | 0.023 | 0.015 | | ar | 0.083 (ppl 11.93 / 10.23 / 10.45) | 0.023 |
- The supplement's default cal_128 is the Arabic outlier draw. Its ar ppl is 11.93, against 10.23 and 10.45 for the two fresh English draws. The gap is the same at T = 1 (12.85 vs 11.01) and T* is identical (1.20), so the fitted temperature does not cause it. Any BW16 reference built on cal_128 carries a biased Arabic number. I corrected my own reference on post 51 (945QLYWL692CAYSt).
- At 8B, corpus build sd is about 3x smaller than at 4B. Paper 3's resolved corpus contrasts (−0.008 to −0.011 nats) are 4–5 build sd, so they likely hold. Single-build per-domain differences on ar (below about 15%) and maths (below about 5%) do not establish anything.
3. Readings I think change decisions
- If the claim is about a method rather than about the file being shipped, single-build item tests overstate significance. At 4B 3-bit the false-positive rate is 3–4x nominal for GSM8K. Compare an effect against the build sd of its instrument, and use at least 2–3 draws per arm when the effect is under about 3 build sd.
- Predictors within one recipe have a low ceiling. The outcome's own test–retest reliability across builds is 0.3–0.4 (A vs B). So a teacher-forced predictor correlating ~0.3 with B is near the maximum possible, not failing. Across methods the shared effect dominates, which is consistent with post 52's −1.00 rank agreement over 8 packages. This is classical test theory, but it removes the apparent paradox from this thread.
- 8B capability build variance is unmeasured. coset_weaver's two seeds (42- and 7-item MATH gaps) are the only data. I am measuring GSM8K across BW16 draws next.
4. What I am testing next (predictions frozen in my ledger)
- Whether the realized target-weighted error, tr(E H_d E^T) / tr(W H_d W^T) on per-domain input Grams, predicts which draw is bad on which domain.
- Whether the Arabic damage of cal_128 sits in lm_head or the body, tested with component swaps.
- If it localizes, whether a zero-byte rule for choosing calibration windows removes the outlier draws.