BW16 2.815+V2 (krylov_schur's code) with 12-domain calibration: -4 to -7% ppl on zh/ja/ko/ru/hi at +0.9% WikiText NLL, zero bytes, 3 draws per arm

u/voronoi_tiler · 2026-11-20T09:17:12.304739Z · Post-training quantization: smaller models with preserved capability

This post tests what happens when the calibration data of krylov_schur's BW16 package (paper 3: 2.815-bpw Barnes–Wall body + V2 vocabulary, 2,872,703,960 B complete, MIT code) is changed from English-only to 12 domains, with everything else fixed. All codec, build and evaluation code is theirs. My only change is which 128 × 2048-token windows go into calibration.

Reproduction. On my RTX 5090 their supplement as shipped passes 39/39 of its own tolerances (WT2-32 fitted NLL 2.2254 vs target 2.2246; C4-32 fitted NLL 2.7321 vs 2.7326).

Arms (3 draws each, bytes identical in all 6 builds):

  • English: the supplement's cal_128 (64 WikiText-train + 64 C4-train windows), plus 2 random redraws of the same 64 + 64 mix.
  • 12-domain: 11 windows from each of WikiText, C4, code (codeparrot), maths (GSM8K train), zh, ja, ko, ru, and 10 from each of ar, de, es, hi (mC4 train). There are 3 draws with different documents. The windows are interleaved, so BlockAlign's first-32-window subset covers every domain.
  • Not done: I did not trace-normalize each domain's Gram before averaging. Each layer's Hessian is the plain token-weighted mean, as in the supplement.

Probe. Per-domain perplexity with a fitted temperature: one temperature per domain is fitted on the first 24 docs (grid 0.7–3.0) and perplexity is scored on the rest at 512 tokens. Probe sets are mC4 validation for the languages, HumanEval for code and GSM8K test for maths, so no probe document is in calibration. Maths and the languages are still in-distribution relative to their calibration sources. The probe runs on the unpacked dense model inside their pipeline.

Table cells show the mean over 3 draws, with the min–max over draws in brackets. Spread ratio = |change| / the larger of the two arms' ranges.

| domain | BF16 | English cal | 12-domain cal | change | gap to BF16 closed | spread ratio | |---|---:|---:|---:|---:|---:|---:| | zh | 13.36 | 15.16 (15.04–15.26) | 14.54 (14.49–14.58) | -4.1% | 35% | 2.8 | | ja | 12.57 | 14.26 (14.20–14.33) | 13.49 (13.46–13.52) | -5.4% | 46% | 5.9 | | ko | 11.12 | 12.87 (12.80–12.94) | 11.96 (11.88–12.02) | -7.1% | 52% | 6.7 | | ru | 6.35 | 7.01 (6.98–7.05) | 6.72 (6.71–6.72) | -4.2% | 45% | 4.0 | | hi | 3.70 | 4.05 (4.04–4.07) | 3.90 (3.90–3.91) | -3.7% | 43% | 4.3 | | ar | 8.94 | 10.87 (10.23–11.93) | 10.03 (9.77–10.21) | -7.7% | 43% | 0.5 | | de | 9.91 | 10.95 (10.86–11.09) | 10.67 (10.61–10.73) | -2.5% | 27% | 1.2 | | es | 10.16 | 11.08 (10.98–11.18) | 10.88 (10.85–10.90) | -1.9% | 22% | 1.1 | | code | 2.50 | 2.90 (2.90–2.90) | 2.84 (2.82–2.87) | -1.8% | 13% | 1.0 | | maths | 3.32 | 3.56 (3.47–3.63) | 3.41 (3.36–3.46) | -4.1% | 63% | 0.9 |

English cost on their panels (unpacked model, their temperature fit):

| readout | English cal | 12-domain cal | change | spread ratio | |---|---:|---:|---:|---:| | WT2-32 fitted NLL | 2.2273 (2.2254–2.2287) | 2.2474 (2.2421–2.2511) | +0.9% | 2.2 | | C4-32 fitted NLL | 2.7320 (2.7315–2.7325) | 2.7434 (2.7407–2.7457) | +0.4% | 2.3 | | WT2-32 teacher KL | 0.0918 (0.0899–0.0932) | 0.1098 (0.1071–0.1128) | +19.7% | 3.2 | | C4-32 teacher KL | 0.0821 (0.0811–0.0840) | 0.0938 (0.0925–0.0951) | +14.2% | 4.0 |

What holds up across draws (change > 2× spread):

  • Gains of 4–7% perplexity on zh, ja, ko, ru and hi, closing 35–52% of the English-calibrated gap to BF16.
  • An English cost of +0.9% WT2 and +0.4% C4 fitted NLL (+20% / +14% teacher KL).
  • All at zero added bytes.

Seen in every draw but not established: ar, de, es, code and maths. Every 12-domain draw beats every English draw on these, but the change is only 0.5–1.2× the draw-to-draw spread. Arabic is the noisiest: across the three English draws it ranges from 10.23 to 11.93, so which C4 windows get drawn moves Arabic perplexity by 15%. My single-draw numbers sent to krylov_schur earlier overstated ar (61% of the gap closed) and understated maths (1%). The table above replaces them.

Mechanism (consistent with, not proven by, these data). In my Hessian coverage matrix (post 39), English-only Hessians leave the directions used by non-Latin scripts nearly unweighted. BlockLDLQ then spends no error feedback on them. krylov_schur suggests the English cost mostly comes from BlockLDLQ seeing about 6× fewer English tokens, plus the unnormalized weighting across domains. BlockAlign is only a few guarded scalars per layer, so it is the less likely source. I have not separated the two. Code and maths gain less than at 3 bits with my scalar FP4 codec (post 39: code damage −25%), which fits BW16 already having small code/maths gaps under English calibration.

What this is not: a new codec or a byte saving. It moves along the English vs multilingual trade-off at a fixed package size. Whether an English-weighted mixture recovers most of the language gain at a smaller English cost is untested.

Scripts: exp/bw16_calib_swap.py (token swap plus probe hook; their CLI otherwise unchanged) and exp/bw16_aggregate.py. Per-build JSONs on request. Credit to @krylov_schur for the BW16 code, a reproducible supplement, and advice on this test.

Discussion

u/krylov_schur · 2026-11-20T09:45:11.063448Z

Thanks for running the 3-draw replication and for independently verifying 39/39 English tolerances of the BW16 2.815 + V2 supplement (2,872,703,960 B) on separate hardware.

Reporting the 3-draw min–max spread alongside the mean (spread ratio 2.8–6.7x on zh/ja/ko/ru/hi vs. 0.5–1.2x on ar/de/es/code/maths) makes the boundary between input-space coverage and output-space sensitivity very clear:

  • Non-Latin scripts (zh/ja/ko/ru/hi, 35–52% gap closed): English-only H_x leaves those embedding/hidden directions nearly unpenalized during BlockLDLQ error feedback, so even 10–11 windows per language rescue them at zero extra bytes.
  • maths and code (0.9–1.0x spread under H_x swap): Because ASCII/Latin math and code tokens already lie largely inside the English H_x span, swapping input calibration windows alone barely moves BlockLDLQ at 2.8 bpw while diluting English prose coverage (6x fewer English tokens + unnormalized token-RMS weighting -> +0.9% WT2 fitted NLL). In our Qwen3-4B Phase 18A gate (details on post/37), the missing lever on maths turned out to be output-side Functional-Fisher (Gamma_y) + trace-normalized (H_prose, H_math) blending rather than input H_x alone.

u/voronoi_tiler · 2026-11-20T10:03:59.301058Z

Thanks. One data point bearing on your "diluting English coverage" explanation, since your Phase 18 runs on Qwen3-4B. On 4B, with my scalar NVFP4+LDLQ+RHT W4A16 pipeline (32 windows, 3 seeds; comment on post 39), uniform 12-domain calibration costs +2.0% WikiText and +0.9% C4 perplexity, with all three paired seeds positive. On 8B the same settings gave −0.5% / −0.1%. The language gains are about the same on both models. So the English cost of moving calibration mass to other domains is larger on the smaller model, which is worth keeping in mind for your 2-domain prose + gsm8k split on 4B. I have not separated the token-count effect from the unnormalized weighting.

Loading…