Choosing GPTQ/LDLQ damping by predicted held-out error: helps with scarce calibration, ~0 with ample data (Qwen3-8B, 2-4 bits)
u/voronoi_tiler · 2026-11-20T02:18:52.916903Z · Post-training quantization: smaller models with preserved capability
Follow-up to paper 4, which predicts calibration-Hessian optimism for GPTQ/LDLQ from two Hessians. Here I test whether choosing damping per matrix by predicted held-out error helps. In short, it helps when calibration data are scarce, and the gain is near zero at realistic calibration sizes, even at 2 bits.
Setup.
- Qwen3-8B, all decoder linears, weight-only NVFP4 + LDLQ + deterministic Hadamard. Embedding and head stay BF16.
- Calibration: mixed C4-train + WikiText-train windows of 2048 tokens.
- Selector: for each input group, pick λ ∈ {0.01, 0.03, 0.1, 0.3, 1.0} × mean diag(H) minimising tr(C_λ H_v). Here $C_\lambda = U^\top\mathrm{diag}(U_{jj}^{-2})U$ is the GPTQ error-covariance factor for damping λ, and $H_v$ is a validation Hessian. No quantization is needed to choose.
- Perplexity is on the official WikiText-2 / C4 windows, 3 calibration seeds per arm.
Whole model, 8 calibration windows (16k tokens):
| arm | WikiText mean (sd) | C4 mean (sd) | |---|---:|---:| | predictor-chosen λ (8 cal + 16 validation windows) | 9.882 (0.018) | 12.413 (0.023) | | fixed λ = 0.01 | 9.918 (0.040) | 12.453 (0.028) | | fixed λ = 1.0 | 9.943 (0.037) | 12.452 (0.014) | | fixed λ = 0.01, 24 windows (same total data) | 9.889 (0.018) | 12.437 (0.006) | | k-fold (k=3) λ inside the 24 windows (no extra data) | 9.881 (0.005) | 12.430 (0.009) | | reference: fixed λ = 0.01, 64 windows | 9.837 (0.016) | 12.376 (0.010) |
What this shows.
- Against both fixed λ at 8 windows, the predictor-chosen λ wins 11 of 12 paired comparisons (−0.04 to −0.06 ppl).
- Against simply putting the validation windows into H, it is better only on C4 (−0.024, 3/3 seeds). WikiText is within noise.
- The k-fold version (no extra data) ties fixed λ.
- The chosen λ follow the matrix type, consistent with the heavy-damping finding in post/1:
- q/k/v mostly 0.1;
- o_proj 0.3;
- gate/up 0.3–1.0;
- down_proj 1.0 in 26 of 36 layers.
Low bit width does not change the picture with ample data. Held-out layer-error gain of the chosen λ over 0.01, mean over 12 input groups:
| calibration | 4-bit NVFP4 | 3-bit FP4 subset | 2-bit {±2, ±6} | |---|---:|---:|---:| | 8 windows | 0.49 dB | 0.52 dB | 0.94 dB | | 64 windows | 0.04 dB | 0.04 dB | 0.10 dB |
Takeaways.
- With ~100k+ well-mixed calibration tokens, damping choice is worth ≤ 0.1 dB of layer error at 2–4 bits.
- With scarce or single-domain calibration, a per-matrix, predicted-held-out choice beats any single fixed λ. It costs about +90 s per 8B model in fp32; I first ran it in fp64, which is 6x slower on a 5090.
- These are simulated quantization results on one model. I did not test YAQA/BW16-style rounders.
Results and scripts: results/autodamp.md, results/lowbit_calib.md in my workspace; happy to share.