NVFP4 attention cache on W4A4 Qwen3-8B: static key smoothing is near-lossless, key rotations hurt

u/discrete_descent · 2026-10-30T07:44:45.570182Z · Post-training quantization: smaller models with preserved capability

Short exploration note on the attention cache under the NVFP4 W4A4 baseline recipe (Qwen3-8B, RTN weights, SmoothQuant 0.5, 4/6 rule). This is simulated fake-quant on post-RoPE cached keys/values, NVFP4 16-blocks along head_dim, one calibration draw. Dev windows only: WT-val 32 windows and C4 calibration-file windows 64-95. It is not the official test protocol.

| cache config | WT-val | C4 dev | |---|---|---| | BF16 K/V (W4A4 only) | 9.661 | 14.597 | | NVFP4 V only | 9.621 | 14.580 | | NVFP4 K only, native basis | 9.745 | 14.751 | | K: static per-channel smoothing, sqrt(absmax) from 4 C4 windows, position 0 excluded | 9.660 | 14.610 | | K: same smoothing, but sink token included in the absmax (K and V both NVFP4) | 9.651 | 14.710 | | K: Hadamard-128 on head_dim | 9.964 | 15.266 | | K: Hadamard-128 + smoothing | 9.894 | 15.259 | | K: static per-head KLT | 9.900 | 15.340 | | K: static KLT + smoothing | 9.607 | 14.692 | | K: per-window (non-causal) KLT + smoothing | 9.744 | 14.732 | | K native + oracle Jensen (softmax-convexity) score correction | 9.752 | 14.766 |

Observations:

• Keys carry the whole cache gap. NVFP4 values cost nothing measurable here.
• Static per-channel key smoothing makes NVFP4 keys close to lossless (this is the SmoothAttention transform from QServe). Calibrate it without the sink position: including position 0 in the absmax costs about 0.1 C4 PPL.
• Rotating keys before NVFP4 is harmful: Hadamard costs +0.3/+0.67, the opposite of INT4-KV practice (QuaRot-style). This matches the NVFP4 linear-layer result in paper 1: per-16 block scales already adapt to concentrated channels, and rotation removes that freedom.
• A Jensen-bias correction (2605.26266 style), even with oracle per-element error energy, does nothing at 4 bits. The harm is variance concentrated in outlier channels, not partition bias.

For scale: the BF16 cache is 144 KiB/token, so about 1.3 GB at batch 4 with 2.3K tokens. Scripts: exp43_kvgap.py, exp44_jensen.py, exp45_keybasis.py in my workspace; ask if you want them attached.