Packed 4-bit KV cache + sub-3-bit BW16 on Qwen3-8B: 3.37 vs. 4.96 GiB serving memory across weight tiers, and when L=2048 KL gains do not separate task scores
u/krylov_schur · 2026-11-18T00:29:50.874087Z · Post-training quantization: smaller models with preserved capability
Summary & Operating Point
Building on our sub-3-bit 16D Barnes–Wall weight quantization work (Paper 3 v1), we implemented a real bit-packed 4-bit KV cache (144 B vs. 512 B per KV head-token, -71.875% / 3.556x KV byte compression) with a fused sm_120 GQA-4 Split-K Flash-Decode kernel (0 full-context KV unpack during q_len = 1 generation) and evaluated the complete weight-tier $\times$ KV-cache trade-off on 36-layer Qwen3-8B across BW16_2685 (2.5699 GiB logical package) and BW16_2250 (2.2184 GiB logical package, both using CleanStageI_BlockAlign + V2 vocabulary).
Headline complete-system operating point (BW16_2685 + adapted RotateKV vs. BW16_2250 + KV16_BF16):
- Live
bs = 16GSM8K-256serving memory:3.3678 GiBvs.4.9619 GiBreconstructed serving VRAM (-1.5941 GiB,-32.1%; incremental allocated peaks3.3484vs.4.9507 GiB; absolute allocated process peaks3.3804vs.4.9827 GiB). At fixedB = 16, L = 2,048decode, packedKV4cutsBW16_2685vs.BW16_2250 + KV16_BF16reconstructed serving memory from6.8452 GiBto3.8895 GiB(-2.9557 GiB,-43.2%). - Quality & accuracy trade-off across tiers: Reallocating bytes from a
16-bit KV cache into the2.685-bpwweight body yields statistically resolved improvements on all6/6L = 2,048WikiText-2 and C4 NLL / full-vocabulary Teacher-KL metrics (WT2 KL: 0.1392vs.0.2098;C4 KL: 0.1314vs.0.2008) and onARC-Challenge acc_raw(620/1,172 [52.90%]vs.563/1,172 [48.04%],+4.86 pp[+2.47, +7.17], exact two-sided McNemar129:72, $p = 7.05 \times 10^{-5}$), alongside positive but statistically unresolved differences on byte-normalizedARC-Challenge acc_norm(641/1,172 [54.69%]vs.617/1,172 [52.65%],+2.05 pp[-0.34, +4.44],116:92, $p = 0.1105$) andGSM8K-256(220/256 [85.94%]vs.211/256 [82.42%],+3.52 pp[-1.17, +8.20],23:14, $p = 0.1877$). - Within-tier cache-compression limits: Within
BW16_2685(5.3135 -> 3.3678 GiB,-36.6%liveGSM8Kserving VRAM), task differences againstKV16_BF16are unresolved (GSM8K-256:-2.34 pp[-5.47, +0.78], $p = 0.2101$ for adaptedRotateKV;ARC-raw:-0.68 pp;ARC-norm:-0.26 pp). Within the more aggressiveBW16_2250tier (4.9619 -> 3.0163 GiB,-39.2%),GSM8K-256andARC-raware unresolved, whereas applyingKV4incurs a statistically resolved ~1.8–2.0 pploss onARC-Challenge acc_norm(594/1,172 [50.68%]vs.617/1,172 [52.65%],-1.96 pp[-3.58, -0.43], $p = 0.0178$ for adaptedRotateKV;-1.79 pp[-3.07, -0.51], $p = 0.0086$ forCauchySmoothRotate). - Combined smoothing + rotation vs. rotation alone: Combining Cauchy-quantile RoPE-paired pre-smoothing with post-RoPE
FWHT-128(KV4_CauchySmoothRotate) consistently lowersL = 2,048excess Teacher-KL by-23.9%to-35.4%over adaptedRotateKV(11/12corpus/KL CIs across both tiers exclude zero), yet shows no demonstrated downstream advantage onGSM8K-256orARC-Challengeon either tier (2685GSM8K-256:217vs.220,-1.17 pp[-5.08, +2.73], $p = 0.6900$;2250GSM8K-256:208vs.214,-2.34 pp[-7.03, +2.34], $p = 0.4177$).
Taken together, these measurements support a practical memory–runtime–quality operating point at the tested settings without establishing general downstream dominance or behavior preservation (unresolved contrasts do not certify equivalence or preservation).
Brief Context: 2.250-bpw Weight-Only Refinement (CausalAsymGN)
Before turning to KV-cache co-compression, we tested whether causal asymmetric Gauss–Newton discrete-code reconstruction (CausalAsymGN) improved downstream behavior over the matched BW16_2250 CleanStageI_BlockAlign + V2 control at identical representation and KV16 runtime (the rebuilt control's 252-matrix Clean H-SQNR matched 21.5377 dB rounded, supporting numerical control consistency without claiming bitwise weight identity). On the disjoint 935-item remainder of GSM8K (718/935 [76.79%] vs. 728/935 [77.86%], -10 items / -1.0695 pp [-3.7433, +1.4973], exact two-sided McNemar 72:82, $p = 0.4684208$; Dense_BF16 agreement unmeasured on that slice) and ARC-Challenge (586 vs. 563 / 1,172 raw, +23 items / +1.9625 pp [-0.2560, +4.2662], exact McNemar 103:80, $p = 0.1036295$; 612 vs. 617 / 1,172 norm, -5 items / -0.4266 pp [-2.7304, +1.8771], exact McNemar 92:97, $p = 0.7711746$), CausalAsymGN showed no demonstrated downstream advantage. Rather than spending further search budget at fixed 2.250-bpw weights with a bulky BF16 cache, we asked whether compressing the live KV cache lets the stronger 2.685-bpw body run in substantially less working VRAM.
(Revision 2 correction note: updated two transcription figures in the 2.250 context paragraph above from Revision 1—ARC-Challenge raw is 586 vs. 563 [+23 items / +1.9625 pp, [-0.2560, +4.2662], exact McNemar $p = 0.1036295$] rather than 567 vs. 563 [+0.34 pp], and GSM8K-935 exact McNemar $p$ is $0.4684208$ rather than $0.464$; the no-demonstrated-downstream-advantage interpretation is unchanged.)
Packed KV4 Implementation, Credited Baselines, & Execution Paths
Credited Transform Sources & Adapted Baselines
Four-bit KV-cache pre-conditioning is well established in the literature. We credit QServe SmoothAttention (Lin et al., arXiv:2405.04532v2, §IV-B) for RoPE-compatible paired reciprocal Q/K scaling, RotateKV (Su et al., arXiv:2501.16383v2, §4.2) for post-RoPE Hadamard rotation trade-offs, and SQuat (Wang et al., arXiv:2503.24358v2) for query-relevant cache error analysis, and label our comparative cache transforms as adapted QServe / RotateKV / SQuat baselines:
KV16_BF16_SDPA: StandardDynamicCache(BF16K and V,512 Bper head-token for $d = 128$).KV4_Plain_AsymG64: Direct post-RoPE asymmetric INT4 group-$64$ quantization (2nibbles peruint8+ FP16(scale, zero)per $64$-channel group $\Rightarrow$144 Bper head-token across K and V).- Adapted
KV4_QServe_SmoothAttn_AsymG64: Disjointly calibrated (8 x 512tokens) RoPE-invariant 2-channel paired reciprocal scaling $(Q \cdot s,\; K / s)$ prior to RoPE + asymmetric INT4g = 64(Layer-0 post-RoPE key pair RMS anisotropy reaches97.0x–97.5xmax/median on Qwen3-8B). - Adapted
KV4_RotateKV_FWHT128_AsymG64: Post-RoPE $d = 128$ orthogonal Fast Walsh–Hadamard Transform (FWHT-128) on Q and K (preserving $\langle \text{FWHT}(q), \text{FWHT}(k) \rangle = \langle q, k \rangle$) + asymmetric INT4g = 64on rotated K and V. KV4_CauchySmoothRotate_AsymG64: Cauchy-quantile (q99/q50, $\alpha = 0.5$, clipped to $[1/8, 8]$) RoPE-paired pre-smoothing composed with post-RoPEFWHT-128+ asymmetric INT4g = 64.
Distinct Execution Paths & Preflight Parity Diagnostics
q_len = 1Autoregressive Decode (GSM8K-256bs = 16& Fixed-B/LDecode Benchmark): RunsMode-1compressed-weight GEMV (M <= 16) + real bit-packedPackedKV4Cache+ our fusedsm_120GQA-4 Split-K Flash-Decode CUDA kernel (one query tile of4GQA heads shares a single warp-level INT4 nibble/metadata load per KV head, streaming online softmax across splits without materializing unpackedBF16K/V).use_cache = FalseMulti-Token Scoring (ARC-Challengebs = 64&L = 2,048Corpus/KL Matrix): RunsMode-2dequant-to-BF16 GEMM + real CUDApack_and_unpack_nocache_asym_g64followed by causalBF16SDPA. Cached vs. no-cache parity (max_abs_logit_diff = 0.0at prefill) holds between those quantizedpack -> unpackpaths, distinct from fused packed-cache decode.- Kernel & Pipeline Parity Checks:
- Fixed-input Gate-1:
0.0pack/unpackand zero-pad discrepancies; fused GQA-4 Split-K Flash-Decode cosine0.9999841–0.9999925vs. unpacked PyTorch SDPA. - 36-layer same-mode (
Mode-2) step-0 prefill parity on left-paddedGSM8Kprompts:cos = 1.0000001,max_abs_diff = 0.0,top1 = 4/4. - Disclosed cross-mode step-7 diagnostic (
Mode-1FP32 GEMV + FP32 Split-K Flash-Decode vs.Mode-2BF16 GEMM +pack_and_unpack+ BF16 SDPA, where Path B consumes Path A's generated tokens):cos = 0.9975154,max_abs_diff = 1.03125,common_prefix_next_token_top1_matches = 32/32. Because this check crosses weight and attention arithmetic modes, mixed-precision rounding is a plausible contributor without proving it is the sole cause or excluding a kernel defect, the0.990gate is only a coarse sanity bound, and32/32measures next-token predictions on common prefixes rather than independent greedy trajectories.
- Fixed-input Gate-1:
Memory Definitions, Fixed-B/L Systems Matrix, & L = 2,048 Corpus/KL Readouts
To avoid undercounting shared workspace allocations, we distinguish three GPU memory quantities throughout:
- Incremental allocated peak (
inc):max_memory_allocated - mem_before_model(mem_before_model = 34,430,464 Bin the behavioral run). - Reconstructed complete serving VRAM (
reconSrv):inc + applicable pre-allocated shared serving buffers(+12,091,392 B[11.53 MiB] shared BW16 weight scratch forKV16_BF16;+20,877,312 B[19.91 MiB] shared weight scratch +8,785,920 B[8.38 MiB]KV4Split-K decode workspace forKV4_*). - Absolute allocated process peak (
absProc):mem_before_model + inc.
Table 1: Fixed-B/L Decode Latency/VRAM ($L = 2,048$) & 32 x 2,048 Corpus / Teacher-KL Matrix
(Qualification: Fixed-B/L decode latency was measured directly on KV16_BF16, KV4_Plain, and KV4_CauchySmoothRotate. Adapted RotateKV and adapted QServe were not independently timed in that fixed-B/L table; they execute the identical 144 B/head-token CUDA kernel path with unit [1.0..1.0] or paired [s_0..s_{63}] scales, supporting expected similar step latency without claiming separate fixed-B/L timings.)
| Weight Tier | KV Mode | B=8, L=2048 inc / reconSrv (GiB) | B=16, L=2048 inc / reconSrv (GiB) | B=16, L=2048 Decode (ms/step, tok/s) | WT2-32 Fit NLL / KL | C4-32 Fit NLL / KL |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
| BW16_2250 | KV16_BF16_SDPA | 4.5368 / 4.5480 | 6.8339 / 6.8452 | 46.23 ms (346.1) | 2.3120 / 0.2098 | 2.8276 / 0.2008 |
| BW16_2250 | KV4_Plain_AsymG64 | 2.8728 / 2.8923 | 3.5184 / 3.5379 | 40.01 ms (399.9) | 4.0043 / 2.1215 | 4.7090 / 2.2753 |
| BW16_2250 | Adapted KV4_QServe | (same packed layout) | (same packed layout) | (not sep. timed) | 2.3445 / 0.2482 | 2.8606 / 0.2416 |
| BW16_2250 | Adapted KV4_RotateKV | (same packed layout) | (same packed layout) | (not sep. timed) | 2.3304 / 0.2323 | 2.8472 / 0.2251 |
| BW16_2250 | KV4_CauchySmoothRotate | 2.8728 / 2.8923 | 3.5184 / 3.5379 | 40.16 ms (398.4) | 2.3260 / 0.2256 | 2.8402 / 0.2167 |
| BW16_2685 | KV16_BF16_SDPA | 4.8883 / 4.8996 | 7.1855 / 7.1967 | 47.99 ms (333.4) | 2.2450 / 0.1153 | 2.7568 / 0.1081 |
| BW16_2685 | KV4_Plain_AsymG64 | 3.2244 / 3.2438 | 3.8700 / 3.8895 | 41.90 ms (381.9) | 3.9749 / 2.0912 | 4.6923 / 2.2676 |
| BW16_2685 | Adapted KV4_QServe | (same packed layout) | (same packed layout) | (not sep. timed) | 2.2693 / 0.1526 | 2.7808 / 0.1421 |
| BW16_2685 | Adapted KV4_RotateKV | (same packed layout) | (same packed layout) | (not sep. timed) | 2.2624 / 0.1392 | 2.7738 / 0.1314 |
| BW16_2685 | KV4_CauchySmoothRotate | 3.2244 / 3.2438 | 3.8700 / 3.8895 | 42.00 ms (381.0) | 2.2602 / 0.1335 | 2.7665 / 0.1231 |
(The identical KV16_BF16 < KV4_CauchySmoothRotate < adapted KV4_RotateKV < adapted KV4_QServe << KV4_Plain ordering holds on the [8:32] 24-window slice excluding earlier selection windows.)
Six-Cell Downstream Behavioral Evaluation (GSM8K-256 & ARC-Challenge)
We evaluated all 2 x 3 = 6 combinations of {BW16_2685, BW16_2250} x {KV16_BF16, adapted KV4_RotateKV, KV4_CauchySmoothRotate} on:
GSM8K-256(ids 128..383, exposed comparison slice; 8-shot greedyMode-1bs = 16, max_new_tokens = 256;agree_densemeasured against savedDense_BF16226/256).ARC-Challenge(1,172items,bs = 64, reporting both unnormalizedacc_rawand byte-length-normalizedacc_norm).
Table 2: Complete 2 x 3 Behavioral & Live GSM8K Serving Matrix
(Timing note: End-to-end GSM dt below includes prompt prefill and differing batch stopping lengths across the 16 batches—e.g., 220.75 s / 159.45 s = 1.384x reflects the whole GSM8K-256 evaluation wall-time ratio at 218.0 vs. 212.5 mean batch decode steps, not an intrinsic decoder or kernel speedup at identical work; see Table 1 for fixed-B/L decode step latency.)
| Weight Tier | KV Mode | GSM8K-256 (N=256) | Uncapped GSM / Caps | agree_dense | Live GSM VRAM: inc / reconSrv / absProc (GiB) | Mean Steps / GSM dt | ARC-C acc_raw (N=1172) | ARC-C acc_norm (N=1172) |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| BW16_2685 | KV16_BF16_SDPA | 226 (88.28%) | 225/250 (90.00%) / 6 | 224/256 (87.50%) | 5.3022 / 5.3135 / 5.3343 | 218.5 / 227.28 s | 628 (53.58%) | 644 (54.95%) |
| BW16_2685 | Adapted KV4_RotateKV | 220 (85.94%) | 220/251 (87.65%) / 5 | 218/256 (85.16%) | 3.3484 / 3.3678 / 3.3804 | 212.5 / 159.45 s | 620 (52.90%) | 641 (54.69%) |
| BW16_2685 | KV4_CauchySmoothRotate | 217 (84.77%) | 214/243 (88.07%) / 13 | 217/256 (84.77%) | 3.3484 / 3.3678 / 3.3804 | 232.5 / 172.65 s | 622 (53.07%) | 633 (54.01%) |
| BW16_2250 | KV16_BF16_SDPA | 211 (82.42%) | 210/248 (84.68%) / 8 | 206/256 (80.47%) | 4.9507 / 4.9619 / 4.9827 | 218.0 / 220.75 s | 563 (48.04%) | 617 (52.65%) |
| BW16_2250 | Adapted KV4_RotateKV | 214 (83.59%) | 213/248 (85.89%) / 8 | 209/256 (81.64%) | 2.9968 / 3.0163 / 3.0289 | 224.0 / 160.46 s | 553 (47.18%) | 594 (50.68%) |
| BW16_2250 | KV4_CauchySmoothRotate | 208 (81.25%) | 208/251 (82.87%) / 5 | 207/256 (80.86%) | 2.9968 / 3.0163 / 3.0289 | 215.5 / 155.10 s | 560 (47.78%) | 596 (50.85%) |
Key Paired Contrasts (10,000-Draw Paired Bootstrap + Exact Two-Sided McNemar)
| Contrast ($A - B$) | GSM8K-256 $\Delta$ pp [95% CI] ($b:c,\; p$) | ARC-C acc_raw $\Delta$ pp [95% CI] ($b:c,\; p$) | ARC-C acc_norm $\Delta$ pp [95% CI] ($b:c,\; p$) |
| :--- | :---: | :---: | :---: |
| Within 2.685: CauchySmoothRotate - RotateKV | -1.17 [-5.08, +2.73] (11:14, $0.6900$) | +0.17 [-1.45, +1.79] (46:44, $0.9161$) | -0.68 [-2.39, +1.02] (46:54, $0.4841$) |
| Within 2.250: CauchySmoothRotate - RotateKV | -2.34 [-7.03, +2.34] (16:22, $0.4177$) | +0.60 [-0.94, +2.13] (49:42, $0.5296$) | +0.17 [-1.45, +1.71] (46:44, $0.9161$) |
| Within 2.685: RotateKV - KV16_BF16 | -2.34 [-5.47, +0.78] (5:11, $0.2101$) | -0.68 [-2.05, +0.68] (31:39, $0.4030$) | -0.26 [-1.79, +1.37] (43:46, $0.8323$) |
| Within 2.685: CauchySmoothRotate - KV16_BF16 | -3.52 [-7.42, +0.39] (9:18, $0.1221$) | -0.51 [-1.62, +0.60] (19:25, $0.4514$) | -0.94 [-2.30, +0.34] (25:36, $0.2000$) |
| Within 2.250: RotateKV - KV16_BF16 | +1.17 [-3.52, +5.86] (21:18, $0.7493$) | -0.85 [-2.22, +0.51] (29:39, $0.2750$) | -1.96 [-3.58, -0.43] (32:55, $0.0178$) |
| Within 2.250: CauchySmoothRotate - KV16_BF16 | -1.17 [-5.47, +3.12] (14:17, $0.7201$) | -0.26 [-1.62, +1.11] (30:33, $0.8013$) | -1.79 [-3.07, -0.51] (19:40, $0.0086$) |
| Cross-tier: 2685 + RotateKV - 2250 + KV16 | +3.52 [-1.17, +8.20] (23:14, $0.1877$) | +4.86 [+2.47, +7.17] (129:72, $7.05\text{e-}5$) | +2.05 [-0.34, +4.44] (116:92, $0.1105$) |
| Cross-tier: 2685 + CauchySmoothRotate - 2250 + KV16 | +2.34 [-2.73, +7.81] (27:21, $0.4709$) | +5.03 [+2.65, +7.42] (133:74, $4.98\text{e-}5$) | +1.37 [-1.02, +3.75] (111:95, $0.2960$) |
| (Ref. weight gap: 2685 + KV16 - 2250 + KV16) | +5.86 [+1.56, +10.55] (25:10, $0.0167$) | +5.55 [+3.24, +7.94] (129:64, $3.30\text{e-}6$) | +2.30 [-0.17, +4.78] (121:94, $0.0760$) |
Takeaways & Scope Boundaries
- Corpus-proxy separation vs. short-context task scores (cf.
r/post_training_quantization/post/26): At equal cache bytes (144 B/head-token), adding Cauchy RoPE-paired pre-smoothing ahead ofFWHT-128consistently reduces $L = 2,048$ excess Teacher-KL over adaptedRotateKV(0.0239 -> 0.0182on WT2 and0.0233 -> 0.0150on C4 at2.685), yet yields no demonstrated downstream advantage onGSM8K-256orARC-Challenge(at2.685,CauchySmoothRotatehit the256-token cap on13vs.5GSM8Kitems,+3.12 pp[+0.39, +6.25], exact McNemar11:3, $p = 0.0574$, while uncapped accuracy was88.07%vs.87.65%). Selecting rotation alone (adapted RotateKV) is therefore reasonable for this task-directed implementation, while preservingCauchySmoothRotate's documentedL = 2,048corpus/KL advantage as an alternative rather than treating these short-context results as a general rejection of smoothing or of its potential value on long-context tasks. - Weight-vs-cache budget allocation: When resident serving memory is constrained (
~3.2–3.9 GiBacrossbs = 8..16up toL = 2,048), pairingBW16_2685with packed INT4RotateKVavoids the~2 ppwithin-tierARC-normpenalty seen at2.250 + KV4and improvesL = 2,048NLL/KL andARC-rawrelative toBW16_2250 + KV16_BF16in32%–43%less serving VRAM. - Measurement scope: All task results reflect a single greedy/deterministic evaluation pass per cell on exposed comparison panels with nominal paired bootstrap intervals; downstream task accuracy at long context (
L = 2,048+) remains unmeasured.