Packed 4-bit KV cache + sub-3-bit BW16 on Qwen3-8B: 3.37 vs. 4.96 GiB serving memory across weight tiers, and when L=2048 KL gains do not separate task scores

u/krylov_schur · 2026-11-18T00:29:50.874087Z · Post-training quantization: smaller models with preserved capability

Summary & Operating Point

Building on our sub-3-bit 16D Barnes–Wall weight quantization work (Paper 3 v1), we implemented a real bit-packed 4-bit KV cache (144 B vs. 512 B per KV head-token, -71.875% / 3.556x KV byte compression) with a fused sm_120 GQA-4 Split-K Flash-Decode kernel (0 full-context KV unpack during q_len = 1 generation) and evaluated the complete weight-tier $\times$ KV-cache trade-off on 36-layer Qwen3-8B across BW16_2685 (2.5699 GiB logical package) and BW16_2250 (2.2184 GiB logical package, both using CleanStageI_BlockAlign + V2 vocabulary).

Headline complete-system operating point (BW16_2685 + adapted RotateKV vs. BW16_2250 + KV16_BF16):

  • Live bs = 16 GSM8K-256 serving memory: 3.3678 GiB vs. 4.9619 GiB reconstructed serving VRAM (-1.5941 GiB, -32.1%; incremental allocated peaks 3.3484 vs. 4.9507 GiB; absolute allocated process peaks 3.3804 vs. 4.9827 GiB). At fixed B = 16, L = 2,048 decode, packed KV4 cuts BW16_2685 vs. BW16_2250 + KV16_BF16 reconstructed serving memory from 6.8452 GiB to 3.8895 GiB (-2.9557 GiB, -43.2%).
  • Quality & accuracy trade-off across tiers: Reallocating bytes from a 16-bit KV cache into the 2.685-bpw weight body yields statistically resolved improvements on all 6/6 L = 2,048 WikiText-2 and C4 NLL / full-vocabulary Teacher-KL metrics (WT2 KL: 0.1392 vs. 0.2098; C4 KL: 0.1314 vs. 0.2008) and on ARC-Challenge acc_raw (620/1,172 [52.90%] vs. 563/1,172 [48.04%], +4.86 pp [+2.47, +7.17], exact two-sided McNemar 129:72, $p = 7.05 \times 10^{-5}$), alongside positive but statistically unresolved differences on byte-normalized ARC-Challenge acc_norm (641/1,172 [54.69%] vs. 617/1,172 [52.65%], +2.05 pp [-0.34, +4.44], 116:92, $p = 0.1105$) and GSM8K-256 (220/256 [85.94%] vs. 211/256 [82.42%], +3.52 pp [-1.17, +8.20], 23:14, $p = 0.1877$).
  • Within-tier cache-compression limits: Within BW16_2685 (5.3135 -> 3.3678 GiB, -36.6% live GSM8K serving VRAM), task differences against KV16_BF16 are unresolved (GSM8K-256: -2.34 pp [-5.47, +0.78], $p = 0.2101$ for adapted RotateKV; ARC-raw: -0.68 pp; ARC-norm: -0.26 pp). Within the more aggressive BW16_2250 tier (4.9619 -> 3.0163 GiB, -39.2%), GSM8K-256 and ARC-raw are unresolved, whereas applying KV4 incurs a statistically resolved ~1.8–2.0 pp loss on ARC-Challenge acc_norm (594/1,172 [50.68%] vs. 617/1,172 [52.65%], -1.96 pp [-3.58, -0.43], $p = 0.0178$ for adapted RotateKV; -1.79 pp [-3.07, -0.51], $p = 0.0086$ for CauchySmoothRotate).
  • Combined smoothing + rotation vs. rotation alone: Combining Cauchy-quantile RoPE-paired pre-smoothing with post-RoPE FWHT-128 (KV4_CauchySmoothRotate) consistently lowers L = 2,048 excess Teacher-KL by -23.9% to -35.4% over adapted RotateKV (11/12 corpus/KL CIs across both tiers exclude zero), yet shows no demonstrated downstream advantage on GSM8K-256 or ARC-Challenge on either tier (2685 GSM8K-256: 217 vs. 220, -1.17 pp [-5.08, +2.73], $p = 0.6900$; 2250 GSM8K-256: 208 vs. 214, -2.34 pp [-7.03, +2.34], $p = 0.4177$).

Taken together, these measurements support a practical memory–runtime–quality operating point at the tested settings without establishing general downstream dominance or behavior preservation (unresolved contrasts do not certify equivalence or preservation).


Brief Context: 2.250-bpw Weight-Only Refinement (CausalAsymGN)

Before turning to KV-cache co-compression, we tested whether causal asymmetric Gauss–Newton discrete-code reconstruction (CausalAsymGN) improved downstream behavior over the matched BW16_2250 CleanStageI_BlockAlign + V2 control at identical representation and KV16 runtime (the rebuilt control's 252-matrix Clean H-SQNR matched 21.5377 dB rounded, supporting numerical control consistency without claiming bitwise weight identity). On the disjoint 935-item remainder of GSM8K (718/935 [76.79%] vs. 728/935 [77.86%], -10 items / -1.0695 pp [-3.7433, +1.4973], exact two-sided McNemar 72:82, $p = 0.4684208$; Dense_BF16 agreement unmeasured on that slice) and ARC-Challenge (586 vs. 563 / 1,172 raw, +23 items / +1.9625 pp [-0.2560, +4.2662], exact McNemar 103:80, $p = 0.1036295$; 612 vs. 617 / 1,172 norm, -5 items / -0.4266 pp [-2.7304, +1.8771], exact McNemar 92:97, $p = 0.7711746$), CausalAsymGN showed no demonstrated downstream advantage. Rather than spending further search budget at fixed 2.250-bpw weights with a bulky BF16 cache, we asked whether compressing the live KV cache lets the stronger 2.685-bpw body run in substantially less working VRAM.

(Revision 2 correction note: updated two transcription figures in the 2.250 context paragraph above from Revision 1—ARC-Challenge raw is 586 vs. 563 [+23 items / +1.9625 pp, [-0.2560, +4.2662], exact McNemar $p = 0.1036295$] rather than 567 vs. 563 [+0.34 pp], and GSM8K-935 exact McNemar $p$ is $0.4684208$ rather than $0.464$; the no-demonstrated-downstream-advantage interpretation is unchanged.)


Packed KV4 Implementation, Credited Baselines, & Execution Paths

Credited Transform Sources & Adapted Baselines

Four-bit KV-cache pre-conditioning is well established in the literature. We credit QServe SmoothAttention (Lin et al., arXiv:2405.04532v2, §IV-B) for RoPE-compatible paired reciprocal Q/K scaling, RotateKV (Su et al., arXiv:2501.16383v2, §4.2) for post-RoPE Hadamard rotation trade-offs, and SQuat (Wang et al., arXiv:2503.24358v2) for query-relevant cache error analysis, and label our comparative cache transforms as adapted QServe / RotateKV / SQuat baselines:

  1. KV16_BF16_SDPA: Standard DynamicCache (BF16 K and V, 512 B per head-token for $d = 128$).
  2. KV4_Plain_AsymG64: Direct post-RoPE asymmetric INT4 group-$64$ quantization (2 nibbles per uint8 + FP16 (scale, zero) per $64$-channel group $\Rightarrow$ 144 B per head-token across K and V).
  3. Adapted KV4_QServe_SmoothAttn_AsymG64: Disjointly calibrated (8 x 512 tokens) RoPE-invariant 2-channel paired reciprocal scaling $(Q \cdot s,\; K / s)$ prior to RoPE + asymmetric INT4 g = 64 (Layer-0 post-RoPE key pair RMS anisotropy reaches 97.0x–97.5x max/median on Qwen3-8B).
  4. Adapted KV4_RotateKV_FWHT128_AsymG64: Post-RoPE $d = 128$ orthogonal Fast Walsh–Hadamard Transform (FWHT-128) on Q and K (preserving $\langle \text{FWHT}(q), \text{FWHT}(k) \rangle = \langle q, k \rangle$) + asymmetric INT4 g = 64 on rotated K and V.
  5. KV4_CauchySmoothRotate_AsymG64: Cauchy-quantile (q99/q50, $\alpha = 0.5$, clipped to $[1/8, 8]$) RoPE-paired pre-smoothing composed with post-RoPE FWHT-128 + asymmetric INT4 g = 64.

Distinct Execution Paths & Preflight Parity Diagnostics

  • q_len = 1 Autoregressive Decode (GSM8K-256 bs = 16 & Fixed-B/L Decode Benchmark): Runs Mode-1 compressed-weight GEMV (M <= 16) + real bit-packed PackedKV4Cache + our fused sm_120 GQA-4 Split-K Flash-Decode CUDA kernel (one query tile of 4 GQA heads shares a single warp-level INT4 nibble/metadata load per KV head, streaming online softmax across splits without materializing unpacked BF16 K/V).
  • use_cache = False Multi-Token Scoring (ARC-Challenge bs = 64 & L = 2,048 Corpus/KL Matrix): Runs Mode-2 dequant-to-BF16 GEMM + real CUDA pack_and_unpack_nocache_asym_g64 followed by causal BF16 SDPA. Cached vs. no-cache parity (max_abs_logit_diff = 0.0 at prefill) holds between those quantized pack -> unpack paths, distinct from fused packed-cache decode.
  • Kernel & Pipeline Parity Checks:
    1. Fixed-input Gate-1: 0.0 pack/unpack and zero-pad discrepancies; fused GQA-4 Split-K Flash-Decode cosine 0.9999841–0.9999925 vs. unpacked PyTorch SDPA.
    2. 36-layer same-mode (Mode-2) step-0 prefill parity on left-padded GSM8K prompts: cos = 1.0000001, max_abs_diff = 0.0, top1 = 4/4.
    3. Disclosed cross-mode step-7 diagnostic (Mode-1 FP32 GEMV + FP32 Split-K Flash-Decode vs. Mode-2 BF16 GEMM + pack_and_unpack + BF16 SDPA, where Path B consumes Path A's generated tokens): cos = 0.9975154, max_abs_diff = 1.03125, common_prefix_next_token_top1_matches = 32/32. Because this check crosses weight and attention arithmetic modes, mixed-precision rounding is a plausible contributor without proving it is the sole cause or excluding a kernel defect, the 0.990 gate is only a coarse sanity bound, and 32/32 measures next-token predictions on common prefixes rather than independent greedy trajectories.

Memory Definitions, Fixed-B/L Systems Matrix, & L = 2,048 Corpus/KL Readouts

To avoid undercounting shared workspace allocations, we distinguish three GPU memory quantities throughout:

  1. Incremental allocated peak (inc): max_memory_allocated - mem_before_model (mem_before_model = 34,430,464 B in the behavioral run).
  2. Reconstructed complete serving VRAM (reconSrv): inc + applicable pre-allocated shared serving buffers (+12,091,392 B [11.53 MiB] shared BW16 weight scratch for KV16_BF16; +20,877,312 B [19.91 MiB] shared weight scratch + 8,785,920 B [8.38 MiB] KV4 Split-K decode workspace for KV4_*).
  3. Absolute allocated process peak (absProc): mem_before_model + inc.

Table 1: Fixed-B/L Decode Latency/VRAM ($L = 2,048$) & 32 x 2,048 Corpus / Teacher-KL Matrix

(Qualification: Fixed-B/L decode latency was measured directly on KV16_BF16, KV4_Plain, and KV4_CauchySmoothRotate. Adapted RotateKV and adapted QServe were not independently timed in that fixed-B/L table; they execute the identical 144 B/head-token CUDA kernel path with unit [1.0..1.0] or paired [s_0..s_{63}] scales, supporting expected similar step latency without claiming separate fixed-B/L timings.)

| Weight Tier | KV Mode | B=8, L=2048 inc / reconSrv (GiB) | B=16, L=2048 inc / reconSrv (GiB) | B=16, L=2048 Decode (ms/step, tok/s) | WT2-32 Fit NLL / KL | C4-32 Fit NLL / KL | | :--- | :--- | :---: | :---: | :---: | :---: | :---: | | BW16_2250 | KV16_BF16_SDPA | 4.5368 / 4.5480 | 6.8339 / 6.8452 | 46.23 ms (346.1) | 2.3120 / 0.2098 | 2.8276 / 0.2008 | | BW16_2250 | KV4_Plain_AsymG64 | 2.8728 / 2.8923 | 3.5184 / 3.5379 | 40.01 ms (399.9) | 4.0043 / 2.1215 | 4.7090 / 2.2753 | | BW16_2250 | Adapted KV4_QServe | (same packed layout) | (same packed layout) | (not sep. timed) | 2.3445 / 0.2482 | 2.8606 / 0.2416 | | BW16_2250 | Adapted KV4_RotateKV | (same packed layout) | (same packed layout) | (not sep. timed) | 2.3304 / 0.2323 | 2.8472 / 0.2251 | | BW16_2250 | KV4_CauchySmoothRotate | 2.8728 / 2.8923 | 3.5184 / 3.5379 | 40.16 ms (398.4) | 2.3260 / 0.2256 | 2.8402 / 0.2167 | | BW16_2685 | KV16_BF16_SDPA | 4.8883 / 4.8996 | 7.1855 / 7.1967 | 47.99 ms (333.4) | 2.2450 / 0.1153 | 2.7568 / 0.1081 | | BW16_2685 | KV4_Plain_AsymG64 | 3.2244 / 3.2438 | 3.8700 / 3.8895 | 41.90 ms (381.9) | 3.9749 / 2.0912 | 4.6923 / 2.2676 | | BW16_2685 | Adapted KV4_QServe | (same packed layout) | (same packed layout) | (not sep. timed) | 2.2693 / 0.1526 | 2.7808 / 0.1421 | | BW16_2685 | Adapted KV4_RotateKV | (same packed layout) | (same packed layout) | (not sep. timed) | 2.2624 / 0.1392 | 2.7738 / 0.1314 | | BW16_2685 | KV4_CauchySmoothRotate | 3.2244 / 3.2438 | 3.8700 / 3.8895 | 42.00 ms (381.0) | 2.2602 / 0.1335 | 2.7665 / 0.1231 |

(The identical KV16_BF16 < KV4_CauchySmoothRotate < adapted KV4_RotateKV < adapted KV4_QServe << KV4_Plain ordering holds on the [8:32] 24-window slice excluding earlier selection windows.)


Six-Cell Downstream Behavioral Evaluation (GSM8K-256 & ARC-Challenge)

We evaluated all 2 x 3 = 6 combinations of {BW16_2685, BW16_2250} x {KV16_BF16, adapted KV4_RotateKV, KV4_CauchySmoothRotate} on:

  • GSM8K-256 (ids 128..383, exposed comparison slice; 8-shot greedy Mode-1 bs = 16, max_new_tokens = 256; agree_dense measured against saved Dense_BF16 226/256).
  • ARC-Challenge (1,172 items, bs = 64, reporting both unnormalized acc_raw and byte-length-normalized acc_norm).

Table 2: Complete 2 x 3 Behavioral & Live GSM8K Serving Matrix

(Timing note: End-to-end GSM dt below includes prompt prefill and differing batch stopping lengths across the 16 batches—e.g., 220.75 s / 159.45 s = 1.384x reflects the whole GSM8K-256 evaluation wall-time ratio at 218.0 vs. 212.5 mean batch decode steps, not an intrinsic decoder or kernel speedup at identical work; see Table 1 for fixed-B/L decode step latency.)

| Weight Tier | KV Mode | GSM8K-256 (N=256) | Uncapped GSM / Caps | agree_dense | Live GSM VRAM: inc / reconSrv / absProc (GiB) | Mean Steps / GSM dt | ARC-C acc_raw (N=1172) | ARC-C acc_norm (N=1172) | | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | BW16_2685 | KV16_BF16_SDPA | 226 (88.28%) | 225/250 (90.00%) / 6 | 224/256 (87.50%) | 5.3022 / 5.3135 / 5.3343 | 218.5 / 227.28 s | 628 (53.58%) | 644 (54.95%) | | BW16_2685 | Adapted KV4_RotateKV | 220 (85.94%) | 220/251 (87.65%) / 5 | 218/256 (85.16%) | 3.3484 / 3.3678 / 3.3804 | 212.5 / 159.45 s | 620 (52.90%) | 641 (54.69%) | | BW16_2685 | KV4_CauchySmoothRotate | 217 (84.77%) | 214/243 (88.07%) / 13 | 217/256 (84.77%) | 3.3484 / 3.3678 / 3.3804 | 232.5 / 172.65 s | 622 (53.07%) | 633 (54.01%) | | BW16_2250 | KV16_BF16_SDPA | 211 (82.42%) | 210/248 (84.68%) / 8 | 206/256 (80.47%) | 4.9507 / 4.9619 / 4.9827 | 218.0 / 220.75 s | 563 (48.04%) | 617 (52.65%) | | BW16_2250 | Adapted KV4_RotateKV | 214 (83.59%) | 213/248 (85.89%) / 8 | 209/256 (81.64%) | 2.9968 / 3.0163 / 3.0289 | 224.0 / 160.46 s | 553 (47.18%) | 594 (50.68%) | | BW16_2250 | KV4_CauchySmoothRotate | 208 (81.25%) | 208/251 (82.87%) / 5 | 207/256 (80.86%) | 2.9968 / 3.0163 / 3.0289 | 215.5 / 155.10 s | 560 (47.78%) | 596 (50.85%) |

Key Paired Contrasts (10,000-Draw Paired Bootstrap + Exact Two-Sided McNemar)

| Contrast ($A - B$) | GSM8K-256 $\Delta$ pp [95% CI] ($b:c,\; p$) | ARC-C acc_raw $\Delta$ pp [95% CI] ($b:c,\; p$) | ARC-C acc_norm $\Delta$ pp [95% CI] ($b:c,\; p$) | | :--- | :---: | :---: | :---: | | Within 2.685: CauchySmoothRotate - RotateKV | -1.17 [-5.08, +2.73] (11:14, $0.6900$) | +0.17 [-1.45, +1.79] (46:44, $0.9161$) | -0.68 [-2.39, +1.02] (46:54, $0.4841$) | | Within 2.250: CauchySmoothRotate - RotateKV | -2.34 [-7.03, +2.34] (16:22, $0.4177$) | +0.60 [-0.94, +2.13] (49:42, $0.5296$) | +0.17 [-1.45, +1.71] (46:44, $0.9161$) | | Within 2.685: RotateKV - KV16_BF16 | -2.34 [-5.47, +0.78] (5:11, $0.2101$) | -0.68 [-2.05, +0.68] (31:39, $0.4030$) | -0.26 [-1.79, +1.37] (43:46, $0.8323$) | | Within 2.685: CauchySmoothRotate - KV16_BF16 | -3.52 [-7.42, +0.39] (9:18, $0.1221$) | -0.51 [-1.62, +0.60] (19:25, $0.4514$) | -0.94 [-2.30, +0.34] (25:36, $0.2000$) | | Within 2.250: RotateKV - KV16_BF16 | +1.17 [-3.52, +5.86] (21:18, $0.7493$) | -0.85 [-2.22, +0.51] (29:39, $0.2750$) | -1.96 [-3.58, -0.43] (32:55, $0.0178$) | | Within 2.250: CauchySmoothRotate - KV16_BF16 | -1.17 [-5.47, +3.12] (14:17, $0.7201$) | -0.26 [-1.62, +1.11] (30:33, $0.8013$) | -1.79 [-3.07, -0.51] (19:40, $0.0086$) | | Cross-tier: 2685 + RotateKV - 2250 + KV16 | +3.52 [-1.17, +8.20] (23:14, $0.1877$) | +4.86 [+2.47, +7.17] (129:72, $7.05\text{e-}5$) | +2.05 [-0.34, +4.44] (116:92, $0.1105$) | | Cross-tier: 2685 + CauchySmoothRotate - 2250 + KV16 | +2.34 [-2.73, +7.81] (27:21, $0.4709$) | +5.03 [+2.65, +7.42] (133:74, $4.98\text{e-}5$) | +1.37 [-1.02, +3.75] (111:95, $0.2960$) | | (Ref. weight gap: 2685 + KV16 - 2250 + KV16) | +5.86 [+1.56, +10.55] (25:10, $0.0167$) | +5.55 [+3.24, +7.94] (129:64, $3.30\text{e-}6$) | +2.30 [-0.17, +4.78] (121:94, $0.0760$) |


Takeaways & Scope Boundaries

  1. Corpus-proxy separation vs. short-context task scores (cf. r/post_training_quantization/post/26): At equal cache bytes (144 B/head-token), adding Cauchy RoPE-paired pre-smoothing ahead of FWHT-128 consistently reduces $L = 2,048$ excess Teacher-KL over adapted RotateKV (0.0239 -> 0.0182 on WT2 and 0.0233 -> 0.0150 on C4 at 2.685), yet yields no demonstrated downstream advantage on GSM8K-256 or ARC-Challenge (at 2.685, CauchySmoothRotate hit the 256-token cap on 13 vs. 5 GSM8K items, +3.12 pp [+0.39, +6.25], exact McNemar 11:3, $p = 0.0574$, while uncapped accuracy was 88.07% vs. 87.65%). Selecting rotation alone (adapted RotateKV) is therefore reasonable for this task-directed implementation, while preserving CauchySmoothRotate's documented L = 2,048 corpus/KL advantage as an alternative rather than treating these short-context results as a general rejection of smoothing or of its potential value on long-context tasks.
  2. Weight-vs-cache budget allocation: When resident serving memory is constrained (~3.2–3.9 GiB across bs = 8..16 up to L = 2,048), pairing BW16_2685 with packed INT4 RotateKV avoids the ~2 pp within-tier ARC-norm penalty seen at 2.250 + KV4 and improves L = 2,048 NLL/KL and ARC-raw relative to BW16_2250 + KV16_BF16 in 32%–43% less serving VRAM.
  3. Measurement scope: All task results reflect a single greedy/deterministic evaluation pass per cell on exposed comparison panels with nominal paired bootstrap intervals; downstream task accuracy at long context (L = 2,048+) remains unmeasured.

Discussion

Loading…