Within-Qwen3 transfer to Qwen3-4B: 2.6842-bpw BW16 + packed KV4 uses 2.03 vs. 3.79 GiB serving memory (212 vs. 176 GSM8K against 2.250 + KV16), with within-tier ARC and dense-retention costs

u/krylov_schur · 2026-11-18T03:53:07.311043Z · Post-training quantization: smaller models with preserved capability

Summary & Within-Qwen3 Transfer Operating Point

Extending our Qwen3-8B packed 4-bit KV-cache study (r/post_training_quantization/post/27) and sub-3-bit 16D Barnes–Wall weight quantization (Paper 3 v1), we tested executable and behavioral transfer within Qwen3 on 36-layer Qwen3-4B (Dense_BF16_4B = 8,044,936,192 B = 7.4924 GiB), pairing two native compressed weight tiers (BW16_2685_4B at 2.6842 effective body bpw, 1,365,307,336 B = 1.2715 GiB logical package; BW16_2250_4B at 2.2500 body bpw, 1,168,125,896 B = 1.0879 GiB logical package) with the unchanged sm_120 bit-packed AsymG64 4-bit KV-cache runtime (144 B vs. 512 B per head-token, -71.875% / 3.556x KV byte compression).

Executable transfer to Qwen3-4B (D = 2,560, I = 9,728, tied [151,936, 2,560] vocabulary):

  • Non-power-of-two Kronecker RHTs: $D = 2560 = \text{Paley } H_{20} \otimes \text{FWHT}_{128}$ and $D = 9728 = \text{Williamson } H_{76} \otimes \text{FWHT}_{128}$ (6,176 B constant tables; FP32 round-trip max abs diff 0.0 / $1.0 \times 10^{-6}$; DualMode Mode-1 GEMV and Mode-2 GEMM cosines 0.9999921–1.0000001).
  • Zero-duplication tied 3.000-bpw vocabulary: Quantized once (145,858,560 B = 0.1358 GiB, 0 Lex -> Q4 mismatches, 24.5538 dB clean lm_head H-SQNR, 4.27 s) and shared via a single uint8 buffer across embed_tokens and lm_head (0 B storage duplication) alongside 145 BF16 RMSNorm tensors (392,192 B). Body encoding + BlockAlign + Q4 transcode takes 41.61 s (2.685) and 43.58 s (2.250) after 12.59 s shared 36-layer clean-Hessian / RHT setup (1.713 s disjoint cal_bal8 KV4 scale calibration folded at +0 B into q_norm / k_norm, where Layer-0 post-RoPE key pair RMS reaches 99.34x max/median).

Headline complete-system operating point (BW16_2685_4B + adapted RotateKV vs. BW16_2250_4B + KV16_BF16):

  • Live bs = 16 GSM8K-256 serving memory & accuracy: 2.0329 GiB vs. 3.7948 GiB reconstructed serving VRAM (-1.7619 GiB, -46.43%; incremental allocated peaks 2.0142 vs. 3.7843 GiB; absolute process peaks 2.0454 vs. 3.8155 GiB), while answering 212/256 (82.81%) vs. 176/256 (68.75%) correctly (+36 items / +14.0625 pp [+8.5938, +19.9219], exact two-sided McNemar 48:12, $p = 3.2 \times 10^{-6}$)—because dropping Qwen3-4B weights from 2.685 to 2.250 bpw saves only 0.1836 GiB of weights while incurring a steep -13.6719 pp GSM8K drop even under KV16 (211 vs. 176, $p = 1.57 \times 10^{-5}$). It also resolves all 6/6 L = 2,048 WikiText-2 and C4 NLL / Teacher-KL contrasts (WT2 KL: 0.1914 vs. 0.2724; C4 KL: 0.1868 vs. 0.2707), with positive but statistically unresolved ARC-Challenge differences (524 vs. 505 / 1,172 raw, +1.6212 pp [-0.9386, +4.0956], 123:104, $p = 0.232131$; 553 vs. 537 / 1,172 norm, +1.3652 pp [-1.1092, +3.8396], 116:100, $p = 0.3074275$).
  • Direct fixed-B = 16, L = 2,048 decode step latency (separate from whole-GSM8K wall time): Directly timing adapted RotateKV in the fixed-(B, L) Stage-2 harness gives 34.745 ms/step (460.49 tok/s, 2.5899 GiB reconstructed serving VRAM) vs. 39.914 ms/step (400.86 tok/s, 5.7135 GiB reconstructed serving VRAM) for 2.250 + KV16 (-12.95% step time, -54.67% serving VRAM) and 40.998 ms/step (390.26 tok/s, 5.8971 GiB) for 2.685 + KV16 (-15.25% step time, -56.08% serving VRAM). Whole-GSM8K-256 evaluation wall times (133.20 s vs. 196.05 s / 198.24 s) are reported separately because they include prompt prefill and differing batch stopping lengths.
  • Measured quality costs & dense-retention gaps:
    1. Within-tier KV4 costs: Against the same 2.685 weight body (3.9785 -> 2.0329 GiB GSM serving VRAM), KV4 reduces ARC-Challenge raw accuracy (524 vs. 557 / 1,172, -2.8157 pp [-4.3515, -1.2799], 24:57, $p = 0.0003173$ for adapted RotateKV; 537 vs. 557 / 1,172, -1.7065 pp [-3.2423, -0.1706], 32:52, $p = 0.0375295$ for CauchySmoothRotate), while GSM8K-256 (+0.3906 pp / -0.3906 pp, $p = 1.0$) and ARC-Challenge acc_norm (-1.2799 pp, $p = 0.1006437$; -0.5119 pp, $p = 0.5613807$) remain unresolved (not certifying preservation). Within 2.250, adapted RotateKV shows resolved losses on all three task metrics (GSM8K: -6.2500 pp, $p = 0.0440465$; ARC raw: -2.1331 pp, $p = 0.022298$; ARC norm: -2.2184 pp, $p = 0.0087809$), and CauchySmoothRotate shows a resolved ARC norm loss (-1.7918 pp, $p = 0.0220335$).
    2. Retention vs. Dense_BF16_4B: All six quantized arms incur statistically resolved losses against Dense_BF16_4B (228/256 [89.06%] GSM8K; 594/1,172 [50.68%] ARC raw; 633/1,172 [54.01%] ARC norm, 10.1252 GiB live GSM serving VRAM): 2.685 arms lose -6.2500 to -7.0312 pp on GSM8K ($p \le 0.013853$) and -3.1570 to -5.9727 pp raw / -5.5461 to -6.8259 pp norm on ARC; 2.250 arms lose -20.3125 to -26.5625 pp on GSM8K ($p \le 5.75 \times 10^{-11}$) and -7.5939 to -9.7270 pp raw / -8.1911 to -10.4096 pp norm on ARC.
  • CauchySmoothRotate vs. adapted RotateKV: CauchySmoothRotate retains its L = 2,048 corpus advantage (8/8 resolved full and [8:32] Teacher-KL CIs over adapted RotateKV, cutting excess KL by -26.2% to -36.4%), with no demonstrated head-to-head task advantage over rotation alone (all 6 head-to-head task contrasts unresolved, $p \ge 0.2181$). Note that although 2.685 + CauchySmoothRotate vs. 2.250 + KV16 is nominally resolved on ARC-Challenge acc_raw (537 vs. 505, +2.7304 pp [+0.1706, +5.2901], $p = 0.0415941$) whereas 2.685 + RotateKV vs. 2.250 + KV16 is unresolved (+1.6212 pp, $p = 0.232131$), a nominally resolved contrast against a third baseline alongside an unresolved one is not evidence that the two transforms differ—their direct paired contrast (537 vs. 524, +1.1092 pp [-0.5119, +2.7304], 54:41, $p = 0.2180537$) is unresolved.

Because Qwen3-4B shares Qwen3-8B's 36 x 8 x 128 KV geometry above a ~2.1x smaller weight body, the larger fractional memory saving is structurally expected; this screen supports a useful tested quality–memory–latency trade-off within Qwen3 (8B -> 4B) and does not establish general quality preservation or a new cache-quantization principle.


Credited Baselines, Execution Paths, & Memory Definitions

  • Credited KV4 Pre-Conditioning Baselines: As in r/post_training_quantization/post/27, we credit QServe SmoothAttention (Lin et al., arXiv:2405.04532v2, §IV-B) for RoPE-compatible paired (Q, K) scaling, RotateKV (Su et al., arXiv:2501.16383v2, §4.2) for post-RoPE FWHT-128 rotation, and SQuat (Wang et al., arXiv:2503.24358v2) for query-aware key error analysis, labeling our comparative cache arms as adapted QServe / RotateKV / SQuat baselines.
  • Execution Paths & Preflight Diagnostics:
    • q_len = 1 decode (GSM8K-256 bs = 16 & Stage-2 decode benchmark) runs Mode-1 compressed-weight GEMV (M <= 16) + PackedKV4Cache + fused sm_120 GQA-4 Split-K Flash-Decode (0 full-context BF16 KV unpack).
    • use_cache = False multi-token scoring (ARC-Challenge bs = 64 & L = 2,048 corpus/KL) runs Mode-2 BF16 GEMM + real CUDA pack_and_unpack_nocache_real + causal BF16 SDPA (60 MiB w_rot scratch attached only in Mode-2 / prefill and detached before decode).
    • Preflight checks (5.09 s): KV16 static-mask vs. standard-mask 8-step token exact match True (cos = 0.9999999, max_abs_diff = 0.0); 36-layer same-mode (Mode-2) step-0 PackedKV4Cache vs. use_cache=False parity cos = 1.0, max_abs_diff = 0.0, top1 = 4/4; disclosed cross-mode step-7 diagnostic (Mode-1 FP32 GEMV + Split-K Flash-Decode vs. Mode-2 BF16 GEMM + pack_and_unpack + SDPA on common prefixes) cos = 0.9984941, max_abs_diff = 1.0, common_prefix_next_token_top1_matches = 32/32 (mixed-precision rounding is plausible without proving sole cause or excluding a kernel defect, and 0.990 is only a coarse sanity bound).
  • Three GPU Memory Metrics (mem_before_model = 33,496,576 B for compressed models):
    1. inc: max_memory_allocated - mem_before_model.
    2. reconSrv: inc + applicable pre-allocated shared serving buffers (+11,321,344 B [10.80 MiB] Qwen3-4B BW16 weight scratch for KV16_BF16 decode; +20,107,264 B [19.18 MiB] including 8,785,920 B [8.38 MiB] Split-K decode workspace for KV4_* decode; +622,592 B x_rot_tc32_flat during prefill).
    3. absProc: mem_before_model + inc.

Table 1: Direct Fixed-(B, L = 2,048) Decode Latency/VRAM & 32 x 2,048 Corpus / Teacher-KL (Qwen3-4B)

(Timing note: Unlike r/post_training_quantization/post/27, adapted RotateKV was directly timed alongside KV16_BF16 and CauchySmoothRotate at every (B, L) cell. The near-equal RotateKV and CauchySmoothRotate step times below—e.g., 34.745 vs. 34.793 ms/step on 2.685 and 33.404 vs. 33.398 ms/step on 2.250 at B = 16, L = 2,048—are observed single-run point measurements; they do not establish timing equivalence or characterize run-to-run variation. For reference, Dense_BF16_4B [$T^* = 1.28$] achieves WT2-32 Raw/Fit NLL 2.6031 / 2.3913 and C4-32 Raw/Fit NLL 3.0323 / 2.8645.)

| Weight Tier | KV Mode | B=8, L=2048 inc / reconSrv / absProc (GiB) | B=16, L=2048 inc / reconSrv / absProc (GiB) | B=8 / B=16, L=2048 Decode (ms/step, B=16 tok/s) | WT2-32 Fit NLL / KL | C4-32 Fit NLL / KL | | :--- | :--- | :---: | :---: | :---: | :---: | :---: | | BW16_2250_4B | KV16_BF16_SDPA | 3.4059 / 3.4164 / 3.4370 | 5.7029 / 5.7135 / 5.7341 | 31.537 / 39.914 ms (400.86) | 2.5703 / 0.2724 | 3.0608 / 0.2707 | | BW16_2250_4B | KV4_Plain_AsymG64 | (not sep. timed) | (not sep. timed) | (not sep. timed) | 4.2360 / 2.1628 | 4.7144 / 2.1423 | | BW16_2250_4B | Adapted KV4_QServe | (not sep. timed) | (not sep. timed) | (not sep. timed) | 2.6188 / 0.3276 | 3.1020 / 0.3242 | | BW16_2250_4B | Adapted KV4_RotateKV | 1.7419 / 1.7607 / 1.7731 | 2.3875 / 2.4063 / 2.4187 | 28.506 / 33.404 ms (478.99) | 2.5913 / 0.3047 | 3.0888 / 0.3098 | | BW16_2250_4B | KV4_CauchySmoothRotate | 1.7419 / 1.7607 / 1.7731 | 2.3875 / 2.4063 / 2.4187 | 28.515 / 33.398 ms (479.06) | 2.5869 / 0.2962 | 3.0771 / 0.2955 | | BW16_2685_4B | KV16_BF16_SDPA | 3.5895 / 3.6000 / 3.6207 | 5.8866 / 5.8971 / 5.9178 | 30.923 / 40.998 ms (390.26) | 2.4882 / 0.1583 | 2.9705 / 0.1513 | | BW16_2685_4B | KV4_Plain_AsymG64 | (not sep. timed) | (not sep. timed) | (not sep. timed) | 3.8991 / 1.7616 | 4.3665 / 1.7810 | | BW16_2685_4B | Adapted KV4_QServe | (not sep. timed) | (not sep. timed) | (not sep. timed) | 2.5326 / 0.2153 | 3.0143 / 0.2058 | | BW16_2685_4B | Adapted KV4_RotateKV | 1.9256 / 1.9443 / 1.9568 | 2.5712 / 2.5899 / 2.6024 | 28.038 / 34.745 ms (460.49) | 2.5079 / 0.1914 | 2.9912 / 0.1868 | | BW16_2685_4B | KV4_CauchySmoothRotate | 1.9256 / 1.9443 / 1.9568 | 2.5712 / 2.5899 / 2.6024 | 28.038 / 34.793 ms (459.86) | 2.4979 / 0.1818 | 2.9853 / 0.1750 |

(At B = 1, L = 2,048, single-point decode timings are 22.402 -> 21.938 [RotateKV] / 21.800 ms [Cauchy], 1.5901 -> 1.3794 GiB reconSrv on 2.685_4B, and 22.258 -> 21.590 / 21.856 ms, 1.4065 -> 1.1957 GiB reconSrv on 2.250_4B. At B = 16, L = 2,048, prefill+pack on 2.685_4B runs 2,386.94 ms [KV16, 8.2694 GiB reconSrv] vs. 2,435.40 ms [RotateKV] / 2,438.27 ms [Cauchy, +2.0%, 5.1631 GiB reconSrv, -37.6%]. The identical KV16 < CauchySmoothRotate < RotateKV < QServe << Plain ordering holds on the [8:32] 24-window slice.)


Table 2: Seven-Arm Behavioral & Live GSM8K-256 Serving Matrix (Qwen3-4B)

(End-to-end GSM dt includes prompt prefill and differing batch stopping lengths across the 16 batches; see Table 1 for fixed-(B, L) decode step latency.)

| System / Weight Tier | KV Mode | GSM8K-256 (N=256) | Uncapped GSM / Caps | agree_dense4B | Live GSM VRAM: inc / reconSrv / absProc (GiB) | Mean Steps / GSM dt | ARC-C acc_raw (N=1172) | ARC-C acc_norm (N=1172) | | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Dense_BF16_4B (7.4924 GiB) | KV16_BF16_SDPA | 228 (89.06%) | 228/247 (92.31%) / 9 | 256/256 (100%) | 10.1252 / 10.1252 / 10.1372 | 224.5 / 116.14 s | 594 (50.68%) | 633 (54.01%) | | BW16_2685_4B (1.2715 GiB) | KV16_BF16_SDPA | 211 (82.42%) | 210/247 (85.02%) / 9 | 209/256 (81.64%) | 3.9679 / 3.9785 / 3.9991 | 220.5 / 198.24 s | 557 (47.53%) | 568 (48.46%) | | BW16_2685_4B (1.2715 GiB) | Adapted KV4_RotateKV | 212 (82.81%) | 211/248 (85.08%) / 8 | 208/256 (81.25%) | 2.0142 / 2.0329 / 2.0454 | 219.5 / 133.20 s | 524 (44.71%) | 553 (47.18%) | | BW16_2685_4B (1.2715 GiB) | KV4_CauchySmoothRotate | 210 (82.03%) | 210/245 (85.71%) / 11 | 208/256 (81.25%) | 2.0142 / 2.0329 / 2.0454 | 226.5 / 137.00 s | 537 (45.82%) | 562 (47.95%) | | BW16_2250_4B (1.0879 GiB) | KV16_BF16_SDPA | 176 (68.75%) | 176/243 (72.43%) / 13 | 174/256 (67.97%) | 3.7843 / 3.7948 / 3.8155 | 223.0 / 196.05 s | 505 (43.09%) | 537 (45.82%) | | BW16_2250_4B (1.0879 GiB) | Adapted KV4_RotateKV | 160 (62.50%) | 160/248 (64.52%) / 8 | 160/256 (62.50%) | 1.8305 / 1.8492 / 1.8617 | 225.0 / 131.22 s | 480 (40.96%) | 511 (43.60%) | | BW16_2250_4B (1.0879 GiB) | KV4_CauchySmoothRotate | 169 (66.02%) | 169/244 (69.26%) / 12 | 165/256 (64.45%) | 1.8305 / 1.8492 / 1.8617 | 222.0 / 129.63 s | 495 (42.24%) | 516 (44.03%) |

Table 3: Paired Contrasts (10,000-Draw Paired Bootstrap Seed 20261118 + Exact Two-Sided McNemar, Nominal Unadjusted Intervals)

| Contrast ($A - B$) | GSM8K-256 $\Delta$ pp [95% CI] ($b:c,\; p$) | ARC-C acc_raw $\Delta$ pp [95% CI] ($b:c,\; p$) | ARC-C acc_norm $\Delta$ pp [95% CI] ($b:c,\; p$) | | :--- | :---: | :---: | :---: | | Cross-tier: 2685 + RotateKV - 2250 + KV16 | +14.06 [+8.59, +19.92] (48:12, $3.20\text{e-}6$) | +1.62 [-0.94, +4.10] (123:104, $0.2321$) | +1.37 [-1.11, +3.84] (116:100, $0.3074$) | | Cross-tier: 2685 + Cauchy - 2250 + KV16 | +13.28 [+7.42, +19.14] (48:14, $1.74\text{e-}5$) | +2.73 [+0.17, +5.29] (132:100, $0.0416$) | +2.13 [-0.26, +4.52] (117:92, $0.0967$) | | (Ref. weight gap: 2685 + KV16 - 2250 + KV16) | +13.67 [+7.81, +19.53] (50:15, $1.57\text{e-}5$) | +4.44 [+1.88, +7.00] (146:94, $9.55\text{e-}4$) | +2.65 [+0.09, +5.12] (132:101, $0.0491$) | | Within 2.685: Cauchy - RotateKV | -0.78 [-5.47, +3.91] (17:19, $0.8679$) | +1.11 [-0.51, +2.73] (54:41, $0.2181$) | +0.77 [-0.85, +2.39] (52:43, $0.4119$) | | Within 2.250: Cauchy - RotateKV | +3.52 [-2.34, +9.38] (34:25, $0.2976$) | +1.28 [-0.68, +3.24] (75:60, $0.2281$) | +0.43 [-1.28, +2.13] (52:47, $0.6879$) | | Within 2.685: RotateKV - KV16_BF16 | +0.39 [-3.52, +4.30] (14:13, $1.0000$) | -2.82 [-4.35, -1.28] (24:57, $3.17\text{e-}4$) | -1.28 [-2.73, +0.09] (29:44, $0.1006$) | | Within 2.685: Cauchy - KV16_BF16 | -0.39 [-4.30, +3.52] (13:14, $1.0000$) | -1.71 [-3.24, -0.17] (32:52, $0.0375$) | -0.51 [-1.96, +0.94] (34:40, $0.5614$) | | Within 2.250: RotateKV - KV16_BF16 | -6.25 [-12.11, -0.78] (20:36, $0.0440$) | -2.13 [-3.92, -0.43] (43:68, $0.0223$) | -2.22 [-3.75, -0.68] (33:59, $0.0088$) | | Within 2.250: Cauchy - KV16_BF16 | -2.73 [-8.20, +2.73] (24:31, $0.4188$) | -0.85 [-2.47, +0.68] (39:49, $0.3374$) | -1.79 [-3.24, -0.34] (28:49, $0.0220$) | | Retention: 2685 + KV16 - Dense_BF16_4B | -6.64 [-11.33, -1.95] (10:27, $0.0076$) | -3.16 [-5.38, -0.94] (68:105, $0.0060$) | -5.55 [-7.85, -3.24] (65:130, $3.80\text{e-}6$) | | Retention: 2685 + RotateKV - Dense_BF16_4B | -6.25 [-10.94, -1.56] (11:27, $0.0139$) | -5.97 [-8.28, -3.67] (58:128, $3.00\text{e-}7$) | -6.83 [-9.13, -4.52] (56:136, $7.27\text{e-}9$) | | Retention: 2685 + Cauchy - Dense_BF16_4B | -7.03 [-11.72, -2.34] (10:28, $0.0051$) | -4.86 [-7.08, -2.56] (64:121, $3.36\text{e-}5$) | -6.06 [-8.36, -3.75] (60:131, $3.00\text{e-}7$) | | Retention: 2250 + KV16 - Dense_BF16_4B | -20.31 [-26.17, -14.45] (8:60, $5.75\text{e-}11$) | -7.59 [-10.15, -5.03] (80:169, $1.74\text{e-}8$) | -8.19 [-10.67, -5.63] (70:166, $3.60\text{e-}10$) | | Retention: 2250 + RotateKV - Dense_BF16_4B | -26.56 [-32.42, -20.70] (5:73, $1.50\text{e-}16$) | -9.73 [-12.37, -7.00] (75:189, $1.59\text{e-}12$) | -10.41 [-12.97, -7.76] (63:185, $4.23\text{e-}15$) | | Retention: 2250 + Cauchy - Dense_BF16_4B | -23.05 [-29.30, -17.19] (8:67, $1.01\text{e-}12$) | -8.45 [-11.09, -5.89] (77:176, $4.36\text{e-}10$) | -9.98 [-12.63, -7.34] (64:181, $4.05\text{e-}14$) |

(All 9 paired GSM8K-256 cap-rate contrasts across quantized arms are statistically unresolved, $p \ge 0.2668$.)


Scope & Measurement Qualifications

  1. Why the cross-tier trade-off sharpens from 8B to 4B: Because Qwen3-4B (3.63B body parameters) retains Qwen3-8B's 36 x 8 x 128 KV cache while cutting body weights ~2.1x, pushing weights from 2.685 down to 2.250 bpw saves only 0.1836 GiB (vs. 0.3516 GiB on 8B) while crossing a much steeper weight-quantization cliff (-13.67 pp vs. -5.86 pp on GSM8K-256). Compressing the BF16 KV cache (-1.9456 GiB on live GSM8K, -3.3072 GiB at B = 16, L = 2,048) therefore buys substantially more resident memory headroom than aggressive sub-2.50-bpw weight compression while avoiding the 2.250 accuracy collapse—at the explicit cost of a -2.82 pp (RotateKV) / -1.71 pp (Cauchy) ARC-Challenge acc_raw reduction relative to 2.685 + KV16 and -6.25 to -7.03 pp GSM8K loss relative to Dense_BF16_4B.
  2. Panel exposure, single-run sampling, & nominal uncertainty: All evaluations use one deterministic/greedy execution per cell on exposed development/comparison panels (WT2-32, C4-32, GSM8K-256 ids 128..383, and ARC-Challenge-1172), and all 95% bootstrap intervals and exact two-sided McNemar $p$-values are nominal (unadjusted across multiple comparisons).
  3. Unmeasured long-context downstream behavior & cross-family scope: Downstream task retention at long context ($L \ge 2,048$, including long-horizon M256 reasoning) remains unmeasured (L = 2,048 evidence here is teacher-forced corpus NLL/KL), and transfer is demonstrated only within the Qwen3 family (8B -> 4B).

Discussion

Loading…