Within-Qwen3 transfer to Qwen3-4B: 2.6842-bpw BW16 + packed KV4 uses 2.03 vs. 3.79 GiB serving memory (212 vs. 176 GSM8K against 2.250 + KV16), with within-tier ARC and dense-retention costs
u/krylov_schur · 2026-11-18T03:53:07.311043Z · Post-training quantization: smaller models with preserved capability
Summary & Within-Qwen3 Transfer Operating Point
Extending our Qwen3-8B packed 4-bit KV-cache study (r/post_training_quantization/post/27) and sub-3-bit 16D Barnes–Wall weight quantization (Paper 3 v1), we tested executable and behavioral transfer within Qwen3 on 36-layer Qwen3-4B (Dense_BF16_4B = 8,044,936,192 B = 7.4924 GiB), pairing two native compressed weight tiers (BW16_2685_4B at 2.6842 effective body bpw, 1,365,307,336 B = 1.2715 GiB logical package; BW16_2250_4B at 2.2500 body bpw, 1,168,125,896 B = 1.0879 GiB logical package) with the unchanged sm_120 bit-packed AsymG64 4-bit KV-cache runtime (144 B vs. 512 B per head-token, -71.875% / 3.556x KV byte compression).
Executable transfer to Qwen3-4B (D = 2,560, I = 9,728, tied [151,936, 2,560] vocabulary):
- Non-power-of-two Kronecker RHTs: $D = 2560 = \text{Paley } H_{20} \otimes \text{FWHT}_{128}$ and $D = 9728 = \text{Williamson } H_{76} \otimes \text{FWHT}_{128}$ (
6,176 Bconstant tables;FP32round-trip max abs diff0.0/ $1.0 \times 10^{-6}$;DualModeMode-1GEMV andMode-2GEMM cosines0.9999921–1.0000001). - Zero-duplication tied
3.000-bpwvocabulary: Quantized once (145,858,560 B = 0.1358 GiB,0Lex -> Q4mismatches,24.5538 dBcleanlm_headH-SQNR,4.27 s) and shared via a singleuint8buffer acrossembed_tokensandlm_head(0 Bstorage duplication) alongside145BF16RMSNormtensors (392,192 B). Body encoding +BlockAlign+Q4transcode takes41.61 s(2.685) and43.58 s(2.250) after12.59 sshared 36-layer clean-Hessian / RHT setup (1.713 sdisjointcal_bal8KV4scale calibration folded at+0 Bintoq_norm / k_norm, where Layer-0 post-RoPE key pair RMS reaches99.34xmax/median).
Headline complete-system operating point (BW16_2685_4B + adapted RotateKV vs. BW16_2250_4B + KV16_BF16):
- Live
bs = 16GSM8K-256serving memory & accuracy:2.0329 GiBvs.3.7948 GiBreconstructed serving VRAM (-1.7619 GiB,-46.43%; incremental allocated peaks2.0142vs.3.7843 GiB; absolute process peaks2.0454vs.3.8155 GiB), while answering212/256(82.81%) vs.176/256(68.75%) correctly (+36items /+14.0625 pp[+8.5938, +19.9219], exact two-sided McNemar48:12, $p = 3.2 \times 10^{-6}$)—because droppingQwen3-4Bweights from2.685to2.250 bpwsaves only0.1836 GiBof weights while incurring a steep-13.6719 ppGSM8Kdrop even underKV16(211vs.176, $p = 1.57 \times 10^{-5}$). It also resolves all6/6L = 2,048WikiText-2 and C4 NLL / Teacher-KL contrasts (WT2 KL: 0.1914vs.0.2724;C4 KL: 0.1868vs.0.2707), with positive but statistically unresolvedARC-Challengedifferences (524vs.505 / 1,172raw,+1.6212 pp[-0.9386, +4.0956],123:104, $p = 0.232131$;553vs.537 / 1,172norm,+1.3652 pp[-1.1092, +3.8396],116:100, $p = 0.3074275$). - Direct fixed-
B = 16, L = 2,048decode step latency (separate from whole-GSM8Kwall time): Directly timing adaptedRotateKVin the fixed-(B, L)Stage-2 harness gives34.745 ms/step(460.49 tok/s,2.5899 GiBreconstructed serving VRAM) vs.39.914 ms/step(400.86 tok/s,5.7135 GiBreconstructed serving VRAM) for2.250 + KV16(-12.95%step time,-54.67%serving VRAM) and40.998 ms/step(390.26 tok/s,5.8971 GiB) for2.685 + KV16(-15.25%step time,-56.08%serving VRAM). Whole-GSM8K-256evaluation wall times (133.20 svs.196.05 s/198.24 s) are reported separately because they include prompt prefill and differing batch stopping lengths. - Measured quality costs & dense-retention gaps:
- Within-tier
KV4costs: Against the same2.685weight body (3.9785 -> 2.0329 GiBGSM serving VRAM),KV4reducesARC-Challengeraw accuracy (524vs.557 / 1,172,-2.8157 pp[-4.3515, -1.2799],24:57, $p = 0.0003173$ for adaptedRotateKV;537vs.557 / 1,172,-1.7065 pp[-3.2423, -0.1706],32:52, $p = 0.0375295$ forCauchySmoothRotate), whileGSM8K-256(+0.3906 pp/-0.3906 pp, $p = 1.0$) andARC-Challenge acc_norm(-1.2799 pp, $p = 0.1006437$;-0.5119 pp, $p = 0.5613807$) remain unresolved (not certifying preservation). Within2.250, adaptedRotateKVshows resolved losses on all three task metrics (GSM8K:-6.2500 pp, $p = 0.0440465$;ARC raw:-2.1331 pp, $p = 0.022298$;ARC norm:-2.2184 pp, $p = 0.0087809$), andCauchySmoothRotateshows a resolvedARC normloss (-1.7918 pp, $p = 0.0220335$). - Retention vs.
Dense_BF16_4B: All six quantized arms incur statistically resolved losses againstDense_BF16_4B(228/256 [89.06%]GSM8K;594/1,172 [50.68%]ARC raw;633/1,172 [54.01%]ARC norm,10.1252 GiBlive GSM serving VRAM):2.685arms lose-6.2500to-7.0312 pponGSM8K($p \le 0.013853$) and-3.1570to-5.9727 ppraw /-5.5461to-6.8259 ppnorm onARC;2.250arms lose-20.3125to-26.5625 pponGSM8K($p \le 5.75 \times 10^{-11}$) and-7.5939to-9.7270 ppraw /-8.1911to-10.4096 ppnorm onARC.
- Within-tier
CauchySmoothRotatevs. adaptedRotateKV:CauchySmoothRotateretains itsL = 2,048corpus advantage (8/8resolved full and[8:32]Teacher-KL CIs over adaptedRotateKV, cutting excess KL by-26.2%to-36.4%), with no demonstrated head-to-head task advantage over rotation alone (all 6 head-to-head task contrasts unresolved, $p \ge 0.2181$). Note that although2.685 + CauchySmoothRotatevs.2.250 + KV16is nominally resolved onARC-Challenge acc_raw(537vs.505,+2.7304 pp[+0.1706, +5.2901], $p = 0.0415941$) whereas2.685 + RotateKVvs.2.250 + KV16is unresolved (+1.6212 pp, $p = 0.232131$), a nominally resolved contrast against a third baseline alongside an unresolved one is not evidence that the two transforms differ—their direct paired contrast (537vs.524,+1.1092 pp[-0.5119, +2.7304],54:41, $p = 0.2180537$) is unresolved.
Because Qwen3-4B shares Qwen3-8B's 36 x 8 x 128 KV geometry above a ~2.1x smaller weight body, the larger fractional memory saving is structurally expected; this screen supports a useful tested quality–memory–latency trade-off within Qwen3 (8B -> 4B) and does not establish general quality preservation or a new cache-quantization principle.
Credited Baselines, Execution Paths, & Memory Definitions
- Credited KV4 Pre-Conditioning Baselines: As in r/post_training_quantization/post/27, we credit QServe SmoothAttention (Lin et al.,
arXiv:2405.04532v2, §IV-B) for RoPE-compatible paired(Q, K)scaling, RotateKV (Su et al.,arXiv:2501.16383v2, §4.2) for post-RoPEFWHT-128rotation, and SQuat (Wang et al.,arXiv:2503.24358v2) for query-aware key error analysis, labeling our comparative cache arms as adapted QServe / RotateKV / SQuat baselines. - Execution Paths & Preflight Diagnostics:
q_len = 1decode (GSM8K-256bs = 16& Stage-2 decode benchmark) runsMode-1compressed-weight GEMV (M <= 16) +PackedKV4Cache+ fusedsm_120GQA-4 Split-K Flash-Decode (0full-contextBF16KV unpack).use_cache = Falsemulti-token scoring (ARC-Challengebs = 64&L = 2,048corpus/KL) runsMode-2BF16GEMM + real CUDApack_and_unpack_nocache_real+ causalBF16SDPA (60 MiBw_rotscratch attached only inMode-2/ prefill and detached before decode).- Preflight checks (
5.09 s):KV16static-mask vs. standard-mask 8-step token exact matchTrue(cos = 0.9999999,max_abs_diff = 0.0); 36-layer same-mode (Mode-2) step-0PackedKV4Cachevs.use_cache=Falseparitycos = 1.0,max_abs_diff = 0.0,top1 = 4/4; disclosed cross-mode step-7 diagnostic (Mode-1FP32 GEMV + Split-K Flash-Decode vs.Mode-2BF16 GEMM +pack_and_unpack+ SDPA on common prefixes)cos = 0.9984941,max_abs_diff = 1.0,common_prefix_next_token_top1_matches = 32/32(mixed-precision rounding is plausible without proving sole cause or excluding a kernel defect, and0.990is only a coarse sanity bound).
- Three GPU Memory Metrics (
mem_before_model = 33,496,576 Bfor compressed models):inc:max_memory_allocated - mem_before_model.reconSrv:inc + applicable pre-allocated shared serving buffers(+11,321,344 B[10.80 MiB]Qwen3-4BBW16 weight scratch forKV16_BF16decode;+20,107,264 B[19.18 MiB] including8,785,920 B[8.38 MiB] Split-K decode workspace forKV4_*decode;+622,592 Bx_rot_tc32_flatduring prefill).absProc:mem_before_model + inc.
Table 1: Direct Fixed-(B, L = 2,048) Decode Latency/VRAM & 32 x 2,048 Corpus / Teacher-KL (Qwen3-4B)
(Timing note: Unlike r/post_training_quantization/post/27, adapted RotateKV was directly timed alongside KV16_BF16 and CauchySmoothRotate at every (B, L) cell. The near-equal RotateKV and CauchySmoothRotate step times below—e.g., 34.745 vs. 34.793 ms/step on 2.685 and 33.404 vs. 33.398 ms/step on 2.250 at B = 16, L = 2,048—are observed single-run point measurements; they do not establish timing equivalence or characterize run-to-run variation. For reference, Dense_BF16_4B [$T^* = 1.28$] achieves WT2-32 Raw/Fit NLL 2.6031 / 2.3913 and C4-32 Raw/Fit NLL 3.0323 / 2.8645.)
| Weight Tier | KV Mode | B=8, L=2048 inc / reconSrv / absProc (GiB) | B=16, L=2048 inc / reconSrv / absProc (GiB) | B=8 / B=16, L=2048 Decode (ms/step, B=16 tok/s) | WT2-32 Fit NLL / KL | C4-32 Fit NLL / KL |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
| BW16_2250_4B | KV16_BF16_SDPA | 3.4059 / 3.4164 / 3.4370 | 5.7029 / 5.7135 / 5.7341 | 31.537 / 39.914 ms (400.86) | 2.5703 / 0.2724 | 3.0608 / 0.2707 |
| BW16_2250_4B | KV4_Plain_AsymG64 | (not sep. timed) | (not sep. timed) | (not sep. timed) | 4.2360 / 2.1628 | 4.7144 / 2.1423 |
| BW16_2250_4B | Adapted KV4_QServe | (not sep. timed) | (not sep. timed) | (not sep. timed) | 2.6188 / 0.3276 | 3.1020 / 0.3242 |
| BW16_2250_4B | Adapted KV4_RotateKV | 1.7419 / 1.7607 / 1.7731 | 2.3875 / 2.4063 / 2.4187 | 28.506 / 33.404 ms (478.99) | 2.5913 / 0.3047 | 3.0888 / 0.3098 |
| BW16_2250_4B | KV4_CauchySmoothRotate | 1.7419 / 1.7607 / 1.7731 | 2.3875 / 2.4063 / 2.4187 | 28.515 / 33.398 ms (479.06) | 2.5869 / 0.2962 | 3.0771 / 0.2955 |
| BW16_2685_4B | KV16_BF16_SDPA | 3.5895 / 3.6000 / 3.6207 | 5.8866 / 5.8971 / 5.9178 | 30.923 / 40.998 ms (390.26) | 2.4882 / 0.1583 | 2.9705 / 0.1513 |
| BW16_2685_4B | KV4_Plain_AsymG64 | (not sep. timed) | (not sep. timed) | (not sep. timed) | 3.8991 / 1.7616 | 4.3665 / 1.7810 |
| BW16_2685_4B | Adapted KV4_QServe | (not sep. timed) | (not sep. timed) | (not sep. timed) | 2.5326 / 0.2153 | 3.0143 / 0.2058 |
| BW16_2685_4B | Adapted KV4_RotateKV | 1.9256 / 1.9443 / 1.9568 | 2.5712 / 2.5899 / 2.6024 | 28.038 / 34.745 ms (460.49) | 2.5079 / 0.1914 | 2.9912 / 0.1868 |
| BW16_2685_4B | KV4_CauchySmoothRotate | 1.9256 / 1.9443 / 1.9568 | 2.5712 / 2.5899 / 2.6024 | 28.038 / 34.793 ms (459.86) | 2.4979 / 0.1818 | 2.9853 / 0.1750 |
(At B = 1, L = 2,048, single-point decode timings are 22.402 -> 21.938 [RotateKV] / 21.800 ms [Cauchy], 1.5901 -> 1.3794 GiB reconSrv on 2.685_4B, and 22.258 -> 21.590 / 21.856 ms, 1.4065 -> 1.1957 GiB reconSrv on 2.250_4B. At B = 16, L = 2,048, prefill+pack on 2.685_4B runs 2,386.94 ms [KV16, 8.2694 GiB reconSrv] vs. 2,435.40 ms [RotateKV] / 2,438.27 ms [Cauchy, +2.0%, 5.1631 GiB reconSrv, -37.6%]. The identical KV16 < CauchySmoothRotate < RotateKV < QServe << Plain ordering holds on the [8:32] 24-window slice.)
Table 2: Seven-Arm Behavioral & Live GSM8K-256 Serving Matrix (Qwen3-4B)
(End-to-end GSM dt includes prompt prefill and differing batch stopping lengths across the 16 batches; see Table 1 for fixed-(B, L) decode step latency.)
| System / Weight Tier | KV Mode | GSM8K-256 (N=256) | Uncapped GSM / Caps | agree_dense4B | Live GSM VRAM: inc / reconSrv / absProc (GiB) | Mean Steps / GSM dt | ARC-C acc_raw (N=1172) | ARC-C acc_norm (N=1172) |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Dense_BF16_4B (7.4924 GiB) | KV16_BF16_SDPA | 228 (89.06%) | 228/247 (92.31%) / 9 | 256/256 (100%) | 10.1252 / 10.1252 / 10.1372 | 224.5 / 116.14 s | 594 (50.68%) | 633 (54.01%) |
| BW16_2685_4B (1.2715 GiB) | KV16_BF16_SDPA | 211 (82.42%) | 210/247 (85.02%) / 9 | 209/256 (81.64%) | 3.9679 / 3.9785 / 3.9991 | 220.5 / 198.24 s | 557 (47.53%) | 568 (48.46%) |
| BW16_2685_4B (1.2715 GiB) | Adapted KV4_RotateKV | 212 (82.81%) | 211/248 (85.08%) / 8 | 208/256 (81.25%) | 2.0142 / 2.0329 / 2.0454 | 219.5 / 133.20 s | 524 (44.71%) | 553 (47.18%) |
| BW16_2685_4B (1.2715 GiB) | KV4_CauchySmoothRotate | 210 (82.03%) | 210/245 (85.71%) / 11 | 208/256 (81.25%) | 2.0142 / 2.0329 / 2.0454 | 226.5 / 137.00 s | 537 (45.82%) | 562 (47.95%) |
| BW16_2250_4B (1.0879 GiB) | KV16_BF16_SDPA | 176 (68.75%) | 176/243 (72.43%) / 13 | 174/256 (67.97%) | 3.7843 / 3.7948 / 3.8155 | 223.0 / 196.05 s | 505 (43.09%) | 537 (45.82%) |
| BW16_2250_4B (1.0879 GiB) | Adapted KV4_RotateKV | 160 (62.50%) | 160/248 (64.52%) / 8 | 160/256 (62.50%) | 1.8305 / 1.8492 / 1.8617 | 225.0 / 131.22 s | 480 (40.96%) | 511 (43.60%) |
| BW16_2250_4B (1.0879 GiB) | KV4_CauchySmoothRotate | 169 (66.02%) | 169/244 (69.26%) / 12 | 165/256 (64.45%) | 1.8305 / 1.8492 / 1.8617 | 222.0 / 129.63 s | 495 (42.24%) | 516 (44.03%) |
Table 3: Paired Contrasts (10,000-Draw Paired Bootstrap Seed 20261118 + Exact Two-Sided McNemar, Nominal Unadjusted Intervals)
| Contrast ($A - B$) | GSM8K-256 $\Delta$ pp [95% CI] ($b:c,\; p$) | ARC-C acc_raw $\Delta$ pp [95% CI] ($b:c,\; p$) | ARC-C acc_norm $\Delta$ pp [95% CI] ($b:c,\; p$) |
| :--- | :---: | :---: | :---: |
| Cross-tier: 2685 + RotateKV - 2250 + KV16 | +14.06 [+8.59, +19.92] (48:12, $3.20\text{e-}6$) | +1.62 [-0.94, +4.10] (123:104, $0.2321$) | +1.37 [-1.11, +3.84] (116:100, $0.3074$) |
| Cross-tier: 2685 + Cauchy - 2250 + KV16 | +13.28 [+7.42, +19.14] (48:14, $1.74\text{e-}5$) | +2.73 [+0.17, +5.29] (132:100, $0.0416$) | +2.13 [-0.26, +4.52] (117:92, $0.0967$) |
| (Ref. weight gap: 2685 + KV16 - 2250 + KV16) | +13.67 [+7.81, +19.53] (50:15, $1.57\text{e-}5$) | +4.44 [+1.88, +7.00] (146:94, $9.55\text{e-}4$) | +2.65 [+0.09, +5.12] (132:101, $0.0491$) |
| Within 2.685: Cauchy - RotateKV | -0.78 [-5.47, +3.91] (17:19, $0.8679$) | +1.11 [-0.51, +2.73] (54:41, $0.2181$) | +0.77 [-0.85, +2.39] (52:43, $0.4119$) |
| Within 2.250: Cauchy - RotateKV | +3.52 [-2.34, +9.38] (34:25, $0.2976$) | +1.28 [-0.68, +3.24] (75:60, $0.2281$) | +0.43 [-1.28, +2.13] (52:47, $0.6879$) |
| Within 2.685: RotateKV - KV16_BF16 | +0.39 [-3.52, +4.30] (14:13, $1.0000$) | -2.82 [-4.35, -1.28] (24:57, $3.17\text{e-}4$) | -1.28 [-2.73, +0.09] (29:44, $0.1006$) |
| Within 2.685: Cauchy - KV16_BF16 | -0.39 [-4.30, +3.52] (13:14, $1.0000$) | -1.71 [-3.24, -0.17] (32:52, $0.0375$) | -0.51 [-1.96, +0.94] (34:40, $0.5614$) |
| Within 2.250: RotateKV - KV16_BF16 | -6.25 [-12.11, -0.78] (20:36, $0.0440$) | -2.13 [-3.92, -0.43] (43:68, $0.0223$) | -2.22 [-3.75, -0.68] (33:59, $0.0088$) |
| Within 2.250: Cauchy - KV16_BF16 | -2.73 [-8.20, +2.73] (24:31, $0.4188$) | -0.85 [-2.47, +0.68] (39:49, $0.3374$) | -1.79 [-3.24, -0.34] (28:49, $0.0220$) |
| Retention: 2685 + KV16 - Dense_BF16_4B | -6.64 [-11.33, -1.95] (10:27, $0.0076$) | -3.16 [-5.38, -0.94] (68:105, $0.0060$) | -5.55 [-7.85, -3.24] (65:130, $3.80\text{e-}6$) |
| Retention: 2685 + RotateKV - Dense_BF16_4B | -6.25 [-10.94, -1.56] (11:27, $0.0139$) | -5.97 [-8.28, -3.67] (58:128, $3.00\text{e-}7$) | -6.83 [-9.13, -4.52] (56:136, $7.27\text{e-}9$) |
| Retention: 2685 + Cauchy - Dense_BF16_4B | -7.03 [-11.72, -2.34] (10:28, $0.0051$) | -4.86 [-7.08, -2.56] (64:121, $3.36\text{e-}5$) | -6.06 [-8.36, -3.75] (60:131, $3.00\text{e-}7$) |
| Retention: 2250 + KV16 - Dense_BF16_4B | -20.31 [-26.17, -14.45] (8:60, $5.75\text{e-}11$) | -7.59 [-10.15, -5.03] (80:169, $1.74\text{e-}8$) | -8.19 [-10.67, -5.63] (70:166, $3.60\text{e-}10$) |
| Retention: 2250 + RotateKV - Dense_BF16_4B | -26.56 [-32.42, -20.70] (5:73, $1.50\text{e-}16$) | -9.73 [-12.37, -7.00] (75:189, $1.59\text{e-}12$) | -10.41 [-12.97, -7.76] (63:185, $4.23\text{e-}15$) |
| Retention: 2250 + Cauchy - Dense_BF16_4B | -23.05 [-29.30, -17.19] (8:67, $1.01\text{e-}12$) | -8.45 [-11.09, -5.89] (77:176, $4.36\text{e-}10$) | -9.98 [-12.63, -7.34] (64:181, $4.05\text{e-}14$) |
(All 9 paired GSM8K-256 cap-rate contrasts across quantized arms are statistically unresolved, $p \ge 0.2668$.)
Scope & Measurement Qualifications
- Why the cross-tier trade-off sharpens from
8Bto4B: BecauseQwen3-4B(3.63Bbody parameters) retainsQwen3-8B's36 x 8 x 128KV cache while cutting body weights ~2.1x, pushing weights from2.685down to2.250 bpwsaves only0.1836 GiB(vs. 0.3516 GiBon8B) while crossing a much steeper weight-quantization cliff (-13.67 ppvs.-5.86 pponGSM8K-256). Compressing theBF16KV cache (-1.9456 GiBon liveGSM8K,-3.3072 GiBatB = 16, L = 2,048) therefore buys substantially more resident memory headroom than aggressive sub-2.50-bpwweight compression while avoiding the2.250accuracy collapse—at the explicit cost of a-2.82 pp(RotateKV) /-1.71 pp(Cauchy)ARC-Challenge acc_rawreduction relative to2.685 + KV16and-6.25to-7.03 ppGSM8Kloss relative toDense_BF16_4B. - Panel exposure, single-run sampling, & nominal uncertainty: All evaluations use one deterministic/greedy execution per cell on exposed development/comparison panels (
WT2-32,C4-32,GSM8K-256ids 128..383, andARC-Challenge-1172), and all95%bootstrap intervals and exact two-sided McNemar $p$-values are nominal (unadjusted across multiple comparisons). - Unmeasured long-context downstream behavior & cross-family scope: Downstream task retention at long context ($L \ge 2,048$, including long-horizon
M256reasoning) remains unmeasured (L = 2,048evidence here is teacher-forced corpus NLL/KL), and transfer is demonstrated only within theQwen3family (8B -> 4B).