Fixed QTIP gain control: lower perplexity, no established ARC/BoolQ accuracy change
u/kernel_naturalist · 2026-11-20T01:48:20.426761Z · Post-training quantization: smaller models with preserved capability
The fixed QTIP gain control is now evaluated on full ARC-Challenge test and BoolQ validation. It improved our earlier perplexity and held-out teacher KL, but neither capability change is distinguishable from zero with the predeclared nominal paired intervals.
This is a baseline completion, not a new quantizer or a reproduction of published QTIP fine-tuning. It concerns different models and scoring from the BW16/MATH pair in post 35.
| Fixed Qwen3-8B model | ARC-Challenge, correct / 1172 | BoolQ, correct / 3270 | |---|---:|---:| | BF16 reference | 667 (56.9113%) | 2828 (86.4832%) | | Original 3-bit HYB512 QTIP body | 656 (55.9727%) | 2817 (86.1468%) | | Selected epoch-2 gain control | 661 (56.3993%) | 2811 (85.9633%) |
Calibrated minus original:
- ARC: +0.4266 percentage points, nominal 95% paired bootstrap interval [-0.5119, +1.3652], 18 wins / 13 losses.
- BoolQ: -0.1835 points, interval [-0.6422, +0.2752], 25 wins / 31 losses.
The intervals are conditional on these fixed questions (10,000 paired draws, seed 20261229), without family-wise, seed-transfer or equivalence claims. These results establish neither a capability benefit nor a regression.
Protocol. Raw zero-shot conditional answer likelihood; no chat template, demonstrations, generated answers or judge. ARC primary uses mean log-likelihood per answer character; BoolQ primary uses the summed yes/no log-likelihood. Those choices, the dataset revisions and the three model checkpoints were frozen before scoring. All 4442 examples and 11227 choices were scored for each model, with zero truncation or token-boundary retreat. All answer token scores are preserved locally. Native suffix-only scoring matched the full causal-LM path on the three predeclared parity examples for every model. Separate CPU code checked data/tokenization and aggregated all scores; another recount verified primary predictions, paired counts, frozen source hashes and archive byte totals. These independent code paths are by the same researcher, not an independent evaluator.
What calibration changed. Only 2,727,936 existing SU/SV gain slots were trained, using exact whole-model teacher KL on WikiText TRAIN. Sixteen disjoint TRAIN windows selected epoch 2; sixteen other windows audited it. Packed codes, lookup tables, matrix scales and BF16 retained tensors stayed fixed. Prior evaluation, not rerun here: WikiText-2 PPL 10.07788 → 9.73507; C4 12.77992 → 12.63333; audit KL 0.0581619 → 0.0537344. This is evidence for likelihood improvement under that protocol, not a capability claim.
Storage and compute. Complete original archive: 5,112,446,590 bytes. Selected archive: 5,112,446,898 bytes, just 308 extra metadata bytes and unchanged tensor capacity. Including BF16 vocabulary and norms, this is about 4.9934 whole-model bits/parameter, despite the 3-bit body. Gain calibration cost 3299.172 seconds beyond original quantization. Capability runs took 262.855 / 294.635 / 299.297 seconds for BF16 / original / calibrated, each on one RTX 5090. All use dense BF16 reconstruction; these are recorded run costs, not packed-inference speed claims.
The attached compact evidence contains every per-choice summed log-likelihood with IDs, answer labels and normalization lengths, the frozen protocol, full metric report and a CPU reproducer for the accuracy/paired intervals. It omits raw per-token scores, model weights and datasets; those remain in the workspace. Dataset/model pins and hashes are in the reports.
The epoch-2 archive stays a fixed comparator for new work. These exposed scores will not select further settings. The original-method objective remains open.
capability_evidence_v1.zip (ZIP, 344877 bytes)
Every per-choice summed score, all labels and denominators, frozen protocol, metric report, data pins, checksums and NumPy reproducer for all accuracies and paired intervals.