Post-training quantization: smaller models with preserved capability

AI · Machine learning

Post-training quantization: smaller models with preserved capability

Motivation and research question

How far can a pretrained language model be compressed while preserving its capabilities? Post-training quantization (PTQ) starts from pretrained weights and maps weights—and often activations—to low-bit representations. Direct four-bit quantization cuts memory traffic and storage, but coarse grids and activation outliers can degrade language modeling and reasoning. Methods may optionally use local calibration or limited fine-tuning to recover quality.

Two recent papers start from the same idea: store most of each weight matrix in four bits, then add a small correction to recover information lost during quantization. The correction is represented by two thin matrices; their shared dimension, called the rank, controls how many correction terms can be stored.

1. H-SVDQuant: Deflating the Hessian asks how much quality a very small correction can recover. It chooses the correction and the quantization together, using example inputs to identify which errors matter to the model. Technically, it combines channel rescaling, a calibration-weighted low-rank correction, and second-order residual quantization. A rank-4 correction matches or surpasses rank-32 SVDQuant on several diffusion transformers and on Qwen3-8B, with substantially lower offline calibration time.
2. BiLoCo: Binary Low-Rank Corrections for Post-Training Quantization asks whether storing more correction terms at lower precision is a better use of memory. It replaces the correction factors' 16-bit values with one-bit signs (+1 or −1), plus scalar coefficients. Before coefficient overhead, the same vector storage holds 512 binary terms instead of 32 BF16 terms. This recovers more information from LLM weights and improves or matches the comparison methods' perplexity. The paper also develops a fused B300 kernel: the binary correction uses CUDA cores while the main four-bit multiplication uses Tensor Cores.

The primary goal is to reduce total stored model size while preserving language-model capability. Calibration and inference cost determine how practical a method is. A strong contribution could demonstrate higher model quality at an equal total byte budget, smaller storable model size at equal quality, or a calibration method that consistently improves this trade-off across models.

Possible research directions

The first four directions build on questions tested directly in H-SVDQuant and BiLoCo. The final two extend their methods to broader compression settings. Other approaches are welcome, including methods without a separate correction branch. Four-bit weights and activations are starting baselines; neither is a restriction.

1. Reduce correction storage while retaining model quality. Aim to match the perplexity and downstream accuracy of the supplied BiLoCo or H-SVDQuant baseline with fewer total stored bytes, or improve quality at the same size. H-SVDQuant uses a small high-precision correction; BiLoCo uses many one-bit correction terms. Explore alternative precisions, ranks, or mixtures of the two. The aim is a smaller complete model, including its corrections. (H-SVDQuant §5–6; BiLoCo §3.1, §4.1)

2. Recover more capability from the same compressed representation. Aim to lower perplexity and improve downstream accuracy without increasing the stored model size. H-SVDQuant shows that choosing the correction and quantization together can outperform fitting them separately. One possible extension is to choose BiLoCo's binary corrections using calibration inputs, giving priority to errors that affect model outputs rather than treating every weight error equally. (H-SVDQuant §3, §6; BiLoCo §3.2–3.3)

3. Reduce the storage needed to handle outliers. Unusually large weights or activations can force a quantizer to waste precision on a few values. Aim to preserve the baseline's quality with fewer correction terms, fewer high-precision exceptions, or fewer bits by improving channel rescaling, rotations, or block scaling. H-SVDQuant reports strong gains from channel rescaling; BiLoCo finds that rotations perform worse than rescaling in its one-model, one-seed NVFP4 experiment. (H-SVDQuant §3.2, §6 Table 7; BiLoCo §6, Appendix A.3)

4. Make accurate compressed models inexpensive to run. Aim to retain the quality and storage benefits of a correction while reducing its inference latency or peak GPU memory. BiLoCo demonstrates low correction overhead for selected B300 projection kernels; extending that benefit to consumer GPUs, more layer shapes, or larger batches would make the representation more useful. A useful outcome would be lower memory use or faster inference for a complete model on the RTX 5090. Calibration speed is another useful practical gain when compressed size and quality are maintained. (BiLoCo §3.4, §4.2; H-SVDQuant §5 Table 2)

5. Save bytes by allocating precision where it matters most. Aim to produce a smaller model with quality competitive with a uniform-bit, uniform-rank baseline. Some layers may need more correction or precision, while others can tolerate much less. Explore layer-wise bit widths, correction ranks, or selective high-precision components under one total storage budget. A rule that transfers across models would be especially useful. (BiLoCo Appendix A.1; H-SVDQuant Appendix C.3, Appendix I)

6. Compress below the four-bit baselines while remaining competitive. Aim to reduce total model storage using 2-bit, 3-bit, ternary, codebook, or mixed-precision representations while retaining perplexity and downstream accuracy close to the supplied four-bit baselines. These extend beyond the papers' main four-bit settings. Corrections are optional, but their cost must be included: a two-bit main matrix is useful only if its extra components leave the whole model smaller. (H-SVDQuant; BiLoCo)

Common starting point and verified baselines

Qwen/Qwen3-8B on one NVIDIA RTX 5090 (32 GB) provides a common starting point; other models and scales are welcome. Download baseline code for the setup, fixed dataset splits, and commands.

The rows below are verified starting recipes on Qwen3-8B, not matched-storage comparisons. All runs evaluate identical raw WikiText-2 test tokens (2,048-token windows) and 256 fixed C4 validation windows using SDPA attention:

| Method | WikiText-2 perplexity ↓ | C4 perplexity ↓ | |---|---:|---:| | BF16 reference | 9.725 | 12.296 | | NVFP4 | 10.097 | 12.699 | | Independent SVD rank 32 + NVFP4 | 10.222 | 12.729 | | BiLoCo rank 512 + NVFP4 | 10.059 | 12.635 | | Independent H-SVDQuant rank 4 + INT4 | 10.473 | 13.241 |

The H-SVDQuant row uses our independent implementation; the BiLoCo row uses the authors’ code adapted to this setup. These commands evaluate accuracy via simulated quantization in BF16 arithmetic; the complete accuracy workflows (loading, calibration, quantization, and both perplexity evaluations, excluding downloads) take about 5.8 minutes (350.52 s) for BiLoCo and 10.8 minutes (648.63 s) for H-SVDQuant on one RTX 5090. Actual packed sizes for these runs have not been measured. Because the two recipes differ in number format (16-element NVFP4 with FP8 block scales vs. group-128 INT4), calibration set, correction budget, and output-head precision (BiLoCo quantizes lm_head; H-SVDQuant keeps it in BF16), use matched controls to isolate a proposed improvement.

Task materials

Type

open-ended

Solve status

open

Access

Invite-only