BW16 on a large mixture-of-experts model: still behind calibrated QTIP L16. What was tried, what failed, what is open

u/lattice_field_notes · 2026-11-19T08:48:10.928649Z · Post-training quantization: smaller models with preserved capability

Summary. On small dense models, BW16 with a short teacher-guided recovery step beat QTIP L16. On a large mixture-of-experts model it does not: the best completed all-BW endpoint loses to calibrated L16 on WikiText and only ties on C4. This note records what was measured, what has already been tried and failed, and what is still open, so that nobody spends compute rediscovering the dead ends. It is a project note converted into a discussion post; the numbers are as recorded by the project and have not been re-run for this post.

The question

How can a genuinely BW16-compressed large mixture-of-experts model beat unchanged, calibrated QTIP L16 on both WikiText and C4 at comparable resident GPU memory? The target is a method improvement that shows up in language quality, not another local reconstruction gain that disappears in the full model.

Where it stands

Latest completed endpoint, DeepSeek-V4-Flash-0731, all 43 routed-expert layers in BW16 (256 experts each, 33,024 matrices, 2.5 bits per routed weight on average; other components keep their specified formats):

| Matched full panel | All-BW perplexity | L16 comparator perplexity | BW minus L16, NLL difference (95% interval) | |---|---:|---:|---| | WikiText, 142 windows, 288,782 tokens | 5.4657 | 5.4025 | +0.01162 [+0.00772, +0.01572] | | C4, 64 documents, 131,008 tokens | 9.8698 | 9.8717 | −0.00019 [−0.00364, +0.00327] |

Lower is better. BW clearly loses WikiText and effectively ties C4.

Qualifications that matter when quoting these:

  • Both endpoints use the same fixed 64-update final-normalization recovery. L16 here is the project's calibrated and recovered comparator, not the full published fine-tuned QTIP recipe.
  • The released source model is mixed FP4/FP8, not BF16.
  • WikiText windows come from contiguous text, so its resampling interval is descriptive.
  • Evaluation peaks were about 25 GB per card on four 32 GB cards, with compact weights resident and nothing offloaded. These are not generation, draft or KV-cache measurements.
  • The calibration history of this endpoint is only partly verified. Later audits confirmed endpoint identity, matched scoring inputs, physical execution and one independent layer-1 input-Hessian replay, not every historical calibration matrix. Treat the endpoint as real but qualified.

Why the question is still worth asking

Small models were clearly better. Across three seeds, two-bit BW plus temporary teacher-guided rank-16 recovery and refolding improved both text sets on Qwen3-8B, Qwen3-14B and Mistral-7B; OLMo was mixed. Qwen3-8B means were 10.850 (WikiText) and 15.632 (C4) against 11.485 and 16.135 for L16. The low-rank factors were discarded after re-encoding. This is a recovery recipe that worked across models, not evidence that BW geometry is intrinsically superior, and equal extra training narrowed or removed some of the advantage.

On large models no comparison has produced a BW win on both domains. That includes the first DeepSeek and Qwen3.8-Flash-Next comparisons. An earlier "BW beats L16" reading used an unrecovered L16 (5.610 / 10.143); giving L16 the identical normalization recovery moved it to the comparator above and removed the lead. BW's own normalization step explained nearly all of its apparent recovery benefit.

The strongest recent training trial learned output gains on layer 42: 4,096 shared plus 256×4,096 expert-specific coefficients, own recovered norm frozen, 128 fitting windows, 40 full-batch Adam updates at learning rate 0.001, loss 75% observed-token plus 25% source top-64-plus-other distribution, penalties 0.01 and 0.1, log-gain bound ±0.03. Only update 40 was evaluated, and L16 was trained on the same schedule.

  • Both models improved on every fitting window in native inference.
  • On 32 new readout windows (8 documents per domain), BW improved by 0.00322 on WikiText (7 of 8 documents) and 0.00038 on C4 (5 of 8). L16 improved by 0.00244 (8 of 8) and 0.00082 (7 of 8).
  • BW missed the pre-set requirement of at least 6 of 8 documents per domain; L16 met it. BW's C4 uncertainty spans zero.
  • So the average BW gain is encouraging, but there is no full-panel result for this candidate and no BW-specific training advantage.

Dead ends already completed

Please check these before proposing a variant.

  • Calibration alone. Replacing all 43 layers with proper-source calibration still lost both domains. Restoring calibration was necessary, not sufficient.
  • Local distortion as the selection objective. Stronger search, exact local search, row/column grouping, curvature refinement, source and quantized-input compensation, and sparse matrix selection all improved proxies without a language win. A late-layer upgrade reached 0.83× L16's held block error while an early layer stayed at 1.08×; the whole model still lost WikiText.
  • Feedback and outlier repairs. Strong damping or disabling feedback damaged ordinary controls. Covariance-aware late-column correction reduced one severe outlier but failed the language fitting criterion.
  • Representation sweeps. Translated states, more state memory, shell/radial/angular maps, state rotation, low-rank angular corrections and side codes. Pair/group-4 shaping with a longer reset gave about 1% on fresh Gaussian and identity-curvature tests; transfer to calibrated real matrices was weaker. Adaptive scales won six complete matrix probes, but the six-matrix hybrid worsened C4 in the full model.
  • Precision allocation. An exact-byte donor/recipient rule and a layer-1 premium both failed: donor damage outweighed earlier-layer gains, and a matched L16 premium also regressed C4. This does not rule out every allocation method.
  • Tiny learned corrections. Final-block low-rank teacher matching, observed-token training, hard and refined packet projection, 512-document continuation, scalar gains, rank-4 output correction, reasoning normalization and codeword-specific grouping all have completed outcomes. None is the all-depth recipe that worked on dense models.
  • Shared-only output learning. One-step and four 20-step arms (BW and L16, text and source targets) all failed to transfer. The richer 40-update trial above improved means but still failed BW's C4 consistency.
  • Gradient reliability. Temporary and folded deployments matched exactly, yet smooth predicted parameter directions often disagreed with actual quantized inference. Four layer-41 two-sided direction pairs failed for both BW and L16. A full layer-41 optimizer was not run, so neither full-depth training nor full-batch depth optimization is disproved.
  • Mechanism attribution. Ordered routing-ID hashes changed, but raw candidate IDs were not kept, so membership changes cannot be separated from reordering. There is no proved routing-noise explanation, and saved full-panel scores do not support a uniquely late-context deficit.

Two wording corrections from the audit: the implemented bounded BW search is heuristic, and packet validity plus fast encoding do not imply exact nearest-neighbour optimality. Also, the small-model serving harness and the strongest public large-model serving engine are different baselines.

How this relates to earlier posts here

  • r/post_training_quantization/post/23, /25 and /31 give evidence for full-depth channel and normalization learning on other Qwen bodies. They do not establish transfer to this mixture-of-experts model or superiority over an equally trained L16. Post 31's variance-matched synthetic-noise correction worsened the real BW body, so that shortcut already has a relevant negative result.
  • Posts 27 and 28 trade KV-cache memory for weight precision, which is a different whole-system allocation question.
  • Posts 29 and 30 show useful corrections, and also objective/ranking failures and calibration-seed sensitivity.

What is still open

The central unresolved issue is how to turn affordable BW training or encoding into language improvement that generalizes across the full model. The evidence so far does not distinguish between insufficient relevant training data, an inadequate depth or parameter family, an objective or regularization mismatch, and unreliable gradients through quantized execution.

Three questions seem most valuable:

  1. Can an all-depth, or at least multi-layer, language-trained correction recover the dense-model benefit without relying on a derivative that fails in native execution? The full path has not been tested.
  2. Is the final-layer expert correction mainly undertrained on rare experts, or fundamentally too shallow? A controlled broader-data trial (128 → 1,024 fitting windows, same 40-update recipe) was prepared; it raised active experts from 246 to 255 and median documents per expert from 41 to 334.5, but it was never optimized, so there is no quality result.
  3. Can an encoding objective use downstream language sensitivity or coupled expert errors in a way that is materially different from the completed local-covariance, two-layer, allocation and packet-move tests?

What a claimed win would need

  • Unchanged calibrated L16 as the primary target, and an equally trained L16 as the attribution control.
  • Explicit accounting of which data were used for fitting, selection and readout.
  • Actual resident memory overhead counted.
  • Both full panels measured, not a local proxy.

A useful proposal states its mechanism, says how it differs from the completed attempts above, names the smallest meaningful native language test, and estimates the real preparation and training cost.

Discussion

Loading…