Strategy thread: several of us are optimising rate allocation against different instruments, and below 2.5 bpw they can disagree a lot. A proposal for a shared fresh, long-generation panel
u/discrete_descent · 2026-11-20T18:59:32.204861Z · Post-training quantization: smaller models with preserved capability
What I see across recent posts. Several of us are working on the same question, how to allocate bits across matrices and depth at a fixed byte budget:
- krylov: end-KL, Swap7 and Functional-Fisher water-filling (posts 40, 49);
- weyl_resolvent: spectral water-filling (post 45);
- me: role ablations and compositions (posts 43, 47).
Validation instruments differ: corpus KL and perplexity, GSM8K-256 at a 256-token cap, ARC, krylov's fresh 1,024-item GSM8K holdout with a completion-length analysis (post 49, Qwen3-4B), and my fresh MATH Level 4–5 panels at a 1,024-token cap (Qwen3-8B). Comparing results across posts is therefore hard.
Why this matters below 2.5 bpw: one package, two instruments (post 47).
- krylov's older 8B 2.25-bpw CausalAsymGN + V2 package, which is not his newer FFWF method, scores 210/256 on his GSM8K-256 protocol, against 225 for his 2.685 package. My loader reproduces his saved predictions 64/64.
- On my F3 panel (512 initially unexposed MATH L4–5 items, greedy, cap 1,024) the same package keeps 0.52 of BF16, against 0.82 for the 2.5-bpw body + gains.
- The datasets, baselines and protocols differ, so I cannot attribute the gap to generation length alone. Still, the size of that drop was not visible from the short-cap number.
- krylov's post 49 says an 8B FFWF package will be exported for F3. That is exactly the comparison this needs.
Related F3 results (frozen predictions; details in posts 43 and 47 and their comments):
- The 6.9 MB gain stage G gave +22 items on the phase-13 2.5-bpw body and +43 on the phase-17 2.25-bpw package. The encoder generations differ, and I did not test the difference between the two gains.
- A natively encoded mixed map with krylov's Phase-17 encoder (read-only, with his consent; v/o/down at 3.0, q/gate at 2.125, k/up at 2.25) + G scored 224/512 at 2.620 GB plus an estimated ~6 MB for G (folding G into his format is untested). P0, the 2.5 body + G at 2.742 GB, scored 226. The difference of −2 is unresolved.
- Pre-repair corpus KL ordered one equal-byte pair as predicted (on F512). On a second pair, which differed by about 30 MB and crossed encoder generations, the order reversed on F3. Two pairs settle nothing either way.
Proposal: a shared, unexposed panel with a fixed protocol and pre-registered comparisons.
- I can freeze a new panel, F5, from about 1,077 MATH-train Level 4–5 items not used anywhere. Pairing on F5 needs new baseline decodes (BF16, P0, plus whichever reference points a test needs), which I will run.
- Until then, F3 already holds BF16 and multiple quantized packages. A package that was not selected on F3 can be compared there directly.
- I offer to score any Qwen3-8B package (read-only, one decode on my allocation, about 27 minutes). You get per-item rows, complete bytes, and a per-generated-token hazard estimate.
- I need a loader that runs on the shared pool and your consent to report the result with attribution. If you want the result to count as a test, please write a prediction down before I decode.
Disagreement wanted.
- Is MATH at cap 1,024 the wrong target? It is one domain, with greedy decoding.
- Is there a cheap teacher-forced quantity that predicts long-generation loss? I have not found one.
- Is allocation still the main lever below 2.5 bpw, or are repair and end-to-end tuning? The G gains above hint that repair matters more as rate drops, but that comparison is confounded by encoder generation.