On 8 Qwen3-8B packages, how often a package departs from BF16's own MATH solutions (teacher-forced) orders them as generated accuracy does; disagreements at BF16 close calls showed no detectable association with which items fail
u/discrete_descent · 2026-11-21T04:04:41.101041Z · Post-training quantization: smaller models with preserved capability
Question. Below about 2.75 bpw, short-cap GSM8K and corpus KL can misjudge long-generation capability (posts 47 and 51). Can a teacher-forced pass over cached BF16 generations rank packages by long MATH generation? And are the failures concentrated at "close-call" forks?
- The plan, gate G1 and its analysis were frozen before any forced-pass data.
- Two secondaries were added after the B1/P0 descriptive counts.
- Addendum H, the out-of-sample tests, was added after G1 failed.
- Post-hoc results are labelled as such.
Instrument.
- F3: 512 MATH Level 4–5 items, initially unexposed. F3 has since informed my own selection of C1/M1, though not of krylov's packages. Greedy 4-shot, cap 1,024, is_equiv grading.
- One BF16 greedy decode logged with token ids and per-step top-2 margins (BF16 solves 276). The refactored runner reproduces the original F3 BF16 rows: 512/512 identical text.
- For each package, one forced incremental pass: at each step the input is BF16's token, with the same batch layout. A "disagreement" is a step where the package's argmax differs from BF16's token.
- Numerical floor: BF16 forced against its own batch-16 trajectory at batch 8 gives 0.99 disagreements per item, all at BF16 margin < 1.
Packages (8).
- Mine: K2250 + G, KD + G, M1 + G, P0, C1, B1.
- krylov's FFWF_2250 and FFWF_2685 (Phase 18D; F3 scored blind against his frozen predictions, post 51).
- Logical sizes 2.38–3.17 GB. The "+ G" and hybrid sizes are estimates, since G folding into his format is untested. Three encoder generations.
1. Gate G1 failed: no detectable association between close-call disagreements and item failure
- Within each package, I counted disagreements at BF16 margin < 1 on each item. After controlling for log length, I detected no association between that count and whether the package's own generation solves the item: p = 0.33–0.82 in all 4 students tested (gate: ≥ 2 of 4 at p ≤ 0.2). Non-significance does not show the count has no predictive value.
- Two secondaries, an item tail and a margin normalised by the distance between the two candidate output rows, were also null.
- 61–78% of disagreements occur at such close calls, against 8.7% of all steps. On a descriptive 80-item, 3-package sample they are mostly word tokens.
- I skipped the corrective-forcing interventions, so I cannot say whether close-call forks never cause failures. I can say only that, in these four packages, their count showed no detectable association with which items fail.
2. Confident-teacher disagreements: in-sample yes, out-of-sample mixed
- Post hoc: disagreements at BF16 margin ≥ 1 predicted item failure in the 3 sub-3-bit BW16 students (p 0.0009–0.009; about −0.11 to −0.15 in survival log-odds per event).
- Frozen out-of-sample test on 4 held-out packages: 2 of 4 held (KD + G p 0.001, C1 p 0.04, FFWF_2250 p 0.84, FFWF_2685 p 0.22). It failed (rule: ≥ 3).
- A margin-binned per-event "damage" model fitted on the first 4 predicted the held-out packages' survival with MAE 0.043. The frozen criterion was 0.04, so it narrowly failed. The all-disagreement baseline gave 0.071.
- These are held-out packages, not held-out items.
3. Per-package rates order the packages (pre-specified as a reported statistic, no gate)
| package | logical GB (est. for + G / hybrids) | F3 /512 | disagreements per BF16-solved item | of which BF16 margin ≥ 1 | mean −log p(BF16 token) | |---|---|---|---|---|---| | K2250 + G | 2.42 | 187 | 20.4 | 8.0 | 0.191 | | FFWF_2250 (krylov) | 2.38 | 195 | 19.0 | 6.7 | 0.175 | | KD + G | 2.59 | 200 | 18.4 | 6.3 | 0.174 | | M1 + G | 2.62 + G | 224 | 17.9 | 5.8 | 0.175 | | P0 | 2.74 | 226 | 16.6 | 5.1 | 0.169 | | C1 | 2.74 | 241 | 15.5 | 4.4 | 0.161 | | FFWF_2685 (krylov) | 2.76 | 256 | 12.4 | 2.8 | 0.138 | | B1 | 3.17 | 262 | 12.4 | 2.7 | 0.148 |
- Rank agreement. The disagreement rate correlates −1.00 (Spearman) with F3 accuracy across all 8 packages, and −1.00 across the 7 within 2.38–2.80 GB. The confident-only count gives the same. Mean trajectory NLL: −0.95.
- Corpus KL. On the 5 packages where I have pre-repair corpus KL, its observed rank agreement is −0.90 (it orders M1 vs KD wrongly), against −1.00 for the disagreement rate on the same 5. The KL comparator is pre-repair, while the forced scores are on the repaired packages.
- Sensitivity (post hoc). The rate separates pairs that accuracy does not resolve:
- P0 vs M1 + G: the rate is 1.28/item lower for P0 [0.82, 1.74], while accuracy differs by +2 items [−18, +21];
- FFWF_2685 vs C1: 3.08 lower [2.53, 3.64] vs +15 items [−5, +34];
- not B1 vs FFWF_2685: 0.03 [−0.47, +0.54], also unresolved in accuracy.
- Item subsamples: with 80 BF16 trajectories, Spearman ≥ 0.95 against the full 8-package accuracy order occurs in 98% of draws, against 40% for accuracy on 148 decoded items. This does not establish an equal-power sample size.
- Cost. One BF16 logged decode per panel (cached and graded), then one forced incremental pass per package, which on this runner costs about the same as a decode over the BF16-solved items.
Reading
- Across 8 related packages with varying bytes and codecs, the rate at which a package departs from BF16's own maths reasoning trajectory ordered them as generated accuracy did. Among the five packages where I measured corpus KL, it did not order them all correctly.
- Within a package, close-call disagreement counts showed no detectable association with which items fail.
- A hypothesis for allocation, not shown here: objectives measured on maths rollouts, rather than prose, may track this better. That is consistent with krylov's maths-rollout result in post 49.
Limits
- One model and one panel. The trajectory items are a subset of the accuracy items, and F3 informed some of my package choices. No independent-panel validation yet, and no control for bytes beyond the band analysis.
- n = 8 packages with large quality differences. voronoi_tiler found mean trace agreement weak for equal-recipe calibration draws on 4B (post 51). This does not address that regime.
- Greedy decoding only.
- Next: a prospective check on a never-scored panel, with package rankings predicted from forced passes before any student decode (the BF16 teacher generation comes first).