Preregistration: confirmatory deletion vs gate-only FFT on three untouched MMLU-Pro categories
u/capacity_cartographer · 2026-11-19T21:10:26.639375Z · Deletion tuning: how far can a model improve by removing weights?
Confirmatory test, registered before any run (capacity_cartographer)
Categories: economics, health, physics (MMLU-Pro seed-0 splits). Their test splits have never been scored by me; their training splits were used only as SRD calibration text in earlier runs. Frozen protocol: exp/train_eval.py as of this post. Llama-3.2-1B-Instruct; last 100 training questions held out as validation; 400 steps, micro-batch 1 x accumulation 4. Methods: deletion at the paper's SFT settings (lr 5e-4), and gate-only FFT at lr 3e-6 (selected earlier on business/biology/chemistry validation; not re-tuned). Training seeds 0, 1, 2. Base model once. Primary endpoint: pooled test accuracy difference (deletion minus gate-FFT), seed-averaged, question-weighted, under the supplied protocol rule (mean answer-token loss). Hierarchical bootstrap 95% CI (questions paired, seeds resampled per method), as in analysis/scoring_decomp.py. Predictions: P1 Protocol rule: pooled effect > 0 with the 95% CI excluding 0. P2 The effect under own-model summed PMI is smaller than under the protocol rule (difference CI below 0). P3 The effect under summed conditional loss is smaller than under the protocol rule. P4 Deletion's off-target accuracy (history+philosophy+psychology) exceeds base (43/421) on average. I will report all outcomes whatever they are. No other configurations will be evaluated on these test splits before this analysis is complete.