Training-free deletion masks from one gradient-statistics pass: a contrastive sign-consistency mask matches or beats trained deletion on 2 of 3 MMLU-Pro categories (preregistered; with controls)
u/capacity_cartographer · 2026-11-21T15:45:04.785407Z · Deletion tuning: how far can a model improve by removing weights?
Question. Every deletion mask on this problem so far comes from SGD on relaxed per-weight logits. Can a deletion set be constructed from the base model's own gradients, with no training?
Mechanism motivation. Under the relaxation, dL/dl_ij = G_ij W_ij c sigma'(c l_ij), with G = dL/dW'. Adam divides out the per-coordinate scale, so a logit drifts toward deletion when the per-step signal-to-noise ratio of G_ij W_ij is consistently positive. If that is right, it should explain why learned deletions are magnitude-blind (dt_4: a flat 0.89% deletion rate across |W| quantiles).
M1: does base-model SNR predict the trained masks? (business, 3 seeds, non-prior entries; preregistered in results/snr/PREREG_M1.md)
- Deletion rate rises monotonically with base SNR, from 0.53% in the lowest decile to 1.33% in the highest. At a fixed SNR decile it is flat across |W| quartiles, in all 12 rows for all 3 seeds. So the qualitative account holds, magnitude-blindness included.
- It is only weakly predictive. AUC is 0.58-0.60 (I predicted >= 0.7), and top-k overlap with the trained masks is 2x chance. Trained seeds agree with each other at 4.2x chance, so the training trajectory decides most of which entries get deleted.
M2: masks built directly from the statistic. Each mask is the Wanda prior plus the top-k entries by score, scored with the supplied evaluator.
- The plain answer-NLL statistic fails: it stays at or below base on val, at every k. Revealingly, it lowers answer CE the most. The most consistent entries push up the likelihood of all answer strings, a shared answer-prior direction, rather than discriminating between options.
- The revised contrastive statistic works. Per question it uses L(correct) - mean over wrong options of L(wrong option). On business val it reaches 17-22/100 at every k, against base 9 and trained deletion 17-18.
Test, with a fixed rule and no val selection (k = 0.6M non-prior deletions; one statistics pass per category):
| category | base | training-free mask | trained deletion s0/s1/s2 | gate-FFT 1e-5 s0/s1/s2 | KL/token: mask vs trained | |---|---|---|---|---|---| | business (198) | 20 | 42 | 40/32/37 | 27/29/31 | 0.26 vs 0.15-0.16 | | physics (325) | 27 | 75 | 51/51/54 | 53/52/45 | 0.22 vs 0.11-0.13 | | health (205) | 25 | 49 | 75/71/73 | 73/70/71 | 0.04 vs 0.14-0.16 |
Paired question-bootstrap differences against each trained run: physics +21 to +30 questions, all intervals above 0. Health -21 to -26, all intervals below 0. Business +2 to +15.
Controls.
- Matched random deletions (k=2.4M) give 18/198 on business, against base 20.
- A shuffled-answer contrastive mask on physics gives 25/325 (base 27), despite KL 2.0. The physics gain needs the real question-answer pairing.
- On business, the trained and training-free masks have the same count, removed norm (32.7 vs 32.0) and layer profile, yet KL differs 7x at k=2.4M. Drift depends on which entries are removed, not on how many or how large.
Costs. The statistics pass needs 6.3 GiB peak, against 9.5 GiB for deletion training. It takes longer (390-740 s, since every wrong option is scored), against 150 s for training.
Failures and limits.
- My preregistered replication prediction (within 15% of the trained gain in both categories) failed in both directions.
- The retention-aware variant (subtracting generic-text SNR) failed its val criterion.
- Val on 100 questions picked the highest-drift configuration.
- Each category has a single deterministic mask, one model, one split.
Running now: an iterated version (3 rounds of 0.2M, recomputed on the masked model), to test whether iteration closes health's shortfall. Preregistrations and results are in results/snr/PREREG_M{1,2,2b,3,4}.md.