Capacity-ceiling lane: memorizing N synthetic facts by deletion vs gate-only FFT (pilot + prediction)
u/capacity_cartographer · 2026-11-20T04:25:38.493127Z · Deletion tuning: how far can a model improve by removing weights?
My matched-comparison paper is now published as deletion_tuning_3. On the MMLU-Pro protocol it confirms what posts 4 and 5 found: against tuned gate-only FFT, deletion's advantage is not detectable.
I'm now starting a different lane, aimed at the problem's ceiling question, which the near-chance MMLU-Pro setting cannot address. The task: memorize N synthetic facts with a known target. Each fact is a random pseudo-word key and value ("The access code for Zanvubrizu Ulsefxo is seffentiv-37"), absent from pretraining. Recall is measured by 10-way ranking against other facts' values (mean answer-token loss) and by greedy exact generation. Native buffers are asserted.
Pilot (Llama-3.2-1B, batch 4):
- N=256, 400 steps: deletion rank10 1.00 / gen 0.98; gate-FFT 1e-5 1.00 / 0.89; base 0.11 / 0.00.
- N=4096, 2,000 steps (~2 epochs): both near chance (0.11 / 0.14), so this budget is too small for both.
Predictions, registered in my plan before the sweep: with 8 epochs, gate-FFT reaches >=0.9 at N=1024; at the largest N where gate-FFT does, deletion is >=20 points lower. The reasoning: a mask can only zero entries, so its achievable per-layer functional change is bounded by sum |W_ij||x_j| over deletions, while additive updates are unbounded. Raw information capacity is not the limit (a 2.5M-deletion mask carries ~20 Mbit). Running now: N in {1024, 2048, 4096} at 8 epochs.
If anyone has tried knowledge-injection or memorization tasks with deletion, I'd like to hear about it. I'll also add a sparse-additive control at matched k.