Deletion tuning: how far can a model improve by removing weights?

AI · Language model adaptation and interpretability

Deletion tuning: how far can a model improve by removing weights?

Motivation and research goal

Adapting a pretrained language model to a new task usually means updating its weights, either through full fine-tuning or through low-rank adapters such as LoRA. Updating weights can also erode capabilities the model already had. Deletion tuning (code, project page) keeps every pretrained value fixed and learns only which weights to set to zero. The resulting task-specific mask is cheap to store. The paper reports that deletion tuning can match or exceed conventional fine-tuning while forgetting less.

This problem asks how far that idea can go. A model adapted only by deletion is limited to the pretrained values it already has, so it may run into a performance ceiling that full fine-tuning does not. On the other hand, that same restriction may be what protects existing capability. We want to know:

Mechanistic analysis of how deletion changes the model's internal computation is one route to these answers. It is optional. Contributions can also come from careful measurement of strengths and weaknesses, or from a better deletion method backed by matched comparisons.

Setting and definitions

Let W be a fixed pretrained weight matrix and M a binary mask of the same shape. The adapted weight is W′ = W ⊙ M, so each entry either keeps its pretrained value or becomes zero. The original method learns masks for the gate_proj matrices of the feed-forward (MLP) blocks. During training it optimizes a continuous relaxation of the mask. At inference it uses a strictly binary mask. Claims about the deployed model should therefore concern the binary mask, not the training surrogate.

Supervised fine-tuning (SFT), self-distillation fine-tuning (SDFT) and GRPO reinforcement learning (Shao et al., 2024) are training objectives, not separate kinds of model. The useful comparison is deletion against full-parameter or LoRA training under the same objective, data and checkpoint selection.

Consider a business question from MMLU-Pro as an example. Both deletion SFT and full fine-tuning train on the text of the correct answer. At evaluation, each candidate answer is scored by its mean answer-token loss, and the model picks the answer with the lowest loss. The two methods differ only in which changes to the network they are allowed to make.

Verified starting point

We reproduced three SFT categories on a single RTX 5090 (32 GB) with Llama-3.2-1B-Instruct. All runs used the official category splits, seed 0, and 400 optimizer steps at an effective batch size of 4 (micro-batch 1, accumulation 4). Within each category, all four models were evaluated on identical inputs, which we confirmed with data hashes.

| Model | Business | Biology | Chemistry | |---|---:|---:|---:| | Base model | 10.10% | 15.56% | 9.54% | | Published author mask (Hugging Face) | 14.65% | 23.33% | 13.78% | | Our trained deletion mask | 17.17% | 23.33% | 15.19% | | Full fine-tuning | 15.66% | 27.22% | 12.37% |

Each training run took about 179 to 230 seconds. Peak allocated memory was about 10.3 GiB for deletion and 23 GiB for full fine-tuning, and each evaluation took 27 to 40 seconds. These single-seed results make a cheap starting point. They also show that deletion does not consistently beat full fine-tuning: it leads on business and chemistry and trails on biology.

We also ran full training in two further regimes on Llama-3.2-1B-Instruct:

| Regime / task | Base accuracy | Final deletion mask | Training scale | Measured runtime | |---|---:|---:|---|---| | SDFT / Science | 14/507 (2.76%) | 60/507 (11.83%) | 166 updates, 2 epochs | 31 min 54 s training | | SDFT / Tooluse | 5/97 (5.15%) | 15/97 (15.46%) | 252 updates, 2 epochs | 52 min 34 s training | | GRPO / GSM8K | Not yet verified | 56.79% | 348 updates, 3 epochs | 103.70 min total job |

The SDFT base and final-mask runs used matching evaluation inputs and gold answers. The paper's SDFT setting uses Qwen2.5-7B-Instruct. On one 32 GB card that model failed during mask initialization, so the Llama-1B setting is our own adaptation. These runs use a single seed, reach low absolute accuracy, and have no matched full-parameter SDFT control yet. The GRPO figure comes from the trainer's own GSM8K evaluation. We claim no gain over the base model until that baseline is measured independently.

Research directions

The directions below are options. They can be combined, and other well-motivated approaches are welcome. SFT is the primary setting. SDFT on the adapted Llama-1B Science and Tooluse tasks is the second priority, used to test whether findings carry over to self-distillation. GRPO on GSM8K is an optional extension and is not required to start the SFT work.

1. Map the ceiling, strengths and weaknesses. Begin with the three verified MMLU-Pro categories. Measure variation across seeds and how much capability is retained on tasks the training did not target. Then extend to further categories and BBEH once the protocol is stable. Possible questions include:

Include a full fine-tuning control restricted to the same gate_proj matrices. That control separates the effect of deleting weights from the effect of where training happens. LoRA is a strong additional baseline if its parameter and compute budgets are reported.

2. Improve the method. Possible changes include allocating masks differently across layers or matrix types, better mask initialization, alternative relaxations or training schedules, and composing masks across tasks. Judge an improvement on three things measured together: target accuracy, retention of untargeted capability, and training and storage cost. Unstructured sparse masks do not by themselves speed up dense inference, so any efficiency claim must be measured.

3. Analyse how deletion changes computation. The paper already analyses weight geometry, including stable rank, spectral drift and overlap with principal weight directions, and links these to forgetting. Reproducing those analyses is a reasonable check but not a new contribution. Functional analysis can begin with descriptive measurements:

Descriptive measures cannot establish cause. A high similarity score or a successful probe can coexist with very different causal pathways, so causal claims need interventions. Useful interventions include:

Zeroing a weight can also reroute activity through the network's nonlinearities, so deletion need not act only by switching pathways off. Two hypotheses deserve explicit tests. Deletion may suppress pathways that interfere with an existing capability, or it may redirect computation toward capabilities that are present but unused. Either, both in different layers, or neither may hold. Findings in this direction are most valuable when they explain a strength or weakness identified in direction 1, or motivate an improvement in direction 2.

What counts as progress

Progress could be:

A clear negative result is also useful when suitable controls rule out a hypothesis.

Match the objective, data, checkpoint selection rule and compute across methods. Choose hyperparameters and checkpoints without consulting the final test split, and disclose any periodic test evaluation. Use several seeds and held-out tasks, and report exact accuracy counts with confidence intervals. Also report mask sparsity, wall time, memory measurement scope and retained capability. Before attributing an SDFT gain to deletion specifically, add a matched full-parameter SDFT control. Release scripts, configurations and data splits so that every number can be reproduced. Setup and command details are in the supplement.

Task materials

Download supplement (.zip)
Type
open-ended
Solve status
open
Access
Invite-only
Loading…