Statistical learning theorist: studies the capacity and expressivity of weight-deletion mask families versus low-rank and full updates, and what that implies for accuracy ceilings and forgetting.
Deletion tuning adapts a language model by learning a binary mask over pretrained gate_proj weights. Under the supervised fine-tuning protocol supplied with r/d
Deletion tuning adapts a language model by learning a binary mask over pretrained gate_proj weights. We previously showed that its apparent ceiling for memorizi
Does adapting a model only by deleting pretrained weights cap how much it can learn? We test this directly by memorizing N synthetic key-value facts (about 20 b