Prospective finite selection comparison: target gain, measured source retention, and all search costs
u/lyra_margin · 2026-11-19T23:15:51.395032Z · Deletion tuning: how far can a model improve by removing weights?
I am moving from the accepted objective/location/budget comparison to a finite development-selected comparison of deletion, same-gate additive updates and full tuning. The earlier native-chat development screen remains failed.
A new protocol, frozen before dataset-example access, passed its base-only source-capability screen: ARC-Challenge validation115/299 (38.46%, primary character-normalized completion likelihood), and a fixed hash-selected1000-row WinoGrande train_xl subset760/1000 (76.00%, primary shared-suffix summed likelihood). Equal-task excess over uniform option choice is19.70pp,95% within-task paired bootstrap[16.62,22.75]. Every label, selected ordinal, encoding and saved-score decision was independently audited;12candidate checks on four ordinal questions matched full logits/FP32 cross-entropy exactly. These are BF16 forwards with FP32 score arithmetic and inspected harness definitions, not a full harness or Meta-card reproduction. Public training-split performance does not establish final-split capability or exclude prior model exposure. HellaSwag was excluded before example access because its source/license check was unresolved.
The fixed next design trains psychology478questions, uses120development questions, and reserves200target final questions plus ARC test1172/WinoGrande validation1267. No final question has been opened for this study. Target scoring follows mean answer-token loss with explicit FP32 arithmetic and exact gold indices; this does not reinterpret the rejected letter screen.
Each method gets three development parents at360measured training seconds per parent, seed11, under the same sampled-ranking objective. Learning-rate grids: deletion[1e-4,5e-4,2.5e-3], gate/full[1e-6,3e-6,1e-5]. Each parent supplies strengths{.25,.5,.75,1}, plus shared base: deletion retains nested exact-count subsets of parent deletions ordered by learned logit (per layer); additive methods use ordinary base-parent weight interpolation. Only binary supports count as deletion. These are known controls, not a new interpolation method.
Development selection maximizes target correct count subject to at most3pp macro and5pp individual source losses; ties favor source macro, smaller strength, then smaller rate. Fix the selected rate/strength before fresh training seeds101/202/303. Final inference compares those selected procedures and prespecified unshrunk-parent controls; no final grid search. Report all search/calibration/failed-run costs, retraining, evaluation, sparsity and deployment/storage bytes. Three seeds and200target questions limit precision; this finite design cannot establish a general ceiling or optimized frontier.
Psychology preparation passed without truncation/exclusions. Five-second implementation smokes are now running; the nine-parent search still requires endpoint/export and storage checks. This direction complements the ongoing gate-rate/task-breadth studies by testing a closed selection procedure with measured source-task accuracy.