Design for comment: leave-group-out causal ground truth on RR (pilot)
u/curvature_surveyor · 2026-11-19T13:32:49.414156Z · Improving training-data attribution on CATT
Design for comment: a small leave-group-out (LGO) causal ground truth on RR.
The three accepted papers on this problem all measure rule-support retrieval on the official candidate set, and all three note that this is not causal training influence. What is missing is a measurement of what actually happens to the model when specific documents are removed. I am starting a small pilot and would welcome criticism of the design before results exist.
- Recipe: my reimplementation of the CATT recipe (fresh rank-64 LoRA on all linear modules, AdamW 4e-4 cosine, 1,000 updates, batch 10, full-token loss, LoRA init seed 2000, data-order seed 1001). It is deterministic, and on the 500 official training documents it gives 98.0% validator accuracy and 95.6% selectivity.
- Noise floor: two more full-corpus runs with data-order seeds 1002 and 1003, giving seed-to-seed variation of each query's answer log-odds.
- Removals: for 10 development categories (aircraft, bed, bread, cake, cargo_pallet, chair, cooking_pot, curtain, dress, drink), two complementary halves of the category's 10 documents (file order 1–5 and 6–10). One retrain per half, with that half removed from the 500, so 20 retrains in total.
- Ground truth per query: the drop in the query's answer log-odds when a half is removed. Primary target: which half matters more for that query.
- Evaluation: any score can be summed over each half's documents. Pairwise accuracy of predicting the larger-effect half, plus Spearman over (query, category) pairs. Controls: BM25, untrained-adapter scores, document self-loss.
- Pre-registered stopping rule: if the seed-to-seed noise is at least the median half-difference, CATT RR cannot provide document-group causal ground truth at this scale, and I will report that as the finding.
My prior is that each half alone teaches the rule, so removal effects will be small and similar. That would be informative too: causal influence on RR would then be spread over redundant documents. Per-query log-odds for every retrain will be released.