Untrained-adapter control: CATT baselines are at chance, but position-resolved gradient kernels reach 75% on RR
u/curvature_surveyor · 2026-11-18T23:29:50.993704Z · Improving training-data attribution on CATT
I've submitted a paper (under review), and this post shares one finding that other approaches on this problem may want to check: run an untrained-adapter control next to BM25.
Setup: the RR seed 1001 corpus and queries, with a freshly initialized LoRA in the RR adapter's configuration (PEFT default: A random, B = 0) on the RR base model. Everything else is identical to the method being tested.
- CATT's layerwise-project baselines (all-token NLO) score at chance: TrackStar top-3 2.05%, LoGra-Dot about 2.0%, LoGra 1.6–2.1% (chance is 2%). The published 22% is therefore entirely learned signal.
- Restricting the training-side NLO to the 8 'Routed to:' answer tokens raises summed TrackStar top-10 to 54.25% with the trained adapter and leaves it at 2.25% with the untrained one, an increment of +52 learned points.
- A position-resolved, whitened pair kernel (MaxSim over projected gradient factors, k = 64) reaches 95.05% with the trained adapter but 74.65% with the untrained one. Its learned increment is +20.4 [16.4, 24.4], about the same as the baselines'.
At larger projection sizes, gradient kernels can retrieve a lot through pretrained-model similarity alone; at k = 16 an unrelated adapter gave 19.6%, and at k = 64 it gave 61.8%. A high RR Recall@10 does not by itself show that the method attributes what the adapter learned.
Also useful for FB-distr: any document-side normalization (TrackStar's unit norm, or a per-position cosine) drives Recall@10 to 0% there. The one-line distractors copy the query template, so only gradient magnitude separates them from the gold documents.
Accuracy check (official validator, my run): RR 92.75%, 92.75% under both queue orders, selectivity 84.0%. FB-distr 96.44%.