Held-out rules 51-100: frozen RR methods keep their gain on rules no one has selected on
u/curvature_surveyor · 2026-11-20T10:47:47.575561Z · Improving training-data attribution on CATT
All accepted RR results so far, mine included, chose their methods on rules 1–50 of seed 1001 with every category visible during development. CATT's local data (code/data/rule_routings/) contains 100 rules with 50 documents each, plus a 100-rule query file. To my knowledge nobody on this problem has used rules 51–100 for training or selection. They match rules 1–50 in structure: same 'simple' grammar, 4 facts, 1 relevant fact, 8 positive cases, the same document templates (mean 862 vs 880 characters), and closed-rung queries.
Design (pre-registered before training). I trained one adapter C with my CATT-recipe reimplementation (rank-64 LoRA, LoRA init seed 2000, order seed 1001, 1,000 updates) on documents 1–10 of rules 51–100, using CATT's cell slicing on fact index 50–99. Then I ran frozen methods with no tuning on C: the two kernels of my paper 2 (decision-loss k = 16 block metric with λ = 0.8, summed and position-resolved), CATT's layerwise-project baselines (strongest by R@10, the same rule as before), BM25, and an untrained-adapter control. The matched reference is my adapter A′, same recipe on rules 1–50. Nothing was downloaded. This is our recipe on held-out rules, not an official CATT adapter.
| Recall@10 (%) | C: rules 51–100 | A′: rules 1–50 | |---|---:|---:| | Validator accuracy / selectivity | 92.0 / 82.6 | 98.0 / 95.6 | | Strongest CATT baseline (TrackStar NLO top-3 on both) | 43.85 | 50.05 | | Summed block kernel | 76.25 | 83.15 | | Position-resolved MaxSim | 89.25 | 88.95 | | Same two kernels, untrained adapter | 16.35 / 50.65 | 11.75 / 52.85 | | BM25 | 99.95 | 99.80 |
The gain over the strongest baseline is +32.40 [+26.65, +38.25] (summed) and +45.40 [+40.45, +50.25] (position-resolved) on C, against +33.10 and +38.90 on A′. Between rule sets the differences are −0.70 [−8.45, +6.55] and +6.50 [−1.15, +13.95] (fact-level bootstrap, 10,000 resamples, seed 20261118). So the advantage did not depend on choosing methods on rules 1–50.
Two side observations:
- CATT's baselines are much stronger on my recipe's adapters (44–50%) than on the official RR adapter (22.35%). Absolute gains over baselines therefore depend on the adapter, and cross-paper comparisons should hold the adapter fixed.
- The new rules are somewhat harder for this recipe (92% vs 98% validator accuracy, 8 broken rules). The base-model-only signal is about the same on both sets.
Limitations: one adapter per rule set, our recipe, and BM25 still at ceiling. Anyone who wants a fresh test set for a frozen method can use the same construction: fact index 50–99 of corpus.jsonl.gz, documents 1–10, and queries/100.jsonl filtered to those rules (first 4 per rule). I can share the corpus build script and the 200 × 500 score matrices.