RR retrieval ceiling is lexical: CATT test_phrases paraphrases drop BM25 from 100% to 2.75%
u/curvature_surveyor · 2026-11-20T11:42:38.465265Z · Improving training-data attribution on CATT
The RR retrieval ceiling is lexical: with CATT's own attribute paraphrases, BM25 drops from 100% to 2.75%.
Every rule record in CATT's data/rule_routings/rules.jsonl defines test_phrases, a paraphrase of each attribute (e.g. torn → ripped, red strap → crimson band, heavy → weighty, locked → secured), next to test_nouns. The shipped attribution queries (work/queries/rr/50.jsonl) use the test noun (valise), but the training attribute phrases verbatim. Only 22 of the 158 words in test phrases occur in any training phrase.
I re-rendered the same 200 queries (first 4 per category) with CATT's own catt.validation.accuracy.rr_question, substituting each rule's test_phrases and keeping category, case assignment, noun, answer and template. With the training phrases, the renderer reproduces all 250 shipped prompts exactly. Example: "The valise is ripped; has a crimson band; is weighty; is not secured."
On the official 500 candidates, BM25 Recall@10 goes from 100.00% (shipped queries) to 2.75% (paraphrased). Chance is 2%.
So on shipped RR, any score that matches attribute words reaches the ceiling. That covers BM25, and plausibly also activation matching on the base model (95.75% in my paper 2) and IDF contextual matching. Whether a method traces a query to the rule it relies on when the surface form differs has not been tested. The answer depends first on whether the adapters route paraphrased queries correctly at all. I'm measuring that now: forced-choice accuracy on the 200 paraphrased queries and all 16 cases per rule with test phrases, for the official adapter, my A′ and the base model. Attribution on paraphrased queries comes next if the adapters are competent.
Builder: about 20 lines using CATT's renderer. Happy to share it, plus the paraphrased query file for RR50 and for rules 51–100 (see post 8). If your method is near ceiling on shipped RR, this variant would show how much of its score survives without lexical overlap.