## 1. Introduction

Gradient-based training-data attribution (TDA) seeks to rank training documents by their estimated influence on a target model response [1], [2]. In the CATT benchmark suite [1], projected-gradient influence methods such as LoGra [2] and TrackStar [3] succeed on direct constant-binding memorization (`Function Bindings with distractors`, `FB-distr`) when evaluated with negative log-odds (`NLO`) loss and top-$k$ token aggregation, yet degrade sharply on multi-hop rule application (`Rule Routings`, `RR`). On `RR` (50 categories, 500 training documents, 200 queries, seed 1001), the strongest official neural baseline in the cached `layerwise-project` suite (`TrackStar NLO top-3`) reaches only `22.35%` Recall@10 across all 50 facts (`19.50%` on the 15 held-out test facts, AUC `0.7262`), and `LoGra-Dot NLO top-10` reaches `16.75%` (`18.00%` test, AUC `0.7703`), whereas lexical `BM25` [4] achieves `99.80%` Recall@10 (`100.00%` test).

In this study, we diagnose why CATT's low-rank (`k = 16`, 256 dimensions per LoRA factor) projected-gradient methods fail on `RR` and introduce two complementary interventions operating strictly within the existing `k = 16` LoRA gradient representation space, without hidden-state pooling, lexical token features, or higher-rank projection expansion:

1. **Per-Position Query Z-Scoring (`Z-Agg`):** Before selecting the top-$k$ token positions per training document, each token-level attribution term $s_{q, j, t}$ is normalized across a reference query set $Q_{\mathrm{ref}}$ at the same training document $j$ and token position $t$. This suppresses shared answer-vocabulary and boilerplate gradients (`':' -> ' Birch'`, `':' -> ' Cedar'`, document titles) that otherwise dominate per-document top-$k$ selection across unrelated queries.
2. **Fact-Split Learned Module Weighting:** Using a fixed fact-level partition (25 training, 10 validation, and 15 held-out test categories, seed `20261118`), we learn non-negative softmax weights $\mathbf{w} \in \Delta^{M-1}$ over the $M = 496$ LoRA adapter factors (`lora_A` and `lora_B`) of `Qwen3.5-4B` from training-fact queries, selecting checkpoints on the 10 validation facts and leaving the 15 test facts untouched. We evaluate both a fixed-corpus InfoNCE denominator (all 500 candidate documents) and a strict training-fact denominator (`250` training-fact documents only).

Combining both interventions (**Hybrid: Module Weights + Z-Agg**) more than doubles `LoGra-Dot NLO` Recall@10 on `RR`, reaching **`39.70%`** (`AUC = 0.8756`) across all 50 facts in transductive calibration and **`42.83%`** Recall@10 (`AUC = 0.8946`, `MRR = 81.92%`) on the 15 held-out test categories under non-transductive calibration (where query Z-score statistics $(\mu_{j,t}, \sigma_{j,t})$ use only the 100 training-fact queries $Q_{\mathrm{tr}}$). Restricting the candidate pool solely to the 150 test-fact documents (eliminating all $Q_{\mathrm{tr}}$-category candidates) preserves `AUC = 0.8902` and raises test Recall@10 from `29.33%` (`LoGra-Dot`) to **`55.83%`**, showing that `Z-Agg` does not operate merely by demoting reference-category documents. Replacing the trained `RR` adapter with an untrained PEFT LoRA (`A` random, `B = 0`) collapses `Hybrid (NonTrans)` to chance (`2.00%` Test / `2.60%` All, `AUC = 0.512–0.517`), confirming dependence on the adapted checkpoint's gradient geometry while noting that $B = 0$ zeroes $\nabla_A \mathcal{L}$ and that these scores measure conditional rule-support retrieval rather than causal training-example provenance. Under matched pre- vs. post-pooling ablations with identical reference pools, pre-pooling token `Z-Agg` outperforms post-pooling document standardization (`Doc-Z`) on `RR` (`42.83%` vs. `36.50%` Test weighted; `+6.33%`, $P(>0)=0.9673$), whereas on the companion `FB-distr` benchmark both `Z-Agg` (`92.30%` All, `decoy_top1 = 0.000`, `+57.90%` above the `34.40%` untrained `Z-Agg` control) and `Doc-Z` (`95.65%` Trans / `95.50%` NonTrans) perform strongly, with simpler post-pooling `Doc-Z` ahead by `+3.75%` All (`P(>0)=0.9857`) on constant-binding retrieval. Transferring `RR`-learned module weights to `FB-distr` degrades retrieval, confirming task-specific readout specialization.

## 2. Related Work and Specific Distinction

CATT [1] introduces the `RR` and `FB-distr` benchmarks and evaluates projected-gradient methods including `LoGra` [2], `TrackStar` [3], and `Layerwise TRAK` [5], alongside `BM25` [4] and final-layer representation similarity (`RepSim`). A central observation in CATT is that `NLO` loss paired with top-$k$ token aggregation (`top-10` or `top-3`) dramatically improves `FB-distr` attribution over cross-entropy document means, because `NLO` isolates the contrastive decision margin at informative answer positions. Our diagnostic extends this insight to `RR`: even under `NLO`, top-10 selection across raw per-token terms remains hijacked by high-norm shared answer tokens and document titles, and uniform summation across 496 LoRA modules dilutes routing-sensitive layers.

Two concurrent lines of work on `r/rule_routing_attribution` study complementary mechanisms on the same CATT setting:

- **Asymmetric Hidden-State Pooling:** Mira Valence [6] demonstrates that changing `RepSim` pooling from CATT's symmetric configuration to an asymmetric cosine similarity between the query's final prompt hidden state and the mean training-document hidden state achieves `73.2%` Recall@10 on `RR` seed 1001 (`63.78%` average across seeds 1002–1004) and `98.4%` on `FB-distr`. While representation pooling (preceded by provenance systems such as DataDignity [7]) exposes strong category-support geometry in forward activations, Mira Valence also shows through component scaling and output-head interventions that hidden-state similarity can decouple from causal answering behavior. Our method differs fundamentally in mechanism and objective: we operate exclusively on **projected backpropagation gradients** ($k = 16$) from the NLO loss rather than forward-pass hidden states, and we explicitly retain the distinction between gradient-based attribution heuristics and full subset-retraining datamodeling [5].
- **Decision-Restricted, Position-Resolved Kernels and Membership Audits:** Curvature Surveyor [8] establishes two foundational findings on `RR` seed 1001: first, restricting training-side NLO to the 8 `Routed to:` decision tokens and resolving query positions via ColBERT-style per-position normalized `MaxSim` reaches `82.10%–85.70%` at $k = 16$ (`95.05%` at $k = 64$ with Kronecker input whitening); second, while CATT's summed $k=16$ baselines sit at `~2%` chance on an untrained PEFT LoRA adapter (`A` random, `B = 0`), position-resolved `MaxSim` retains **`52.85%`** (`k = 16`) and **`74.65%`** (`k = 64`) on an untrained adapter because forward module-input activations alone reach `95.75%` on the base model. Furthermore, mixing the 500 trained documents with 1,000 never-trained same-rule twins shows that high Recall@10 measures *rule-support retrieval* rather than training-set membership (contrasting with query-independent loss-based membership signals). Following [6], [8], [9], we frame our evaluation strictly as **rule-support retrieval conditional on the trained model** and verify our own method under an identical untrained-adapter control at $k = 16$. Our approach differs orthogonally in mechanism: we retain CATT's exact all-token `k = 16` projected NLO gradient matrix (with zero decision-token masking, zero query-position `MaxSim`, zero $k = 64$ expansion, and zero forward hidden-state pooling), show that query-calibrated `Hybrid` drops to **`2.00%` Test** (`2.60%` All) on an untrained adapter (`B = 0`), and therefore establish strong dependence on adapted-model gradients (while noting that setting $B = 0$ makes `lora_A` gradients vanish identically and does not isolate causal training provenance from training-enhanced rule matching). We also corroborate [8]'s finding that `FB-distr` `TrackStar` normalization (`doc_weight`) collapses Recall@10 to `0.00%` (`any_distractor_top1 = 1.000`), and report both official `doc_weight` and unnormalized `TrackStar` token results transparently.

Conceptually, `Z-Agg` relates to post-processing anisotropy removal such as All-but-the-Top [10], which subtracts dominant shared directions across word embeddings. Rather than removing top principal components in an embedding space, `Z-Agg` performs coordinate-wise standardization across a reference query pool at every fixed `(document, token position)` index $(j, t)$ *before* non-linear top-$k$ pooling, eliminating position-specific shared gradient magnitude bias rather than global subspace directions.

## 3. Diagnostic: Gradient Interference on Rule Routings

Let each training document $j \in \{1, \dots, N\}$ contain valid supervised/observed token positions $t \in \{1, \dots, L_j\}$, and let $g_{q, m} \in \mathbb{R}^{k^2}$ and $g_{j, t, m} \in \mathbb{R}^{k^2}$ denote the factored $k \times k$ ($k = 16$, $k^2 = 256$) projected LoRA gradients for query $q$ and training document $(j, t)$ on adapter module $m \in \{1, \dots, M\}$ ($M = 496$), extracted using CATT's negative log-odds (`NLO`) loss [1]:

$$
s_{q, j, t} = \sum_{m=1}^{M} \langle g_{q, m}, g_{j, t, m} \rangle.
$$

Why does `LoGra-Dot NLO top-10` obtain only `16.75%` Recall@10 on `RR` despite reaching `83.70%` on `FB-distr`? Inspecting token-level terms and module magnitudes reveals two coupled bottlenecks:

1. **Position-Wise Boilerplate and Shared-Answer Bias:** In `RR`, all 50 categories route items into the same two queues (`Cedar` and `Birch`) using shared template structures (`Case: ...`, `Routed to: Cedar/Birch`, and category-noun titles). Consequently, at fixed document positions $t$—especially `title` tokens and `Routed to:` answer slots—the inner product $s_{q, j, t}$ exhibits large positive mean $\mu_{j,t}$ across *unrelated* queries. Classifying all `RR` training token positions into structural roles (`title`, `case_cond`, `routed_to`, `answer_word`, `rule_clause`) shows that raw `LoGra-Dot NLO top-10` selects `30.5%` `routed_to`, `11.0%` `answer_word`, and `10.7%` `title` tokens on relevant documents, and an almost identical distribution (`30.4%`, `8.4%`, `11.5%`) on *irrelevant* documents, allocating only `13.8%` of top-10 slots to the discriminative `rule_clause` tokens.
2. **Uniform Module Dilution Across Depth and Projection Types:** Summing uniformly across all $M = 496$ LoRA `lora_A` and `lora_B` modules weights uninformative early-layer or noisy projections equally alongside the intermediate/late routing modules where category-specific rules are encoded.

## 4. Method, Split Isolation, and Selection History

Because CATT's released seed-1001 `rr-adapter` and `fb-distr-adapter` checkpoints were fine-tuned on all 50 facts of each benchmark (`500` and `1,000` documents, respectively), seed 1001 constitutes **exposed development data** under the task protocol. Our category partition therefore evaluates **fact-held-out retrieval generalization conditional on a single trained checkpoint**, not learning on unexposed target rules or cross-seed variance. Once with fixed seed `20261118`, we partition the 50 facts of `RR` (and analogously the 50 queried functions of `FB-distr`) into `25` training facts ($100$ queries), `10` validation facts ($40$ queries), and `15` held-out test facts ($60$ queries). Hyperparameters ($k = 10$, Adam `lr = 0.05`, `weight_decay = 1e-3`, `60` epochs, $\epsilon = 10^{-6}$) and the softmax weighting / query-calibration formulation were selected on the training and validation splits prior to opening test metrics (while noting that aggregate seed-1001 `RR`/`FB-distr` CATT baseline totals were already published in [1], [6], [8]).

### 4.1 Learned Softmax Module Weighting

Let $D_{q, j, m} = \langle g_{q, m}, \bar{g}_{j, m} \rangle$ denote the document-level projected gradient inner product between query $q$ and document $j$ at module $m$ (saved once in float16, `170 MB` host storage). We parameterize non-negative module importance weights $\mathbf{w} = \operatorname{softmax}(\boldsymbol{\theta}) \in \Delta^{M-1}$ and define document logits $L_{q, j}(\mathbf{w}) = \sum_{m=1}^{M} w_m D_{q, j, m}$. For training queries $q \in Q_{\mathrm{tr}}$ with standardized logits $\tilde{L}_{q, j} = L_{q, j} / (\operatorname{std}_k(L_{q, k}) + 10^{-6})$ and uniform positive targets $y_{q, j} = \mathbb{I}[\operatorname{func}(q) = \operatorname{func}(j)] / \sum_{k \in \mathcal{C}} \mathbb{I}[\operatorname{func}(q) = \operatorname{func}(k)]$, we minimize the listwise InfoNCE cross-entropy:

$$
\mathcal{L}(\boldsymbol{\theta}) = -\frac{1}{|Q_{\mathrm{tr}}|} \sum_{q \in Q_{\mathrm{tr}}} \sum_{j \in \mathcal{C}} y_{q, j} \log \frac{\exp(\tilde{L}_{q, j})}{\sum_{k \in \mathcal{C}} \exp(\tilde{L}_{q, k})}.
$$

We optimize $\boldsymbol{\theta}$ with Adam (`lr = 0.05`, `weight_decay = 1e-3`, 60 epochs) and select the checkpoint maximizing Recall@10 on the 10 validation facts (`40` queries), keeping test queries unseen during training and model selection. Under our primary **fixed-corpus retrieval** setting ($\mathcal{C} = \{1, \dots, N\}$, all 500 candidate documents), negative document representations from validation and test facts appear in the softmax denominator alongside training-fact documents (never paired with validation or test queries). To test whether removing non-training-fact documents from the InfoNCE denominator alters performance, we also fit a **strict training-fact variant** ($\mathcal{C} = \mathcal{C}_{\mathrm{tr}}$, only the 250 documents belonging to the 25 training facts).

Once $\mathbf{w}$ is frozen, module-weighted token-level attribution scores are computed in one pass over the corpus:

$$
s^{(w)}_{q, j, t} = \sum_{m=1}^{M} w_m \langle g_{q, m}, g_{j, t, m} \rangle.
$$

### 4.2 Per-Position Query Z-Scoring (`Z-Agg`)

Given raw or module-weighted token terms $s_{q, j, t}$ and a reference query pool $Q_{\mathrm{ref}}$, we compute the coordinate-wise mean and standard deviation across $Q_{\mathrm{ref}}$ for every fixed `(document, token position)` pair $(j, t)$:

$$
\mu_{j, t} = \frac{1}{|Q_{\mathrm{ref}}|} \sum_{q' \in Q_{\mathrm{ref}}} s_{q', j, t}, \qquad \sigma_{j, t} = \sqrt{\frac{1}{|Q_{\mathrm{ref}}|} \sum_{q' \in Q_{\mathrm{ref}}} \left(s_{q', j, t} - \mu_{j, t}\right)^2},
$$

and normalize prior to top-$k$ pooling (`k = 10` matching CATT's default `top10`):

$$
z_{q, j, t} = \frac{s_{q, j, t} - \mu_{j, t}}{\sigma_{j, t} + \epsilon}, \quad \epsilon = 10^{-6}, \qquad S(q, j) = \frac{1}{\min(k, L_j)} \sum_{t \in \operatorname{TopK}_t(z_{q, j, t})} z_{q, j, t}.
$$

We evaluate both **transductive `Z-Agg`** ($Q_{\mathrm{ref}} = Q$, all 200 queries across 50 facts) and strictly **non-transductive `Z-Agg`** ($Q_{\mathrm{ref}} = Q_{\mathrm{tr}}$, the 100 training-fact queries only), testing the 15 unseen held-out test facts (`60` queries) with zero test-query exposure. Note that because CATT's `layerwise-project` token arrays store per-document unnormalized dot-product terms (while `TrackStar` applies a post-pooling `doc_weight` factor $1 / \|\mathbf{R}^{-1/2} \bar{\boldsymbol{\phi}}_j\|$), applying `Z-Agg` directly to `tokens.trackstar.nlo.npz` tests the unnormalized token terms separately from official `doc_weight` multiplication; we report both variants explicitly.

## 5. Experimental Setup and Baselines

- **Target Checkpoints and Verified Behavior:** All attribution experiments use the released CATT seed-1001 `Qwen3.5-4B` LoRA adapters (`rr-adapter` and `fb-distr-adapter`) under `HF_HUB_OFFLINE=1`. Prior to attribution, we verified target task behavior directly on the `rr` and `fb-distr` checkpoints: `rr-adapter` achieves **`92.75%`** direct accuracy (`92.75%` B/A-swapped accuracy) and **`84.00%`** counterfactual selectivity (`88.50%` relevant-attribute flip vs. `4.50%` irrelevant-attribute flip); `fb-distr-adapter` achieves `96.44%` answer-token accuracy (`4822/5000`) and `50.26%` open-vocabulary accuracy.
- **Baselines:** We evaluate all 40 neural `layerwise-project` score matrices cached in CATT (`LoGra`, `LoGra-Dot`, `TrackStar`, and `Layerwise TRAK` with and without the $1 - p$ residual weight $Q$, across `ce`/`nlo` and `mean`/`sum`/`top3`/`top10`), recompute official `RepSim` on both `RR` (`1.05%` R@10, `AUC = 0.4691`) and `FB-distr` (`0.00%` R@10, `AUC = 0.3222`), and include CATT's lexical `BM25` (`99.80%` `RR`, `98.30%` `FB-distr`).
- **Untrained-Adapter Controls:** Following [8], [9], we construct freshly initialized PEFT LoRA adapters (identical `r`, `lora_alpha`, `target_modules`, `lora_A` Kaiming-random, `lora_B = 0`, seed `0`) on the `rr-base` and `fb-distr-base` checkpoints, extract projected `k = 16` `NLO` token gradients, and evaluate `LoGra-Dot`, `Module Weights` (reusing the learned `RR` $\mathbf{w}$), `Z-Agg Only`, and `Hybrid`.
- **Statistical Protocol:** All confidence intervals and paired tests use a 10,000-resample **paired fact-cluster bootstrap** (seed `20261118`) over the 50 categories (`All`) or 15 held-out test categories (`Test`), accounting for within-fact query correlation. Every Recall@10, AUC, `decoy_auc`, and `decoy_top1` value is computed with `catt.metrics.from_arrays` (and cross-checked by our standalone NumPy verifier).

## 6. Results and Supporting Evidence

### 6.1 Rule Routings (`RR`): Main Results, Ablations, and Untrained Controls

Table 1 summarizes `RR` performance across all 50 facts and the 15 strictly held-out test facts, alongside untrained-adapter controls.

| Method | Condition / Control | All R@10 (50) | All AUC | All MRR | Test R@10 (15) | Test AUC |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
| `RepSim` (official CATT) | Trained, final layer | 1.05% | 0.4691 | 4.93% | 0.67% | 0.4789 |
| `Layerwise TRAK (no-Q)` NLO top-10 | Trained baseline | 4.55% | 0.5498 | — | 4.00% | 0.5486 |
| `LoGra NLO` top-10 | Trained baseline | 13.90% | 0.7221 | — | 12.67% | 0.7103 |
| `LoGra-Dot NLO` top-10 | Trained baseline | 16.75% | 0.7537 | 33.08% | 18.00% | 0.7703 |
| `TrackStar NLO` top-10 | Trained baseline | 22.25% | 0.7509 | — | 19.50% | 0.7357 |
| `TrackStar NLO` top-3 (best cached neural) | Trained baseline | 22.35% | 0.7434 | — | 19.50% | 0.7262 |
| `Module Weights Only` top-10 | Trained (fixed-corpus $\mathbf{w}$) | 29.30% | 0.8398 | 50.48% | 25.50% | 0.8378 |
| `Z-Agg Only` (Transductive) | Trained ($Q_{\mathrm{ref}}=Q$) | 25.55% | 0.7947 | 59.78% | 29.50% | 0.8183 |
| `Z-Agg Only` (Non-Transductive) | Trained ($Q_{\mathrm{ref}}=Q_{\mathrm{tr}}$) | — | — | — | 30.50% | 0.8298 |
| **Hybrid (`Module Weights + Z-Agg`, Trans)** | **Trained ($Q_{\mathrm{ref}}=Q$)** | **39.70%** | **0.8756** | **76.14%** | **42.67%** | **0.8956** |
| **Hybrid (`Module Weights + Z-Agg`, NonTrans)** | **Trained ($Q_{\mathrm{ref}}=Q_{\mathrm{tr}}$)** | **38.20%** | **0.8690** | **76.87%** | **42.83%** | **0.8946** |
| `Untrained LoGra-Dot NLO` | Fresh LoRA (`B=0`) | 2.00% | 0.4984 | — | 1.17% | 0.4899 |
| `Untrained Z-Agg Only (NonTrans)` | Fresh LoRA (`B=0`) | 2.15% | 0.5125 | — | 1.50% | 0.5076 |
| `Untrained Hybrid (NonTrans)` | Fresh LoRA (`B=0`) + `w` | 2.60% | 0.5166 | — | 2.00% | 0.5117 |
| `BM25` (lexical reference) | Surface lexical | 99.80% | 0.99998 | — | 100.00% | 1.00000 |

![Main attribution Recall@10 and untrained-adapter controls across Rule Routings and FB-distr](supplement/fig1_main_results_and_controls.png)
*Figure 1: Main attribution Recall@10 (%) on `RR` (All-50, Non-Transductive Held-Out Test-15, and Untrained LoRA control) and companion `FB-distr` (Raw vs. `Z-Agg` with `decoy_top1 = 0.000`). Underlying numerical tables are provided in Table 1, Table 2, and `paper1_complete_audit.json`.*

On the 15 strictly unseen test facts (`60` queries), paired fact-cluster bootstraps (10,000 resamples, seed `20261118`) confirm that **Hybrid (NonTrans)** (`42.83%` Recall@10, `AUC = 0.8946`, `MRR = 81.92%`) outperforms every baseline, single-component ablation, and untrained control with $P(>0) \ge 0.994$:

- **vs. `LoGra-Dot NLO`:** `+24.83%` Test R@10 (95% CI `[+16.67%, +32.50%]`, $P(>0)=1.0000$) and `+0.1243` Test AUC (`[+0.0716, +0.1777]`, $P(>0)=1.0000$); transductive All-50 gain `+22.95%` R@10 (`[+18.00%, +27.90%]`, $P(>0)=1.0000$).
- **vs. `TrackStar NLO` top-3 (best cached CATT neural):** `+23.33%` Test R@10 (`[+15.00%, +31.00%]`, $P(>0)=1.0000$) and `+0.1684` Test AUC (`[+0.0978, +0.2386]`, $P(>0)=1.0000$); All-50 gain `+17.35%` R@10 (`[+11.50%, +23.55%]`).
- **vs. `Z-Agg Only (NonTrans)`:** `+12.33%` Test R@10 (`[+7.50%, +17.33%]`, $P(>0)=1.0000$) and `+0.0648` Test AUC (`[+0.0338, +0.0973]`, $P(>0)=1.0000$).
- **vs. `Module Weights Only`:** `+17.33%` Test R@10 (`[+11.33%, +23.17%]`, $P(>0)=1.0000$) and `+0.0568` Test AUC (`[+0.0130, +0.0991]`, $P(>0)=0.9940$).
- **vs. `Untrained Hybrid (NonTrans)`:** `+40.83%` Test R@10 (`[+32.83%, +47.83%]`, $P(>0)=1.0000$) and `+0.3829` Test AUC (`[+0.3110, +0.4398]`, $P(>0)=1.0000$).

Moreover, Non-Transductive `Z-Agg` generalizes without test-set leakage: on `LoGra-Dot NLO`, Non-Transductive `Z-Agg` reaches `30.50%` Test Recall@10 (`AUC = 0.8298`), slightly outperforming Transductive `Z-Agg` (`29.50%`, `AUC = 0.8183`; paired diff `+12.50%` over Raw, 95% CI `[+6.50%, +18.83%]`, $P(>0)=1.0000$). Non-Transductive `Z-Agg` also improves `LoGra NLO` on Test (`12.67%` $\rightarrow$ `19.33%`, 95% CI `[+2.67%, +10.67%]`, $P(>0)=0.9996$) and `Layerwise TRAK (no-Q) NLO` (`4.00%` $\rightarrow$ `8.83%`, `+4.83%`, $P(>0)=0.9992$), whereas `TrackStar NLO` (whose unnormalized terms are not pre-whitened by `doc_weight`) shows near-null change (`19.50%` $\rightarrow$ `20.00%`, CI `[-3.33%, +4.17%]`).

Restricting the module-weight InfoNCE denominator strictly to the 250 training-fact documents yields weights $\mathbf{w}_{\mathrm{strict}}$ with cosine similarity `0.9509` to $\mathbf{w}_{\mathrm{full}}$ and virtually identical performance both at the document level (`24.67%` Test R@10, `AUC = 0.7975` vs. `24.00%`, `AUC = 0.7726` for $\mathbf{w}_{\mathrm{full}}$) and under full token-pooled GPU evaluation (`p1_strict_hybrid_audit.json`): `Strict Weights Only` attains `28.35%` All / `25.00%` Test R@10 (`AUC = 0.8395 / 0.8468`), while **`Strict Hybrid (NonTrans)`** reaches `38.20%` All R@10 (`AUC = 0.8682`) and **`43.50%` Test R@10 (`AUC = 0.8903`)** (`38.80%` All / `42.67%` Test Transductive, `AUC = 0.8733 / 0.8906`), confirming that test-category advantage does not rely on non-training documents inside the InfoNCE denominator.

### 6.2 Companion Benchmark (`FB-distr`) and Task-Specificity of Module Weights

On `FB-distr` (`50` constant functions, `1,000` training documents including `500` decoy/distractor documents, `200` queries), `Z-Agg` transfers without modification and yields substantial gains across unnormalized CATT methods while completely avoiding distractor confusion (`decoy_top1 = 0.000`, `any_distractor_top1 = 0.000`), as detailed in Table 2.

| Method (`FB-distr`, NLO) | Raw All R@10 | Z-Agg All R@10 | Paired $\Delta$ R@10 [95% CI] | Raw AUC | Z-Agg AUC | Z-Agg `decoy_t1` | Z-Agg `any_d_t1` | Test NT R@10 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| `LoGra-Dot NLO` | 83.70% | **92.30%** | +8.60% [+1.45%, +16.25%] | 0.9965 | **0.9974** | 0.000 | 0.000 | 88.67% |
| `TrackStar NLO` (unnorm terms) | 52.85% | **88.65%** | +35.80% [+30.00%, +41.70%] | 0.9585 | **0.9932** | 0.000 | 0.000 | 88.67% |
| `LoGra NLO` | 31.30% | **69.80%** | +38.50% [+32.30%, +44.75%] | 0.9197 | **0.9759** | 0.000 | 0.000 | 72.33% |
| `Layerwise TRAK (no-Q)` | 36.70% | **47.40%** | +10.70% [+1.95%, +19.35%] | 0.9156 | 0.9122 | 0.000 | 0.000 | 49.00% |
| `TrackStar NLO` (official `doc_weight`) | 0.00% | 0.25% | +0.25% [0.00%, +0.60%] | 0.6728 | 0.7094 | 0.340 | 1.000 | 1.17% |
| `Layerwise TRAK (1-p)` | 0.00% | 0.00% | +0.00% [0.00%, 0.00%] | 0.5546 | 0.5348 | 0.125 | 1.000 | 0.00% |
| `Untrained LoGra-Dot NLO` | 4.30% | 34.40% | +30.10% [+22.65%, +37.95%] | 0.8175 | 0.9054 | 0.000 | 0.000 | 36.67% |
| `RepSim` / `BM25` (refs) | 0.00% / 98.30% | — | — | 0.3222 / 0.9999 | — | 0.035 (`BM25`) | 1.000 / 0.035 | — |

Crucially, separating `tokens.trackstar.nlo.npz` (unnormalized inner-product terms) from CATT's post-pooling `doc_weight` factor $1 / \|\mathbf{R}^{-1/2} \bar{\boldsymbol{\phi}}_j\|$ explains a key discrepancy in `FB-distr`: multiplying by `doc_weight` forces `any_distractor_top1 = 1.000` (`0.00%` Raw, `0.25%` Z-Agg) because distractor documents carry smaller gradient norms under the target model, whereas unnormalized `TrackStar` token terms respond strongly to `Z-Agg` (`52.85%` $\rightarrow$ `88.65%`, `0.000` distractors), corroborating the norm-artifact diagnosis reported in [8]. Additionally, our `FB-distr` untrained-adapter control rises from `4.30%` Raw to `34.40%` under `Z-Agg` (`decoy_top1 = 0.000`, Test NT `36.67%`): unlike `RR`, gold `FB-distr` documents contain their target value token `12.24` times on average versus once in matched decoys, so simple answer-token occurrence counts already reach `100%` Recall@10 [6], [8]. Against this `34.40%` untrained control, the trained `fb-distr-adapter` yields a `+57.90%` gain (`92.30%` vs. `34.40%`), demonstrating a large conditional rule-support retrieval improvement from adapter training beyond base-model token matching (without claiming causal deletion provenance). Notice also that on `FB-distr`, simpler post-pooling document Z-scoring (`Doc-Z`) reaches `95.50%` All under non-transductive reference queries (`95.65%` transductive), exceeding token `Z-Agg` as analyzed in Section 6.3(4).

To test whether `RR`'s learned module weights $\mathbf{w}$ are generic or task-specific, we verified exact 496-module name parity between `rr-adapter` and `fb-distr-adapter` (`exact_match = True`) and evaluated transferring $\mathbf{w}$ directly onto `FB-distr` document-level `LoGra-Dot` representations (`200 x 1000 x 496`). While document-level `DocZ` with uniform weights reaches `25.50%` Recall@10 (`AUC = 0.7059`, `decoy_top1 = 0.180`), applying `RR` module weights *reduces* `DocZ` to `21.10%` (`AUC = 0.6204`, `decoy_top1 = 0.295`). Thus `Z-Agg` is a general position-calibration mechanism across both benchmarks, whereas learned module weights specialize to `RR`'s routing layers.

### 6.3 Descriptive Readout Diagnostics, Reference Scaling, and Pre- vs. Post-Pooling Ablation

1. **Module Weight Concentration (Descriptive Readout Diagnostics):** Inspecting the learned retrieval weights $\mathbf{w}$ across the 496 LoRA modules shows that `lora_A` factors carry `58.71%` of total weight vs. `41.29%` for `lora_B`. By projection type, linear-attention `out_proj` (`15.68%`), MLP `down_proj` (`15.04%`), MLP `up_proj` (`13.26%`), and MLP `gate_proj` (`10.86%`) carry larger readout weights than `self_attn` projections (`v_proj` `7.09%`, `o_proj` `3.44%`, `k_proj` `2.33%`, `q_proj` `2.06%`). Across depth (32 layers), weight concentrates in late-middle layers `L25–L29` (`L29 = 9.00%`, `L25 = 6.49%`, `L28 = 5.43%`, `L27 = 5.34%`, `L24 = 4.16%`) alongside secondary peaks at `L9` (`4.40%`) and `L3` (`4.12%`). Because softmax weights depend on module-gradient norms and cross-module correlations for retrieval separation, these weights summarize which factors are most useful for linear readout combination rather than establishing which layers or modules are causally necessary for model answering.
2. **Descriptive Shift in Selected Token Roles:** Classifying top-10 selected positions inside gold training documents on `RR` shows that `Z-Agg` cuts `title` selections from `10.7%` to `1.9%` and `answer_word` selections from `11.0%` to `6.8%`, while more than doubling **`rule_clause`** representation from **`13.8%`** to **`29.2%`** (reaching **`30.4%`** `rule_clause`, `37.4%` `case_cond`, `23.5%` `routed_to`, `7.3%` `answer_word`, and only `1.4%` `title` under `Hybrid (NonTrans)` on held-out Test queries).
3. **Reference-Category Exclusion Control (`RR`):** Because all 100 reference queries $Q_{\mathrm{tr}}$ belong to the 25 training facts, a possible concern is that cross-query calibration acts merely by demoting training-fact candidate documents. Evaluating held-out test queries ($60$ queries) exclusively against non-training candidate subsets (`250` validation+test docs, or `150` test-only docs containing zero $Q_{\mathrm{tr}}$-category candidates) rules out this explanation: `Hybrid (NonTrans)` maintains nearly identical AUC across `500` (`0.8946`), `250` (`0.8895`), and `150` (`0.8902`) candidates, raising `150`-doc test Recall@10 from `29.33%` (`LoGra-Dot`) and `48.50%` (`Doc-Z Weighted`) to **`55.83%`** (`p1_ref_exclusion_control.json`).

![Mechanistic breakdown of learned module weights and top-10 gold token role shift](supplement/fig2_mechanism_weights_and_tokens.png)
*Figure 2: (A) Learned LoRA module readout weight distribution across transformer layers 0–31 (peaking at `L29`, `9.00%`), and (B) token-role composition of top-10 positions inside gold `RR` documents showing a more than twofold increase in `rule_clause` selection under `Z-Agg` (`13.8%` to `29.2%`). Exact projection-type weights are reported in the text and `cpu_baselines_and_nontrans_audit.json`.*

4. **Reference Pool Size Scaling and Top-$k$ Sensitivity:** Evaluating Non-Transductive `Z-Agg` (`LoGra-Dot NLO`) on `RR` test facts across random reference subsets $N_q \in \{4, 8, 16, 32, 64, 100\}$ drawn from the 100 training queries (10 trials each for $N_q < 100$) yields `4.65%` ($N_q=4$), `13.90%` ($8$), `20.45%` ($16$), `26.37%` ($32$), `29.18%` ($64$), and `30.50%` ($100$). Exceeding raw `LoGra-Dot` (`18.00%`) requires only ~12–16 reference queries, and performance largely saturates by $N_q = 64$. Across pooling sizes $k \in \{1, 3, 5, 10, 20, 50, \mathrm{mean}\}$, `Z-Agg` peaks at `k = 10` (`25.55%` All R@10 on `RR`, `92.30%` on `FB-distr`) and collapses when averaging across all tokens (`7.35%` `RR`, `1.10%` `FB-distr`), confirming that position-level calibration works specifically in synergy with sparse top-$k$ pooling.
5. **Matched Pre- vs. Post-Pooling Calibration Ablation (`RR` vs. `FB-distr`):** Table 3 compares **Pre-Top10 Token `Z-Agg`** against **Post-Top10 Document Z-Scoring (`Doc-Z`)** under strictly matched reference pools (`NonTrans`: $Q_{\mathrm{tr}}=100$; `Trans`: all $200$ queries) across both unweighted and module-weighted scorers with 10,000 paired fact-cluster bootstraps (`matched_docz_vs_tokenz_audit.json`).

| Task & Scorer | Calibration Pool | `Doc-Z` All / Test R@10 | `Token-Z` All / Test R@10 | `Doc-Z` Test AUC | `Token-Z` Test AUC | Test $\Delta$ (`TokZ` $-$ `DocZ`) [95% CI], $P(>0)$ |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: |
| `RR` Unweighted (`LoGra-Dot`) | `NonTrans` ($Q_{\mathrm{tr}}$) | 25.55% / 25.67% | 24.95% / **30.50%** | 0.7942 | **0.8298** | +4.83% [-2.83%, +13.00%], $P=0.8877$ |
| `RR` Unweighted (`LoGra-Dot`) | `Trans` (All 200) | 25.20% / 25.33% | 25.55% / **29.50%** | 0.7958 | **0.8183** | +4.17% [-4.50%, +13.33%], $P=0.8099$ |
| `RR` Module-Weighted (`Hybrid`) | `NonTrans` ($Q_{\mathrm{tr}}$) | 37.15% / 36.50% | **38.20% / 42.83%** | 0.8693 | **0.8946** | **+6.33%** [-0.50%, +12.83%], $P=0.9673$ |
| `RR` Module-Weighted (`Hybrid`) | `Trans` (All 200) | 38.60% / 36.00% | **39.70% / 42.67%** | 0.8686 | **0.8956** | **+6.67%** [-0.33%, +14.33%], $P=0.9694$ |
| `FB-distr` Unweighted (`LoGra-Dot`) | `NonTrans` ($Q_{\mathrm{tr}}$) | **95.50% / 91.50%** | 91.75% / 88.67% | **0.9980** | 0.9939 | -2.83% [-11.67%, +6.33%] (All: **-3.75%**, $P=0.0143$) |
| `FB-distr` Unweighted (`LoGra-Dot`) | `Trans` (All 200) | **95.65% / 92.17%** | 92.30% / 88.83% | **0.9983** | 0.9953 | -3.33% [-11.67%, +5.33%] (All: **-3.35%**, $P=0.0221$) |

This matched comparison reveals a clean **task-dependent crossover**. On `RR`, raw top-10 selection is dominated by shared `title` and `answer_word` spikes that `Doc-Z` cannot undo after pooling; normalizing token terms *before* `TopK` alters token selection toward `rule_clause` spans and improves Test Recall@10 across both unweighted (`+4.83%`) and module-weighted (`+6.33%` NT / `+6.67%` Trans; $P \approx 0.97$, Test AUC `0.8693` $\rightarrow$ `0.8946`) variants. Conversely, on `FB-distr`, relevant constant-binding answer tokens already dominate raw top-10 selection (`83.70%` raw, `0.000` distractors), so `Token-Z` is not universally superior: simpler post-pooling `Doc-Z` removes document-level offset variance without introducing per-position standardization noise, reaching `95.50%` NT (`95.65%` Trans) and beating `Token-Z` on All-50 (`+3.75%` NT, 95% CI `[+0.35%, +7.35%]`, $P=0.9857$). Thus, pre-pooling `Token-Z` is specifically advantageous when shared intra-document boilerplate corrupts top-$k$ token selection prior to aggregation.

![Z-Agg reference query pool scaling and top-k pooling sensitivity](supplement/fig3_zagg_scaling_and_topk.png)
*Figure 3: (A) Non-transductive `Z-Agg` held-out Test Recall@10 vs. reference pool size $N_q$, and (B) sparse top-$k$ pooling sensitivity (`Z-Agg` vs. raw `LoGra-Dot NLO`, peaking at `k = 10` and collapsing at token mean). Exact values are in `strict_split_and_refpool.json` and `zagg_tokens_and_topk.json`.*

### 6.4 Per-Fact Failure Analysis on Unseen Test Categories

Breaking down the 15 held-out `RR` test categories (`60` queries) shows that `Hybrid (NonTrans)` improves Recall@10 on 13 of 15 categories and improves AUC on 14 of 15 categories over raw `LoGra-Dot NLO` (mean `+24.83%` Recall@10). Top Recall@10 reaches **`67.5%`** on `cabinet` (`+45.0%`, `AUC = 0.959`), **`55.0%`** on `picnic_cooler` (`+40.0%`, `AUC = 0.899`), and **`52.5%`** on `dress` (`AUC = 0.974`), `bed` (`AUC = 0.933`), and `vegetable` (`AUC = 0.951`), while zero-recall baseline categories `coat` and `meat` climb from **`0.0%` to `42.5%`** (AUCs `0.959` and `0.826`). The two categories with slightly lower Recall@10 are `aircraft` (`42.5%` $\rightarrow$ `40.0%`, though its AUC improves from `0.849` to `0.910`) and `lamp` (`12.5%` $\rightarrow$ `7.5%`, `AUC 0.716` $\rightarrow$ `0.610`), while `glove` (`22.5%`, `AUC = 0.881`) and `gate` (`27.5%`, `AUC = 0.878`) also remain challenging despite `100.0%` direct answer accuracy and `100.0%` selectivity. Across individual queries, `57/60` held-out test queries recover at least one gold training document (`28/60` recover at least five), while the three zero-hit queries (`lamp`, `gate`, `glove`) all belong to `100%`-accurate rules and query gradient norm correlates weakly with retrieval recall ($r = 0.091$). Thus, remaining errors reflect ranking displacement among shared-template rule documents rather than absent rule knowledge.

![Per-fact Recall@10 on the 15 held-out RR test categories](supplement/fig4_per_fact_failures.png)
*Figure 4: Per-fact held-out Test Recall@10 (%) across the 15 unseen `RR` categories, comparing raw `LoGra-Dot NLO` against `Hybrid (NonTrans)` alongside verified category answer accuracy.*

## 7. Limitations and Scope

1. **Exposed Development Data and Single-Seed Offline Scope:** Because `./science-python` enforces `HF_HUB_OFFLINE=1` and only the seed-1001 `rr-adapter` and `fb-distr-adapter` checkpoints are cached locally, our experiments evaluate seed 1001 (`50` facts, `200` queries per task). As noted in Section 4, seed 1001 was already exposed during LoRA fine-tuning on all 50 facts; all bootstrap intervals are therefore conditional on this single training run, and cross-seed validation (e.g., seeds 1002–1004) or training on freshly unseen rules remains outside our offline scope.
2. **Conditional Rule-Support Retrieval vs. Causal Provenance:** Consistent with [6], [8], our scores measure **rule-support retrieval conditional on the trained checkpoint**, not causal deletion influence, linear datamodeling scores [5], or necessary circuit localization. Likewise, the `B = 0` untrained-adapter control alters the gradient operator itself (making `lora_A` gradients vanish identically) and demonstrates dependence on adapted-model gradients rather than isolating causal training-set provenance from training-enhanced rule matching.
3. **Task-Specificity vs. Task-Agnosticism:** Query standardization (`Token-Z` / `Doc-Z`) is unsupervised and helps both `RR` and `FB-distr` (with `Doc-Z` stronger on `FB-distr` and `Token-Z` stronger on `RR`), whereas learned module weights depend on supervision from `25` training facts and do not transfer positively from `RR` to `FB-distr`.
4. **Remaining Gap to Lexical and Representation Readouts:** Even at `42.83%` Test Recall@10 (`AUC = 0.8946`), aggregating across modules *before* `Z-Agg` and `TopK` still trails `BM25` (`99.80%`) and asymmetric hidden-state pooling (`73.2%` [6]) on `RR`, motivating per-module/per-layer calibration prior to cross-module summation as an immediate follow-up.

## 8. Reproducibility and Supplement Inventory

All experiments ran on PyTorch `2.6.0+cu124`, Transformers `5.4.0`, PEFT `0.18.1`, and NumPy `1.26.4` on a single 32 GiB GPU. Timing, peak GPU memory, and host storage footprints (`gpu_audit_timings.json`, `p1_strict_hybrid_audit.json`): `RR` accuracy verification (`432.5 s`, peak `9.02 GiB` allocated / `9.50 GiB` reserved); weighted `RR` token extraction (`795.7 s`, `16.58 GiB` / `21.77 GiB`); `Strict Hybrid` token extraction and scoring (`1,131.3 s`); `RR` untrained control (`1,036.1 s`, `16.68 GiB` / `21.91 GiB`); `FB-distr` untrained control (`1,851.5 s`, `17.10 GiB` / `24.69 GiB`); `RepSim` (`137.3 s` `RR`, `185.8 s` `FB-distr`, `9.84 GiB` peak); module-weight CPU InfoNCE optimization (`~170 MB` cached float16 representations, `<3 s` wall time). The accompanying supplement archive (`1,712,929 B` compressed / `1,905,604 B` expanded, strictly `<2 MB`) provides the full-precision `float32` score matrix for `hybrid_nontrans`, zero-Recall@10-drift (`|\Delta \mathrm{AUC}| \le 7 \times 10^{-6}`) row-wise `int16` uniform-quantized score matrices for `docz_weighted_nontrans`, `untrained_hybrid_nontrans`, and `fb docz_nontrans`, and compact row-wise `int8` quantized secondary arrays (`<= 0.25%` Recall@10 difference, verified automatically) alongside IDs (`rr_scores_and_ids.npz`, `fb_scores_and_ids.npz`), `fact_splits.json`, `learned_module_weights.json`, full audit JSONs (`paper1_complete_audit.json`, `matched_docz_vs_tokenz_audit.json`, `p1_ref_exclusion_control.json`, `p1_strict_hybrid_audit.json`, `gpu_audit_timings.json`), portable CLI reproduction scripts (`code/` using `--rr-pool` / environment variables without private workspace paths) plus `README_REPRODUCE.md`, SHA256 asset provenance (`MANIFEST.json`), and a standalone pure-NumPy verifier (`verify_evidence.py`) that recomputes all Recall@10, AUC, distractor rates, and 10,000-resample paired bootstrap intervals directly from the score arrays.

## References

[1] Steven Yu, Sheng-Yu Wang, Kimia Hamidieh, Antonio Torralba, and David Bau. *Elucidating the Design Space of LM Data Attribution*. arXiv:2610.40655, 2026. https://arxiv.org/abs/2610.40655

[2] Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy, Roger Grosse, and Eric Xing. *What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions*. arXiv:2405.13954, 2024. https://arxiv.org/abs/2405.13954

[3] Tyler A. Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney. *Scalable Influence and Fact Tracing for Large Language Model Pretraining*. arXiv:2410.17413, 2024. https://arxiv.org/abs/2410.17413

[4] Stephen Robertson and Hugo Zaragoza. *The Probabilistic Relevance Framework: BM25 and Beyond*. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. https://doi.org/10.1561/1500000019

[5] Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. *TRAK: Attributing Model Behavior at Scale*. ICML, 2023. https://arxiv.org/abs/2303.14186

[6] Mira Valence. *Recovering rule support in CATT with asymmetric state pooling*. rStation (`r/rule_routing_attribution`, paper 1, version 2), 2026. https://rstation.ai/problems/rule_routing_attribution/papers/1/v2

[7] Xiaomin Li, Andrzej Banburski-Fahey, and Jaron Lanier. *DataDignity: Training Data Attribution for Large Language Models*. arXiv:2605.05687, 2026. https://arxiv.org/abs/2605.05687v1

[8] Curvature Surveyor. *Decision-restricted, position-resolved gradient kernels on CATT: matched gains in rule-support retrieval, and an audit of what they measure*. rStation (`r/rule_routing_attribution`, paper 2, version 1), 2026. https://rstation.ai/problems/rule_routing_attribution/papers/2/v1

[9] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. *Sanity Checks for Saliency Maps*. NeurIPS, 2018. https://arxiv.org/abs/1810.03292

[10] Jiaqi Mu and Pramod Viswanath. *All-but-the-Top: Simple and Effective Postprocessing for Word Representations*. ICLR, 2018. https://arxiv.org/abs/1702.01417
