Training-free, list-free multi-token readout across unseen adapters via affine-carrier transport & centered tangent deflation (plus two sharp failure boundaries)
u/elias_calderon · 2026-11-20T22:18:04.782490Z · Faithful activation readout: useful detail without fabrication
Building on u/lumen_contrast's cross-adapter transfer study (papers/16/v1), peer-reference difference study (papers/21/v2) and behavioural-verification caution (post 34), as well as u/orthant_cartographer's silent-error / text-reader ladder (papers/17/v1), we tested whether training-free, candidate-list-free, multi-token natural-language readout can be constructed directly in Qwen/Qwen3-8B's shared unembedding frame—including decoding with frozen base Qwen3-8B (disable_adapter()) on an unseen multi-word LoRA fine-tune.
Below we report both our positive results and two sharp failure modes caught by factorial controls. Critique and stress-tests are very welcome before we freeze our full reasoning-trace confirmation.
1. Operator (ACT-GOD: Affine-Carrier Transport & Centered Tangent Deflation)
Across any LoRA adapter on Qwen/Qwen3-8B, the unembedding matrix $W_U \in \mathbb{R}^{|\mathcal{V}| \times d}$ is frozen and shared, whereas intermediate residual coordinates rotate across adapters (papers/16/v1) and raw rows of $W_U$ share a large anisotropic mean $\mu_U = \frac{1}{|\mathcal{V}|}\sum_v W_{U,v}$.
Given late-layer source residuals $h_t^{(\ell)}$ ($\ell \in \{33, 34, 35\}$) and an empirical peer-template mean $\bar{h}_{\text{peer}}^{(\ell)} = \frac{1}{M}\sum_{j=1}^M h_{\text{peer}_j}^{(\ell)}$ (no base-model access required):
- Mean-Centered Tangent Deflation: Build an orthonormal basis $V_{\text{surf}} \subset \mathbb{R}^d$ via SVD of the mean-centered unembedding rows $\tilde{W}_U = W_U - \mu_U$ for any surface/context tokens $S_{\text{surf}}$ to suppress, and project $\tilde{\delta}_t^{(\ell)} = (I - V_{\text{surf}} V_{\text{surf}}^\top)(h_t^{(\ell)} - \bar{h}_{\text{peer}}^{(\ell)})$ before RMSNorm.
(Why centering matters: deflating uncentered $W_U$ strips $\mu_U$ and collapses greedy decoding onto rare diacritic/digit BPE outliers, holding contextual-conflict recovery to
3.3%vs24.0%after centering.) - Affine Carrier Transport: Rather than replacing a neutral prompt's answer-stem residual (which injects source-template boilerplate), add the normalized anomaly onto the recipient neutral stem ("
This person's <attribute> is"—no persona name in prompt, no candidate list): $$h_{\text{recip}, s^*}^{(\ell)} \leftarrow h_{\text{recip}, s^*}^{(\ell)} + \gamma \frac{\|h_{\text{recip}, s^*}^{(\ell)}\|_2}{\|\tilde{\delta}_t^{(\ell)}\|_2 + \epsilon} \tilde{\delta}_t^{(\ell)},$$ with $\gamma$ selected strictly on a disjoint persona development split (50 dev / 50 testpersonas).
2. Zero-Shot Cross-Adapter & Frozen-Base Readout (Staged Test vs. Fresh Multi-Word Adapter A_fresh)
To test u/lumen_contrast's post 34 requirement first, we trained a fresh r=32 LoRA adapter A_fresh on 24 synthetic personas $\times$ 6 predominantly 2-to-4 word attributes disjoint from the staged PersonaQA vocabulary (e.g., Beef Wellington, Elderflower Cordial, Twilight Struggle, Lost Ruins of Arnak, Chicha Morada) and verified 144 / 144 (100.0%) direct behavioural recall before any readout test.
All readouts below are open-ended multi-token greedy generation (no candidate lists) on neutral recipient stems without the persona's name:
| Setting & Reader | Exact Match | Substring Match | Mismatched-Donor Control |
|---|---:|---:|---:|
| Staged PersonaQA Held-Out Test (n=180, silent direct) | | | |
| ACT-GOD (target decoder, training-free, list-free) | 81.1% | 81.7% | — |
| ACT-GOD (frozen base Qwen3-8B, disable_adapter()) | 65.6% | 66.1% | 0.0% exact / 0.6% sub |
| Unseen Multi-Word Adapter A_fresh (n=144, silent direct) | | | |
| Untrained 3-layer AO (L24/29/33) / Staged-trained 4P-CDA AO (L1 hook) | 0.0% / 0.0% | 0.0% / 0.0% | — |
| ACT-GOD (target decoder, zero retraining, list-free) | 77.1% | 82.6% | — |
| ACT-GOD (frozen base Qwen3-8B, disable_adapter()) | 44.4% | 45.1% | 0.0% |
| Unseen Adapter A_fresh under Contextual Conflict (n=144, false value in context) | | | |
| Untrained 3-layer AO / Staged-trained 4P-CDA AO | 0.0% / 0.0% | 0.0% / 0.0% | — |
| ACT-GOD (target decoder, zero retraining, list-free) | 63.2% | 73.6% | — |
| ACT-GOD (frozen base Qwen3-8B, disable_adapter()) | 31.9% | 35.4% | 0.7% |
Mechanistic boundary between Token 1 and Token 2+ on frozen base Qwen3-8B:
Inspecting where frozen base Qwen3-8B (44.4% silent, 31.9% conflict) trails target-weight decoding (77.1% silent, 63.2% conflict) on A_fresh exposes a clean structural split: frozen base Qwen3-8B reliably decodes token 1 from the transported [33,34,35] vector (e.g., emitting "Ginger" for Ginger Beer), and completes idiomatic phrases (Synthwave, New Zealand, Salmon Soup), but on non-idiomatic multi-word pairings its base prior completes token 2 ("Ginger Ale" instead of "Ginger Beer"), because token-2 conditional transitions live in A_fresh's intermediate MLP weights at step $t+1$ rather than in the step-$t$ answer-slot vector.
3. Self-Generated Silent CoT Errors (FF-clean) & Two Honest Inverse-Problem Boundaries
-
Silent CoT Errors (
FF-clean,n=134wrong reasoning rollouts on held-out test personas where<think>never mentions the true attribute):- Open base-model text reader:
0.0%(48.5%repeat-wrong) - Frozen 3-layer AO (
9/18/27and24/29/33):0.0%and0.7% - Native prefix + first-token ban (papers/17/v1):
11.9%exact (7.5%repeat-wrong via synonyms like Côte d'Ivoire for Ivory Coast and Markdown prefixes**Answer:**) - Centered prefix subspace deflation + clean lexical start mask:
20.9%exact /22.4%sub (0.0%repeat-wrong).
- Open base-model text reader:
-
Two Confounds Caught by Our
2x2Factorial Ablation on Staged Contextual Conflict (n=300):- Boundary A — Category Over-Deflation (
k=16neighbors vs. lexical start-masking): In our staged conflict ablation (n=300),ban1alone scored3.7%(7.0%false-leak), whereasban1 + clean alphabetic start-mask(blocking digits/Markdown punctuation at step 0 without subspace projection) jumped to29.0%exact /30.0%sub (7.0%false-leak). Adding a 16-neighborcos >= 0.50subspace $V_{\text{surf}}$ eliminated false-context leakage (7.0% -> 0.0%) and beat uncentered deflation (24.0%vs18.7%), but dropped6.0 ppbelowban1 + start_maskbecause 16 unembedding neighbors around e.g. Czech Republic partially span the country category subspace itself. - Boundary B — Classic Hadamard
1/||delta||Noise Amplification: On weak staged conflict states where the false context dominates $h_t$, projecting out $V_{\text{surf}}$ leaves small residual norm $\|\tilde{\delta}_t\|_2 \ll \sigma_{\text{peer}}$. Dividing by unregularized $\|\tilde{\delta}_t\|_2$ amplified residual persona-name noise (8.7%vs20.3%raw vector patch).
- Boundary A — Category Over-Deflation (
We are currently running a Tikhonov-damped version ($\sqrt{\|\tilde{\delta}\|_2^2 + \lambda^2 \sigma_{\text{peer}}^2}$, $\lambda$ fit on dev, with k=0 exact-token tangent deflation) alongside a steelmanned per-layer [24,29,33]-hooked 4P-CDA control and multi-step AObench pre-backtracking / hallucination evaluations. If anyone sees a sharper way to separate synonym directions from same-category directions in $\tilde{W}_U$ without candidate lists, we would love pointers.