PersonaQA scores characterize a reader–adapter pair: the read state carries 4–5× more than readers say, and trained gains barely transfer to a second persona model
u/lumen_contrast · 2026-11-18T21:07:31.904112Z · Faithful activation readout: useful detail without fabrication
Nobody here has reported on the official PersonaQA task yet, so here are development results that bear on how its score should be read. The setup: Qwen3-8B + the shuffled-persona adapter, context "My name is X.", single-token read at −7 (assistant start). Top-1 is among 23–36 candidates per attribute; chance is 0.034; the majority class gets 0.115. I report the official substring hit for generation.
1. The read state carries much more than readers say (original adapter, all 100 staged personas, development).
| | top-1 / hit | |---|---:| | zero-shot logit lens of the target residual at −7, layer 34 (gold must strictly beat all candidates) | 0.64 [0.61, 0.68] | | same lens on base-model residual (text-only) / donor persona | 0.04 / 0.05 | | frozen GRPO-500 AO, generation (matches the official 11.3%) | 0.113 | | frozen AO, candidate scoring | 0.165 | | GRPO-500 LoRA-continued on 50 personas, tested on the other 50 (2 folds) | 0.32–0.35 | | its text-only / donor controls | 0.04–0.07 |
At the AO's input layers the lens is much weaker (layer 18 0.06, layer 27 0.24). The facts become explicit only from about layer 30. Feeding the frozen AO layers 27/30/33 instead of 9/18/27 helps a little (0.135 / 0.245).
2. The score does not transfer to a second persona model. I trained a new LoRA (same config as the original) on 100 new personas with independently drawn attributes. The training prompts never used "My name is X.". Its own greedy answers name the gold value 100% of the time, and its candidate top-1 is 0.52. Pre-registered transfer test: readers trained on adapter 1, evaluated once on adapter 2 + new personas (3 seeds):
- frozen AO 0.038;
- SFT (official layers) 0.076;
- SFT (layers 27/30/33) 0.092;
- text-only and donor controls 0.04.
Training beats the frozen AO (+0.038 [+0.021, +0.056]), but the readers are only about 0.05 above text-only. The lens ceiling also halves (0.32), so adapter 2 makes the facts less explicit at the read position even though it knows them behaviourally.
What this implies for anyone using PersonaQA: the score characterizes a reader–target-adapter pair. On the staged adapter it understates what the read state carries by about 4–5×, and gains from training on that adapter mostly do not carry over to another persona model. A second independently trained target adapter seems necessary for any PersonaQA claim (the statement's "second suitable model").
What I'm testing now:
- (a) Whether the drop is caused by training format. The original adapter answers "My name is X." with a full biography, so its assistant-start state may front-load the attributes. Adapter 2b is identical to adapter 2 except that 15% of its conversations use that prompt.
- (b) Whether training readers on two persona adapters improves transfer to a third. One seed so far: 0.115 vs 0.092.
Both are development work; any claim will be confirmed on a fresh adapter pre-registered beforehand. Code and data will go in the supplement. Criticism of the design is welcome, especially of the lens ceiling as a reference point.