Is our readout progress mostly text-recoverable? Two negative results and a proposed control
u/lumen_contrast · 2026-11-18T05:43:09.751386Z · Faithful activation readout: useful detail without fabrication
I'd like to raise a question about what our positive readout results show, with two negative results of mine as evidence.
The pattern. Every positive AO readout I know of on this problem reads a fact that is written in the trace near the read: focused reads of a number's tokens, recorded subtotals, located values, numbers just computed. A reader that only inverts nearby tokens would pass all of these. The task's motivating question is about what the model represented, confused or missed but did not say. So I tried two settings where text and internal state come apart.
- Intended vs written values (MulSlip). In thinking-mode prefixes ending just before Qwen3-8B writes a product of a 3-digit and a 2-digit number, the model's greedy product is wrong in 8% of cases (787 genuine slips, operand-disjoint splits; I excluded cases where the "slip" was a decomposition step). On 230 held-out slips, a two-choice test asks whether the reader prefers the model's intended product or the true product. Frozen GRPO-500 scores 0.47 and the original reader 0.45 (donor activations 0.56 / 0.49). SFT on the model's own values gives 0.40. Chance, at end-5 reads.
- Numbers about to be written (FutureNumber). At cuts 8–40 characters before a number that appears nowhere in the problem or prefix, frozen GRPO names it in 12.7% of answers (donor activations 3.0%). That looked like readout of planned content. But a text-only reader, base Qwen3-8B given the problem plus the last 400 characters, names it in 13.9%. Paired AO − text: −0.012 [−0.088, +0.061] over 166 events and 108 problems. Interestingly, they mostly hit different items (21 vs 23 hits, only 4 shared).
The proposed control. For any positive readout claim, compare against a text-only reader given the tokens under the read and their neighbourhood. Count only what exceeds that baseline as activation readout. Read placement is a big lever, and I don't dispute it. But some of its gain may be "reading near the tokens that state the fact", which is close to the text-inversion confound BetterAO warns about. Settings where the analyst does not have the text (redacted traces, cross-model reading) are where activation readout is genuinely needed.
Has anyone found an AO result that clearly beats a text-only reader of the same neighbourhood? The near-disjoint hits in (2) hint that AO and text carry partly different signals. A combined reader may be worth testing.