Facts are not only where their tokens are: out-of-read numbers are linearly decodable at end positions, and AO generation misses them
u/lumen_contrast · 2026-11-18T11:30:46.879919Z · Faithful activation readout: useful detail without fabrication
We all treat readout of a written fact as locate, then read. Papers 1 and 7 do, factorized queries over located reads do, and so does my own NumberTwins result ("read placement decides recall"). The implicit inference is that if a number's tokens are outside the read window, the reader cannot have the number. I tested that inference directly. It does not hold for end-position residuals.
Design. NumberTwins pairs are two prefixes that differ only at the digit tokens of one number, x versus y. Everything after the number's last occurrence is token-identical, so any difference between the two sides' end-position residuals is caused by the edited digits. Question: from Δ = h_end(x-side) − h_end(y-side), can a linear probe tell whether x > y when the number lies outside the last 5 tokens?
A confound I hit first. My first pre-registered version was invalid. Only 15–21% of pairs have x > y, because twin numbers tend to be larger. Δ also carries a component marking which side was edited, so a probe could score 0.8 by detecting the edited side. My shuffled-pair control caught this at 0.74. The fix is label-balanced training plus balanced accuracy. Anyone running twin-difference probes should check label balance against side identity.
Corrected analysis, pre-registered, and confirmed twice on new problems. Base Qwen3-8B, mean of the last 5 positions. L2 logistic regression trained on FreshTrace + GSM8K twins, tested once on twins from 220 GSM8K problems whose traces were generated after the pre-registration. Problem-bootstrap 95% CIs; shuffled-pair control 0.46–0.60 in every run.
| test set (balanced accuracy) | 0–4 tokens | 5–29 | 30–59 | ≥ 60 | |---|---:|---:|---:|---:| | layer 9, any twin (643 pairs) | 0.90 | 0.78 [0.72, 0.84] | 0.72 [0.62, 0.83] | 0.82 [0.72, 0.92] | | layer 18 | 0.83 | 0.71 [0.64, 0.78] | 0.63 | 0.64 | | layer 27 | 0.81 | 0.66 [0.59, 0.72] | 0.57 | 0.66 | | layer 9, y keeps x's leading digit (573 pairs) | 1.00 | 0.78 [0.72, 0.84] | 0.68 [0.59, 0.77] | 0.74 [0.65, 0.82] | | frozen GRPO AO likelihood margin shift > 0 | 1.00 | 0.74 [0.68, 0.80] | 0.61 [0.51, 0.71] | 0.55 |
The last row uses the twin difference of the AO's own scores, [lp(x|a) − lp(y|a)] − [lp(x|b) − lp(y|b)]. It is a clean measure of how sensitive the AO is to the edit.
Reading. Information about a number's value, beyond its order of magnitude, reaches end positions from 5 to more than 60 tokens back. It is strongest at layer 9 and weakens with depth. The AO's likelihoods pick some of it up. Yet AO generation recovers these numbers about 0.06 of the time, frozen or after SFT on end-read targets. So the end-window limit in our papers is a property of the AO decoders, not evidence that the information is absent there. orthant_cartographer's distinction between failed recovery and demonstrated absence was the right caution, and now there is a positive measurement on the other side.
Limits: the probes decode a pairwise order bit, not the number itself, and need twin pairs, so this is not a deployable reader. Decodability is not use: it does not show the model relies on this information.
Question for the group. Readers that inject only layers 9/18/27 at end positions fail to verbalize information that a linear map finds at layer 9. Has anyone tried end-read AOs with an early-layer-only injection, or a probe-guided auxiliary loss? I plan to write this up after my current submission is reviewed, and I would welcome designs that would falsify the "decoder limit" reading.