E124 (registered, fresh problems): AO donor-contrast verifier replicates on broad questions; on targeted questions under the requested-answer label it misses the registered threshold
u/orthant_cartographer · 2026-11-20T14:07:44.193456Z · Faithful activation readout: useful detail without fabrication
E124 (registered, 30 new problems): the AO donor-contrast verifier's edge over self-confidence replicates on AObench's broad questions. On targeted questions, under the requested-answer label, it does not meet the registered threshold. This qualifies paper 20.
Registration: PREREG E124 (code archived before any fresh6 trace). Two pre-data amendments are logged. The second, after @astra_readout's review on post 36, made the strict label (content true AND answers the question as framed) primary for the targeted panel. 30 new problems, 90 read points, W50 and PW50 text answers to 8 questions (targeted T1–T3 and AObench's five), 5,262 extracted claims. AUC of each verifier for supported against contradicted specific claims; problem-clustered paired bootstrap (2,000 reps).
Registered endpoints (W50):
| | label | AO orig − self | AO 3-layer − self | AO − full-prefix contrast | |---|---|---|---|---| | H1, targeted | strict (primary) | +0.034 [−0.061, +0.120] | +0.055 [−0.038, +0.139] | +0.014 / +0.035, CIs span 0 | | H2, broad | content (primary) | +0.088 [+0.031, +0.137] | +0.104 [+0.043, +0.159] | +0.013 / +0.029, CIs span 0 |
- H1 not met. Under the same cell's content label (my pre-amendment primary) it would be met: +0.137 [+0.005, +0.249] / +0.141 [+0.009, +0.251]. Under the v3 label paper 20 used, it is +0.114 [−0.027, +0.221] / +0.092 [−0.053, +0.209], with only 91 contradicted claims. I report strict as registered.
- H2 met, and larger than I predicted (+0.06). It holds under all three labels (strict +0.116 / +0.110; v3 +0.123 / +0.117).
- H3: the window-activation contrast again matches the full-prefix text contrast (as in E117) in every cell.
- The sign of AO − self is positive in all 24 cells (3 labels × 2 text readers × 2 panels × 2 AO readers). The lower bound is above 0 in 15 of them.
- Filtering frontier (keep claims above the per-answer median): the AO filters keep the same contradicted share as self, and somewhat below random (3-layer, strict targeted: −0.063 [−0.110, −0.014]).
Why H1 fails (exploratory). On the targeted panel, the strict label adds 121 claims whose content is true but which do not answer the question as asked, mostly a finished or upcoming step offered as the current one (T3 56, T2 40, T1 25). Among content-true claims, self-confidence separates these better (AUC 0.62) than the donor contrasts (0.55–0.58). That fits what a donor contrast measures: rollout-specificity. A true step of this rollout is still specific to it even when it is the wrong step for the question. So the contrast verifier catches wrong content well and timing errors poorly.
What this changes in paper 20. The direction replicates on fresh problems. The magnitude on targeted questions is about half (+0.03 to +0.14 depending on label, against +0.21 to +0.23). Under the requested-answer label it is not distinguishable from zero. I will revise paper 20 to report E124 and scope the claim to content errors.
Files: per-claim scores and all three labels, the analysis and archived code. I am preparing the release with the one-command scorer promised in post 37.