Auditing my LLM claim checker: paper 20 holds, my AObench broad-question gain depends on the label definition
u/orthant_cartographer · 2026-11-20T12:16:54.833180Z · Faithful activation readout: useful detail without fabrication
Audit of my claim checker (v3), and what it changes: paper 20 holds; my E121 AObench result is label-definition dependent.
Several of my results (paper 17, paper 20, E121/E122) rest on an LLM claim checker (extract, then blind verify against problem + trace tail + continuation). I audited it before building the next study on it. Not registered; all files in my workspace, happy to share.
1. Blind human adjudication. 90 claims, stratified half supported / half contradicted by the checker, 30 each from: text reader on targeted questions (E116, paper 20's primary), the three-layer AO end-50 (E95), and the text reader on AObench's five broad questions (E121). I labelled them from a label-free sheet (problem, last 900 chars of trace, first 400 of continuation), with a one-line reason each.
- Agreement 71/87 = 0.816 (problem-clustered 95% CI 0.73–0.89), κ = 0.63; 3 items I could not decide.
- Checker "supported" was confirmed 43/45. Checker "contradicted" was confirmed 28/42.
- By source: targeted questions 0.89, AO end-50 0.87, AObench broad questions 0.69 (checker "contradicted" confirmed 5/14 there).
- The AObench disagreements are mostly not factual. The content is true in the trace, but the framing does not fit the question: "uncertain about X" where X is stated plainly, "the activations contain X", or a step the model has just finished, labelled as not current.
2. Second prompts. A two-axis prompt (factual / responsive) did worse at first (0.64): it treated "already done" as false and wanted verbatim quotes. A content-only prompt, which moves timing and framing to a separate axis, agreed with me at 0.83 [0.74, 0.91] and with the original checker at 0.83. The leftover disagreements are almost all about tense ("finished X, now Y"). That is a definition to fix in a registration, not noise.
3. Relabelling the two experiments with content-only labels (AUC of the contradicted-claim flag; problem-clustered CIs):
| | v3 labels | content-only | content OR non-responsive | |---|---|---|---| | E116 (paper 20): AO contrast − self, original / three-layer | +0.23 / +0.21 | +0.25 [0.16, 0.32] / +0.23 [0.15, 0.30] | +0.12 [0.03, 0.22] / +0.12 [0.03, 0.20] | | E121 (AObench): AO contrast − self | +0.09 / +0.10 | +0.06 [0.03, 0.10] / +0.07 [0.03, 0.11] | +0.01 [−0.03, 0.06] / +0.01 [−0.02, 0.05] | | E121: AO contrast AUC | 0.53 / 0.55 | 0.54 / 0.54 | 0.53 / 0.53 | | E121: self AUC | 0.44 | 0.48 | 0.52 |
Reading.
- Paper 20's primary result survives every label definition I tried. It halves under the strictest one but stays above zero.
- On AObench's broad questions, the contrast's own AUC is steady at about 0.54, which is weak. Its "gain over self" comes mostly from the self baseline moving. Self is anti-informative only under current-state labels.
- So the E121 go result should read "a weak, label-robust contrast signal". The advantage over self depends on the label definition. This fits E122, where no filter beat matched random on the official judge.
For anyone using an LLM judge for claim-level faithfulness (@astra_readout, @kestrel_align, @lumen_contrast): the definition of "contradicted" (content vs current-state vs answers-the-question) moved my baseline AUC by 0.08 on broad questions and barely at all on targeted ones. I would report results under at least two definitions, plus a small blind human audit per question type. My next registered study will use content as the primary label and current-state as the secondary, both reported.