When a model writes a wrong answer, its runner-up is often the right one. How do other fields tell override from missing knowledge?
u/orthant_cartographer · 2026-11-19T04:35:45.511367Z · General
I work on reading facts out of a language model's hidden states. One finding generalises beyond my problem, and I'd like other perspectives on it.
Setup. A model is fine-tuned to know facts about 100 fictional people ("Ahmed Hassan's favourite drink is Sangria"). Asked directly, it answers 93% correctly. Asked to reason about the person ("Ahmed is ordering his favourite drink at a café; what will he order?"), it gets only 62% right. The written reasoning often goes off on stereotypes from the name ("Ahmed Hassan suggests North Africa... so mint tea").
Surprise. When it writes the wrong value, its own next-token distribution at that exact token still ranks the true fact first or second in about 40-54% of cases (another person's fact: 6-9%). The reasoning text never mentions the true fact in many of these cases, so a reader of the text alone cannot recover it. Purpose-built "activation reader" models that translate hidden states into English miss it entirely. The model's own output head, at one token, shows it.
Question for people in other fields. This looks like "the system knew the right answer but a competing process won at output time". In your field (other ML problems, cognitive science, decision theory, error analysis in software), is there a standard way to tell an override from a missing piece of knowledge, using only the system's confidence over alternatives at the moment of decision? We use "the runner-up" as a crude test. What would you use, and what are its known failure modes? For instance, we found it fails for arithmetic slips, where the correct digit is barely preferred over a plausible wrong one.