Faithful activation readout: useful detail without fabrication
AI · Mechanistic interpretability
Faithful activation readout: specific detail without fabrication
1. Motivation and research goal
When a language model gives a wrong answer, we want to understand what went wrong: did it represent the relevant information, confuse two facts, or miss an important distinction? Its answer alone may leave these questions unresolved. Activations—the numerical representations inside the model as it processes text—offer another source of evidence. Activation Oracles (AO) and natural-language autoencoders turn these representations into descriptions that people can read.
These tools could help researchers investigate model errors, but their descriptions can contain errors of their own. A reader may invent a number or confuse which person an attribute belongs to. Our goal is to reduce these hallucinations while preserving useful information about the model. We begin with AO on Qwen3-8B.
2. What the baselines reveal
We benchmark three AO variants for Qwen3-8B: the original reader, a variant that reads activations from three layers, and one trained via GRPO (step 500). We tested these on the AObench hallucination task, which contains 197 trajectories from 29 source problems, with five prompts per trajectory, and PersonaQA, which measures the recovery of six attributes for 100 personas from one model with a shuffled-persona adapter.
| Measurement | Original AO | Three-layer AO | GRPO step 500 | |---|---:|---:|---:| | Judge-labelled detail error or off-domain | 54.52% | 80.71% | 42.03% | | Judge-labelled vague | 38.68% | 9.85% | 49.44% | | Judge-labelled correct | 6.80% | 9.44% | 8.53% | | PersonaQA official substring accuracy | 9.00% | 16.50% | 11.33% |
The GRPO-trained reader has the lowest labelled error rate but the highest rate of vague responses. The three-layer reader recovers more persona attributes but generates more detail errors. The hallucination judge sees the source question and the reader’s answer. Its labels assess whether the description fits the question. The upstream summary score counts every response except those labelled obviously wrong, including vague answers and plausible-but-wrong details. We therefore report all four categories and add checks of individual claims against evidence from the model being examined.
3. Research opportunities
We explore two routes. Route A keeps pretrained readers fixed. First, generate several descriptions of the same activations and measure how often at least one contains correct, useful details. Then, we compare whether activation-dependent selection—using the activations to score and select the best candidate—improves upon random choice or likelihood ranking. Likelihood ranking involves selecting the answer the reader assigns the highest probability. Compare these selection methods using the same candidate answers and computational budget.
Route B tests lightweight reader updates. Pairs of descriptions that differ in one factual detail could teach the reader to distinguish a correct description from a similar but wrong one. We compare this approach against standard fine-tuning on the same data under similar compute resources. Candidate selection results guide which route to follow. We welcome methods and findings that advance the goal of faithful readout.
4. What would count as progress
Progress means providing more correct, useful details at a given error rate, or making fewer errors while recovering a similar amount of useful information. Define how to identify and check individual claims on development data. Record which claims the available evidence supports, contradicts, or leaves unresolved. Report these results alongside the benchmark labels and both official and exact-match PersonaQA accuracy.
Compare against a reader given only the text and a reader given activations from a different example. These comparisons test how much useful information comes from the corresponding model activations rather than from the question alone. We report separately on methods using external reasoning traces at inference time; traces provide evidence of the model’s expressed reasoning, whereas assessments of internal computation require additional, independent evidence.
The supplied records have already been examined. Use them for development and collect independent, previously unused cases for final evaluation. We group trajectories by source problem and personas by identity, then verify that there is no overlap between training, development, and test sets. We repeat experiments across multiple random seeds and test on a second suitable task or model. Finally, we estimate uncertainty at the problem or persona level and audit judge disagreements.
5. Starting resources
Checkpoints and the environment are staged on the host. The supplement includes evaluation scripts, outputs, audit cases, manifests, and reproduction instructions. Begin with the README.
Each baseline inference run used one RTX 5090 with about 16.3 GiB peak allocated memory. The six runs total 1.19 GPU hours, excluding loading, downloads, and judging. Report training, extraction, generation, selection, and evaluation costs, including code, configurations, and data splits to enable reproduction.
References and baseline sources
- Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers · original implementation.
- Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations.
- Building Better Activation Oracles: AObench and improved readers.
- Evaluated checkpoints: original AO, three-layer AO, GRPO step 500.
- Code revision used for these runs. Table values are our local measurements; configurations and outputs are in the task supplement.
Task materials
Download supplement (.zip)- Type
- open-ended
- Solve status
- open
- Access
- Public