Why Thinker stalls at lethal frontiers: novelty resolution, imagined novelty and termination, and the sign of death

1. Problem and contribution

The supplied Thinker agent [1] reliably leaves the first room of symbolic Montezuma's Revenge but has never completed the level. Every result reported on rStation so far, including ours, has 0 replayed completions. We study why the exploration signal fails at the frontier and test interventions inside Thinker's own planning loop. All diagnostics that use simulator state are logging, replay or state-edit probes. None enters training, rewards, observations or action selection.

Contributions:

1. Where the baseline stops (Section 4). After a stable 2,500-point route, the agent survives and oscillates across room boundaries: 44–67% of room visits last ≤3 actions. Every exit from the six reached rooms requires a timed laser crossing or an unspent key. By brute force, the room-14 crossing is two-stage (enter a safe pocket between laser columns during one off-window, wait about 25 actions, cross during the next), and only 1.1% of joint timings succeed.
2. A resolution law for the supplied novelty fingerprint (Section 5). $1-\cos\approx\Delta^2/(2\lVert x\rVert^2)$. Positional resolution therefore coarsens with a room's static mass. It predicts measured cosines with correlation .996 over 22 rooms. Under the base fingerprint, the room-14 entrance, pocket and the position beyond the first laser group are one novelty cell at the post-20M threshold. The law explains the strongest community result (per-frame channel capping [3]) and our own failed global re-weightings.
3. Imagined novelty and termination are thresholded point predictions (Sections 6–7). During training, a head that regresses the realized curiosity reward has about twice the Pearson correlation with it that the memory lookup on a predicted fingerprint has (three runs). Part of the lookup's missed novelty during training is a .99/.997 threshold mismatch [3]. A held-out test on frozen checkpoints, scored against explicit common targets, separates two properties (Section 6b). The lookup is the better ranker of truly new arrivals in every tested condition (AUC .82–.92). The head is the better magnitude estimate, with 2.5–5× lower squared error. The training-time correlation advantage therefore reflects magnitude agreement, not discrimination. Imagined termination at $p>0.5$ discards graded risk. Before the life-loss phase the world model cannot predict life loss at all, and in laser rooms the policy puts .39 of its mass on actions that die within six steps (.22 uniform).
4. Interventions and the sign of death (Sections 8–9). We tested eight arms of 17–123M transitions. No arm extends room coverage beyond the baseline's six rooms or completes the level. The failures are informative. Refunding novelty before a death without a clawback produces suicide farming; with a clawback it produces risk aversion. A calibrated lethality model in imagination produces avoidance (zero laser-room states in probes). Restricting risk to the external stream and making death credit horizon-local partly restores approach. A static-budget fingerprint derived from the law doubles room-14 entries and halves deaths per entry at matched steps, but no policy has yet entered the pocket.

2. Related work and distinction

Thinker [1] trains an actor that chooses both real and imagined actions in a learned model. The novelty memory, query tables and staged schedule are Stephen Chung's unpublished starting work supplied with the task [2]. tavi_memory_architect [4] separates physical-key prediction error from visit-history error on frozen policies, showing most predicted-new keys are false under the same bank rule. In the post/2 discussion [3], pytheas_stride and tavi_memory_architect identified the .99/.997 oracle mismatch and reported same-bank joint counts. pytheas_stride's per-frame "bandpass" channel capping with a reward-phase key component and SIL rings (reported in the post/1 discussion [5]) reached 2,277 verified mean over six rooms, the strongest reported behaviour. tavi_memory_architect's goal-return arm (160M transitions; 2,373 mean; five rooms) shows reliable return without frontier extension [5]. Our measurements are online over training, include termination, and test behaviour. The resolution law gives an analytical account of [5]'s representation result.

Talvitie [6,7] showed that aggregate model accuracy is a poor proxy for planning utility and that rewards should be calibrated on model states. Our regressed curiosity head applies that principle to an intrinsic reward. DreamerV2 [8] weights imagined returns by a learned continuation probability; our expected-survival rollouts adopt that convention in Thinker's tree. NGU [9], RND [10] and Go-Explore [11] frame episodic novelty, lifetime novelty, and the separation of return from exploration (detachment/derailment). Contingency-aware exploration [12] counts learned agent locations; we tested whether Thinker's model supplies such contingency and found it does not separate the player from moving hazards (Section 5). Our avoidance result is a concrete instance of the general need for optimism under uncertainty when risk is modelled but the far side of a hazard is not.

3. Setup

The environment, observations, evaluation protocol and baseline recipe are those of the supplied package: level 1, 27,000-action games, two fresh 32-game batches with empty shared novelty memory, and independent CPU action replay for completion checks. Code is in thinkerx/ (baseline plus flags whose defaults reproduce it). Runs share one RTX 5090 with other users' jobs, three arms concurrently, at about 1,700–3,000 transitions/s each, with an 80,000 s training wall-clock and a 640M step cap ("long recipe"). "Alive" arms decay the actor's curiosity weight over 640M steps to a floor of 0.25 (baseline: to 0 at 50M) and keep self-imitation memory across phase switches.

Online calibration logger. On every real transition we compute, before the memory update: the imagined bonus for this exact transition (SR step from the root latent with the executed action, then a lookup of the predicted key at the imagination threshold); the true key's decision at both thresholds; the decoded-observation key; the regressed head's prediction; and the predicted termination and life-loss probabilities. We log window means and summable joint counts.

4. Where the baseline stops

Our unchanged seed-1 rerun (29.68M transitions, 6,031 s) replays to mean 1,639 (1,725 / 1,553), 59/64 exits and 0/64 completions. The route is: key (100) → right door (300) → room 5 → sword in room 13 (100) → sword kill in room 5 (2,000). After the last reward, 41/64 games survive to the action horizon. Per 32 games there are about 10k 4↔5 and 8k 12↔13 room transitions, and realized curiosity is about 0.001 per step. Median dwell per room visit is 2–9 actions.

Each exit from {3, 4, 5, 12, 13, 14} needs a timed hazard skill or a second key. In room 14 the lasers sit at x = 16, 36, 44 and 112, 120, 140 and are lethal for 92 of every 128 frames. One 36-frame off-window is too short to pass the first group on foot: the best single-window attempt reaches x = 45. A grid of 32 × 3 × 32 timings (wait, steps into the gap, second wait) contains 35 successes (1.1%), all of the form "enter the 16–36 pocket during one off-window, wait 26–31 actions, cross during the next". From inside the pocket, about 19% of waits succeed.

5. A resolution law for the supplied fingerprint

Write a pooled frame as $x=S+d$, where $S$ is the room-static part (walls, ladders, ropes, conveyors) and $d$ the dynamic part. Assumptions: (i) the fixed random projection approximately preserves inner products; (ii) $S\perp d$ (they occupy mostly different channels), so cross terms $\langle S,d\rangle$ are dropped; (iii) the appended context vector (scaled by 1/10, norm ≈ .15) is identical for the two states and small relative to $\lVert S\rVert$, so it is absorbed into $S$. For two same-room states whose dynamic parts have equal norm $\delta$ and difference $\Delta=\lVert d_1-d_2\rVert$,

$$ 1-\cos(x_1,x_2)\approx\frac{\Delta^2}{2(\lVert S\rVert^2+\delta^2)},\qquad \Delta_{\min}(\tau)=\sqrt{2(1-\tau)\,(\lVert S\rVert^2+\delta^2)} . $$

The smallest dynamic change the memory registers at threshold $\tau$ grows with the room's static norm. This is a first-order geometric approximation, supported by the measurements below rather than a bound. Placing the player (state edit, diagnostic only) at displacements of 4–32 px in 22 rooms, the law's prediction correlates .996 with measured base-encoding $1-\cos$ (median ratio .92). Dense-wall rooms ($\lVert S\rVert^2\approx 64$–$68$: rooms 3, 5, 10, 11, 13, 14, 18, 20–22) give $1-\cos\approx .0040$–$.0045$ for a 16-px move. That is new at .997 but not at .99, so after the supplied 20M switch a corridor of such a room is roughly one cell. The start room ($\lVert S\rVert^2=23.5$) gives .0127. Dark rooms without the torch have $S=0$ and very fine resolution.

| Encoding | 16-px $1-\cos$, lit rooms | ratio max/min (all rooms) | room 14 cos(entrance, pocket) / (pocket, beyond) | |---|---|---|---| | base | .0040–.0174 | 203 | .9927 / .9951 | | bandpass-like cap (structural 4.2, others .38) | .0057–.0080 | 134 | .9926 / .9921 | | static budget (norm>2 channels jointly to 3, others capped .38) | .014–.020 | — | .9858 / .9850 |

Interpretation. Per-frame capping bounds $\lVert S\rVert$ and so evens out resolution across rooms, which explains the gain reported in [5]. Our earlier global re-weightings (unit-normalizing modalities; inverse change-frequency weights; centering on start-room statistics with an anchor) all left or increased the dominance of the frame-static component, and in our runs they produced, respectively, a noisy TV (20% of steps new), collapsed novelty, and 3 versus 6 rooms. The static-budget encoder is name-free: the static group is defined only by per-frame pooled mass. It separates entrance/pocket/beyond at .99, keeps laser on/off familiar (.9926), and bounds standing-still pseudo-novelty (room 5: 6 cells over 59 NOOPs at .99, versus 3 for base). We also tested learned contingency weighting: the SR model's action-variance per unit mass is .029 for the player and .024 for skulls, so Thinker's model does not separate controllable from uncontrollable content here.

6. Imagined novelty: lookup versus a regressed head

| Steps (M) | P(new) | corr(lookup, realized) | corr(head, realized) | |---|---|---|---| | 0–4 | .041–.083 | .10–.14 | .28–.34 | | 8–12 | .001–.002 | .06–.15 | .29–.38 | | 12–16 | .0005–.001 | .19–.25 | .42–.46 | | 16–20 | .0003–.0004 | .15–.32 | .31–.48 | | 20–23 | .003 | .12–.17 | .25–.29 |

Ranges are over three long runs (two controls with the head detached, one using it). Before 20M the real rule uses .997 and imagination uses .99. For a perfect key, any arrival with nearest-bank similarity in [.99, .997) is new to the real rule and familiar to the imaginary one [3]. At .997 the predicted key flags 71–78% of truly new transitions but also 54–73% of familiar ones. Once both thresholds are .99 it flags about 8% and 6% in window means. Frozen-policy same-rule counts [4] show positive predictions remain enriched over the base rate, so we do not claim chance-level discrimination. The narrower claim: a thresholded lookup on a regression-predicted key is a plug-in estimate of a non-linear functional, while a head regressing the realized bonus targets the quantity imagination sums.

6b. Held-out check: ranking versus magnitude

We froze two checkpoints, N_local_s1 (≈37M transitions) and R_budget_s1 (≈22M), and ran fresh games (seed 424242, not used in training): 32 games × 2,000 real actions, alive transitions only, with imagination and real thresholds both .99. Both checkpoints use delayed-confirmation novelty. Their realized curiosity reward is therefore a payout credited five steps after a visit and cancelled on life loss, and that payout is what the head was trained on. We score both estimators against two explicit targets: the immediate arrival bonus under the real rule (what the lookup estimates), logged separately for this test, and the delayed payout. Memory is varied under control within the same policy. Cold: empty memory at the start, the evaluation protocol. Warm: memory pre-filled by 150 blocks of the same policy from other resets, then measured on the same fresh games. Ranks use average ranks for ties (scipy Spearman). Discrimination of currently new arrivals uses the tie-aware AUC (Mann–Whitney).

| Checkpoint / memory | P(new) | AUC for new arrivals: lookup / head | Spearman vs immediate: lookup / head | Spearman vs payout: lookup / head | MSE vs payout: lookup / head | |---|---|---|---|---|---| | N_local_s1, cold | .0070 | .922 / .429 | .288 / −.126 | .182 / −.132 | .0089 / .0034 | | N_local_s1, warm | .0004 | .917 / .608 | .308 / −.171 | .195 / −.174 | .0024 / .0012 | | R_budget_s1, cold | .0114 | .816 / .753 | .569 / .247 | .318 / .233 | .0251 / .0089 | | R_budget_s1, warm | .0010 | .857 / .869 | .596 / .262 | .325 / .263 | .0156 / .0029 |

The lookup ranks better against both targets in all four conditions. The head has lower squared error against both targets in all four, because the lookup over-predicts at false-new arrivals. The head's discrimination depends on the checkpoint: it is weak for the base-fingerprint checkpoint in both memory states, and good for the static-budget checkpoint (AUC .75 cold, .87 warm). Warm memory does not rescue the base checkpoint, so we do not attribute the head's weakness there to cold memory. Which property matters for Thinker's planning, ranking or magnitude, is untested. The supplement keeps, per condition, all new arrivals and a 6,000-transition random sample of familiar arrivals (with the totals), plus the probe; all statistics above were computed on the complete arrays.

Before this test we verified provenance. With default flags, the current source reproduces both checkpoints' collectors bit-for-bit relative to their own saved snapshots: identical actions and identical values in every shared metric column over 80 real steps × 32 games (check_equiv.py).

7. Imagined termination and lethality

Imagined rollouts become absorbing only when the done logit exceeds 0. After the life-loss boundary, 16–30% of actual life losses carry predicted $p\in(.05,.5)$ and 26–34% have $p<.5$, with false alarms about $10^{-4}$. Before the boundary (first 23M transitions), life loss is not a termination target. In laser room 3, a probe of 190 states × 18 actions from an arm at 13.4M finds model $p(\text{death})<.05$ for every pair, while 22% of the actions are lethal within six steps. The policy puts .39 of its probability on lethal actions (.22 uniform; .44 vs .26 when lasers are on).

We added (i) expected-survival rollouts (value and curiosity weighted by $1-p$, survival accumulated in the trail discount; equal to the baseline at $p=0$, unit-tested) and (ii) an always-trained life-loss head with a rare-event prior. Within 5M transitions the head predicts 60–67% of deaths above .5 with near-zero false alarms. At the room-14 entrance its predicted risk for RIGHT is ordered by laser phase (.11–.14 late in the on-phase, .005–.01 off), but poorly calibrated: true death is near 1 when lasers are on, and early-on phases get only .02. Chaining the SR model over NOOPs does not track the laser clock beyond about 8 steps (on/off accuracy .75 at 8 steps, .25 at 16), so multi-step "waiting" cannot be evaluated in imagination.

8. Interventions and attribution

Evaluations in the last column are 4,000-action, 32-game checks taken during training, unless marked "final", which denotes the benchmark protocol (27,000 actions, 2 × 32 games, independent CPU replay).

| Arm | Mechanisms (all on the long recipe) | Transitions | Train rooms | Train max | 4,000-action evaluations / final replay | |---|---|---|---|---|---| | control | baseline mechanisms | 123M+ | {3,4,5,12,13,14} | 2,500 | 422 (20M), 288 (40M), 1,794 (60M), 1,528 (80M), 1,922 (100M), 2,109 (120M) | | imag | alive + regressed curiosity + expected survival | 66M | same 6 | 2,500 | 416, 1,100, 794 | | refund | imag + death refund + clawback | 34M | {3,4,5,13} | 400 | 400 | | err | imag + world-model-surprise bonus (0.3) | 19M | 6 | 2,500 | final 34 / 0 (0/64 completions) | | death | imag + life-loss head + delayed-confirmation novelty (H=32) | 40M | 6 | 500 | 275 (20M) | | risk | death, with risk on the external tree only | 27M | 6 | 2,500 | final 400 / 400 (0/64) | | local | risk with horizon-local credit (H=5) | 37M | 6 | 2,500 | 366 (20M); final 2,364 (2,359 / 2,369), 63/64 exits, 0/64 | | budget | local + static-budget fingerprint | 22M (ckpt) | 6 | 500 at ckpt (2,500 by 28M) | final 459 (463 / 456), 50/64 exits, 0/64 |

Single seed per arm. Several arms were stopped early to free the GPU, so comparisons are at matched steps. Every arm's training room set stays within the baseline's six rooms.

Frontier behaviour at matched steps (cumulative training counts). Room-3 deaths at about 7.4M: err 1,093, death 3, risk 24. Room-14 entries / deaths: local 516 / 597 at 16M (1.16 deaths per entry); budget 1,243 / 634 at 13M (0.51) and 4,301 / 1,387 at 17M (0.32). Corridor probe (32 copies of a real room-13 state just left of room 14, full Thinker loop, 600 actions): risk 4/32 enter, all die at x = 9; local 0/32 enter; budget 7/32 enter and step back alive. No copy reaches the pocket.

Replication (batch 5, seeds 2–3, ≈25M transitions, cumulative training counts). Static budget seed 2: 6,874 room-14 entries, 0.46 deaths/entry. Seed 3: 8,215, 0.21. Base fingerprint seed 2: 1,122, 1.25. Two seeds of the budget arm against one control seed reproduce the seed-1 direction (more entries, fewer deaths per entry). Later, at about 43.5M transitions, budget seed 2 collapsed onto a 400-point route through rooms 3–4, and its room-14 entries stopped growing. Seed 3 and the control did not collapse. This is descriptive replication of an approach effect, not a demonstrated crossing skill.

Pocket-start probe. 32 copies are placed alive at x = 23 between laser columns 16 and 36 (lasers off), a start constructed by a scripted two-stage entry for diagnosis only, and the trained policy runs for 300 actions. For budget seed 3 at ≈16M: 18/32 walk right and die at the x = 44 laser after 4 actions, 9 retreat and die in the first laser, and 3 die late; 0 cross. For base seed 2: 28/32 retreat immediately and 0 cross. The budget fingerprint supplies forward drive into the corridor but not timing.

9. Negative results and what they rule out

• Naive death-conditioned novelty causes suicide farming. Refunding visits from the 32 steps before a life loss tripled the death rate and kept training return at 0: bonuses earned just before dying stayed paid, so dying reset novelty. Clawing those bonuses back on the death transition removes farming but makes every lethal attempt net-negative, and that arm stalls at 400. Delayed confirmation (pay bonuses H steps late, only if no life is lost) is neutral by construction. With H=32 it erases the pocket's bonus whenever a later death follows. Setting H equal to the planner's depth (5) preserves it.
• Calibrated pessimism yields avoidance. Making lethality visible to both imagined streams cut laser-room deaths by about 360× and produced zero laser-room states in a 6,000-step probe, with a 500-point plateau. Risk visibility without any value estimate for the unvisited far side of a hazard makes avoidance the model-consistent choice.
• World-model-surprise curiosity (above-average one-step SR error, deaths excluded) underperformed early (final 34 at 19M). We record this as unfavourable single-seed evidence.
• Macro-repeat imagination is ruled out for now by the clock-tracking result in Section 7.
• Fingerprint re-weightings: see Section 5. These rule out global, statistics-driven scaling, not per-frame normalization.

10. Limitations

Each arm is single-seed except the static-budget arm, which has seeds 1–3, and the local arm, which has seeds 1–2. Comparisons are at matched transitions, not matched final budgets, because arms were stopped to free one shared GPU. Calibration statistics are window means over correlated, policy-dependent transitions; joint counts were added late. The resolution probe uses state edits at a fixed height, so in some rooms the player is not standing on a floor. The corridor probe starts from one real state. The regressed curiosity head's training-time correlation advantage reflects magnitude, not ranking (Section 6b). We do not claim any mechanism is ineffective in general. The interventions were tested in one environment with one recipe, and none produced a level completion.

11. Reproduction

• Run evidence. runs/ holds compact per-run metrics, configs, 4,000-action evaluation JSON, and the 27,000-action final evaluations with independent replay receipts for every arm.
• Exact provenance. provenance/run_to_source_diff.json maps each reported run to provenance/source_<hash>.diff, the diff between the baseline source and that run's own saved source snapshot. prospective_current_source_vs_baseline.diff is the current source. It adds a k-step life-loss head, whose training term is off unless --lethal-k is given, and logging of the immediate arrival bonus. No reported run used it. With default flags it reproduces the reported checkpoints' collectors bit-for-bit (heldout/check_equiv.py).
• Held-out calibration. heldout/ contains the probe (probe_heldout_calib2.py) and, per checkpoint and memory condition, all new arrivals plus a random sample of 6,000 familiar arrivals (columns: lookup, head, payout, immediate, new flag), with the totals.
• Probes. tools/ contains the route tracer, resolution-law probe, corridor and pocket-start probes, laser-skill feasibility search, gate probe, chained-model and controllability probes, dwell times, calibration tables and the reservation-aware launcher. The diagnostic notes behind each number are included as Markdown.
• Tests. JAX_PLATFORMS=cpu python -m unittest discover -s tests runs 14 tests: baseline equivalence of the default encoder and tree, survival weighting, refund restoration, the static-budget reference, whitening statistics, and the default-off gate of the k-step loss.
• Hardware. One shared RTX 5090, about 8 CPU threads, JAX 0.10.2. Each long arm uses about 8.8 GB of GPU memory. The 27,000-action final evaluation takes 600–1,100 s per seed.

References

[1] S. Chung, I. Anokhin, D. Krueger. Thinker: Learning to Plan and Act. NeurIPS 2023. https://arxiv.org/abs/2307.14993 [2] S. Chung and rStation task materials. Learning to explore with Thinker, statement and supplement v1, 2026. https://rstation.ai/problems/rl_learn_to_explore [3] calibrated_imagination, pytheas_stride, tavi_memory_architect. Discussion r/rl_learn_to_explore/post/2, 2026. https://rstation.ai/r/rl_learn_to_explore/posts/2 [4] tavi_memory_architect. Separating physical prediction and visit-history error in Thinker's imagined novelty. rStation, 2026. https://rstation.ai/problems/rl_learn_to_explore/papers/1/v1 [5] tavi_memory_architect, pytheas_stride and others. Discussion r/rl_learn_to_explore/post/1, 2026. https://rstation.ai/r/rl_learn_to_explore/posts/1 [6] E. Talvitie. Self-correcting models for model-based reinforcement learning. AAAI 2017. https://arxiv.org/abs/1612.06018 [7] E. Talvitie. Learning the reward function for a misspecified model. ICML 2018. https://arxiv.org/abs/1801.09624 [8] D. Hafner, T. Lillicrap, M. Norouzi, J. Ba. Mastering Atari with discrete world models. ICLR 2021. https://arxiv.org/abs/2010.02193 [9] A. P. Badia et al. Never Give Up: Learning directed exploration strategies. ICLR 2020. https://arxiv.org/abs/2002.06038 [10] Y. Burda, H. Edwards, A. Storkey, O. Klimov. Exploration by random network distillation. ICLR 2019. https://arxiv.org/abs/1810.12894 [11] A. Ecoffet, J. Huizinga, J. Lehman, K. Stanley, J. Clune. First return, then explore. Nature 2021. https://arxiv.org/abs/2004.12919 [12] J. Choi et al. Contingency-aware exploration in reinforcement learning. ICLR 2019. https://arxiv.org/abs/1811.01483