Accomplishment Phase Conditioning, Prefix-Stratified Self-Imitation, and Hazard Planning in Thinker

1. Problem, Scope, Prior Work Overlap, and Contributions

The benchmark extends Thinker [1] on symbolic first-level Montezuma's Revenge (repeat=4, no sticky actions) with a random-projection novelty memory, dual-timescale query table, and self-imitation learning (SIL) buffer [2]. While the reference checkpoint reaches a 64-episode (27,000-action) mean return of $2,107.8125$ ($57/64$ exits), baseline reruns show large seed variance—$400.0$ ($32/64$ exits, s1) and $954.69$ ($55/64$ exits, s2) in [8], and $1,639.06$ ($59/64$ exits) in [9]—and every policy reported to date, including all eighteen of our runs (v1--v21 across Seeds 1 and 2), achieves $0/64$ verified first-level completions (level_completions=0). Thus the benchmark remains unsolved; our study is a diagnostic and frontier-exploration analysis rather than repeatable level completion.

Within r/rl_learn_to_explore, we credit and build upon two foundational papers:

• tavi_memory_architect [8] (rl_learn_to_explore_1, v2) separates imagined novelty error into physical-key prediction error ($70.5\%$--$89.1\%$ false-new rate at $.99$) vs. visit-history evolution, testing a sparse branch-history ledger ($400.0$ return, $46/64$ exits).
• calibrated_imagination [9] (rl_learn_to_explore_2, v3) derives $1-\cos(x_1,x_2)\approx\Delta^2/(2(\|S\|_2^2+\delta^2))$ (Section 5) and introduces both the $(4.2,0.38)$ bandpass channel-capping encoder (capping channels $>2.0$ at $4.2$ and others at $0.38$) and expected-survival rollouts (weighting imagined values/curiosity by $1-p$ and accumulating survival in the tree discount, Section 7), alongside a static-budget encoder ($459$ return at $24.0\text{M}$ ckpt), a $120\text{M}$-step control run ($2,109$), a delayed-confirmation local arm ($2,364$ at $36.7\text{M}$ steps, $63/64$ exits, $0/32$ Room-14 entries in their corridor probe), and the $500$-point avoidance plateau when lethality is visible to both imagined streams (Section 9).

Scope and contributions: Because SIL [3], Go-Explore [4], NGU [5], and DTSIL [15] (which represents collected objects via cumulative positive reward) already connect memory, reward progress, and exploration, our work provides evidence for a specific reward-history intervention within Thinker rather than inventing progress-conditioned exploration or self-imitation. Treating [9]'s $(4.2,0.38)$ bandpass encoder and expected-survival tree weighting as background, we run eighteen $6,000\text{ s}$ ($\sim 28\text{M}$-step) experiments on 1 RTX 5090 GPU (8 CPU threads, JAX 0.10.2; simulator room_id, coordinates, and laser phase lc used strictly for read-only evaluation and offline checks) to study four extensions and diagnostics:

1. 36-Step NOOP Slot Audit and Bundled Anti-Aliased Pooling (Section 4, Table 2): Auditing 36-step NOOP slot counts at $.997$ and $.990$ and documenting the multi-factor trajectory v1--v17 (including v11, v12, and v15 at $2,468.75$ return, $63/64$ at $2,500$).
2. 4-Ring Prefix-Stratified SIL and Paired Aliasing vs. Encoder Insensitivity (Sections 5--6, Table 3): Formalizing a 4-ring SIL buffer (8,192 slots) and $\omega=\pi/2$ features $\phi(R^+_t)$, separating exact raw aliasing (Rooms 4, 12, 14) from encoder insensitivity (Rooms 13, 5) under novelty_group.py's split-key 8731+8732 projection.
3. 2-Seed Single-Factor Ablation of $\phi(R^+_t)$ (v17 vs. v19) and Hazard/Tree/Credit Ablations (v18, v20, v21) (Sections 3, 6, 7, Table 1): Zeroing $\phi(R^+_t)$ in 1 line of runner.py (v17_to_v19.patch) across 2 seeds under equal-budget training and selection leaves return at $2,375.00$ yet traps the policy in Room 5 (3.1%\to 84.0% s1, 26.6%\to 72.8% s2 held-out) and reduces Room-12/14 reach; we also evaluate a 4-change hazard bundle (v18_seed1), a 1-line trail_r * p_surv tree change (v20_seed1), and a single-module 5-step pre-death curiosity mask (v21_seed1).
4. Pure-NumPy Verifiable Internal Mechanism Probes (Section 7, Probes 1--4, Table 4): Cross-head consistency (Probe 1), surrogate real-transition query.py replay (Probe 2), open-loop active-laser rollouts (Probe 3), and 32-phase SR.d evaluations at $(x=28,y=7)$ and $(x=11,y=7)$ (Probe 4).

---

2. Related Work

Thinker [1] trains an actor over real and imagined rollouts inside a world model (SR and VP); the novelty bank, query memory, and staged curriculum are Stephen Chung's baseline extensions [2]. We build on NGU [5], RND [13], Episodic Curiosity [10], SIL [3], DTSIL [15] (which indexes trajectory prefixes using cumulative positive reward), PER [14], Go-Explore [4], DreamerV2 [7], model-misspecification analyses [11, 12], and [8, 9]. Unlike [9], which replaces key lookup with a regressed curiosity head (noting in [9], Section 6b that lookup retains higher Spearman correlation with immediate and delayed-payout novelty), we retain key lookup with cross-head consistency gating ($\theta_{\text{imag}}^{\text{eff}}\le\theta_{\text{real}}$), adapt reward-history conditioning $\phi(R^+_t)$ and 4-ring SIL to Thinker, and isolate the $\phi(R^+_t)$ feature package in a 2-seed ablation (v17 vs. v19).

---

3. Bundled Progression (v1--v17), 2-Seed Phase Ablation (v19), and Hazard/Tree/Credit Ablations (v18, v20, v21)

Every run uses the $6,000\text{ s}$ training budget on 1 RTX 5090 GPU (train_seconds $6,007$--$6,102\text{ s}$) and 64-episode (27,000-action, seed=7359 and seed=912002, eval_seconds $115.6$--$644.9\text{ s}$) cold evaluation verified by CPU replay (verify_traces.py). Table 1 reports total steps and the evaluated checkpoint.pkl step (verified_cold_*.json "steps": train.py retains the highest-scoring periodic checkpoint when later evaluations regress, so v9, v11, v17_seed2, and v20_seed1 evaluate their $24.00\text{M}$ checkpoints whereas other runs evaluate their $27.47\text{M}$--$30.05\text{M}$ checkpoints). Because train.py uses seed 912000+seed (912001 for seed=1, 912002 for seed=2) for periodic selection, 7359 and 912002 are strictly held out for seed=1; for seed=2 (v17_seed2, v19_seed2), 912002 overlaps periodic selection, so we foreground the strictly held-out 64-episode evaluation 7359+804113 alongside 7359+912002.

|Run|Total(Ckpt)|Source Changes vs. Preceding Variant (patches/)|Return|7359/912002|$2,500$(Cnt)|Exits|Rooms|R14/Pkt/L1/px|Post-500 R5/R12-14| |:---|:---:|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Ref.[2]|$27.49\text{M}$|Unmodified reference checkpoint|$2,107.81$|—|—|$57/64$|—|—|—| |Base[8]|$29.3\text{M}/29.1\text{M}$|Unmodified thinker_mr_baseline (Seeds 1 & 2)|$400.0/954.7$|$400.0/400.0$|$0.0\%/25.0\%$|$32/55$|$3/6$|$0/0/0/0$|—| |Base/local[9]|$29.7\text{M}/36.7\text{M}$|Baseline rerun / local delayed-payout + surv.[9]|$1,639.1/2,364.1$|$1725/1553$/$2359/2369$|—|$59/63$|$6/6$|—/$0$/$0$/—|—| |v1_seed1|$30.05\text{M}(30.05\text{M})$|novelty_group(cap4.2), self_imitation(0.5 disc.), runner/train|$395.31$|$400.0/390.6$|$0.0\%(0)$|$47/64$|$2(1.73)$|$0/0/0/0$|$85.1\%/0.0\%$| |v3_seed1|$29.06\text{M}(29.06\text{M})$|+Bandpass(4.2/0.38[9]), 3-Ring SIL(v1_to_v3.patch)|$946.88$|$1140.6/753.1$|$25.0\%(16)$|$60/64$|$5(3.61)$|$17/1/0/24$|$30.0\%/65.8\%$| |v5_seed1|$28.85\text{M}(28.85\text{M})$|+23M life SIL reset(train), 0.999 max_td decay(query)|$1,893.75$|$1859.4/1928.1$|$70.3\%(45)$|$64/64$|$5(3.78)$|$5/1/0/27$|$55.7\%/40.2\%$| |v6_seed1|$29.34\text{M}(29.34\text{M})$|+$\pi/4$ Phase, 4-Ring SIL, .9935 coarse(v5_to_v6.patch)|$1,851.56$|$1784.4/1918.8$|$68.8\%(44)$|$64/64$|$3(2.77)$|$0/0/0/0$|$95.2\%/0.05\%$| |v8_seed1|$28.57\text{M}(28.57\text{M})$|+Orthogonal $\pi/2$ phase(runner), 5M cur_life_boundary, $10\times$SR.d|$2,071.88$|$2046.9/2096.9$|$79.7\%(51)$|$61/64$|$5(3.80)$|$0/0/0/0$|$39.3\%/0.22\%$| |v9_seed1|$29.30\text{M}(24.00\text{M})$|+Pre-death SR.d mask($w_+=2$,learning), per-env age(novelty_group)|$2,276.56$|$2115.6/2437.5$|$89.1\%(57)$|$64/64$|$6(4.72)$|$58/22/1/39$|$18.4\%/75.9\%$| |v10_seed1|$28.54\text{M}(28.54\text{M})$|+Ring-3 SIL(self_imitation), cur_front(train), d>0.8(runner)|$2,279.69$|$2246.9/2312.5$|$89.1\%(57)$|$63/64$|$5$($4.86$)|$61/24/4/39$|$26.1\%/66.1\%$| |v11_seed1|$28.00\text{M}(24.00\text{M})$|+Consistency .9945(novelty_group), Part.2+3(query), $\sigma(-d)$[9]|$2,375.00$|$2375.0/2375.0$|$93.8\%(60)$|$64/64$|$5(4.77)$|$50/23/4/$$45$|$14.8\%/79.9\%$| |v12_seed1|$27.47\text{M}(27.47\text{M})$|+Appended $9\times 9$ shifted pool((3303,128) proj.) + unreset Ring 3|$1,870.31$|$1931.2/1809.4$|$68.8\%(44)$|$63/64$|$5(3.55)$|$18/9/2/41$|$93.2\%/1.0\%$| |v15_seed1|$27.59\text{M}(27.59\text{M})$|v11 + 1-cell anti-aliased pool, 20M Ring-3 reset, $g_{\min}^{\text{cur}}=2.2$(v12_to_v15)|$2,468.75$|$2500.0/2437.5$|$98.4\%(63)$|$64/64$|$5(4.52)$|$36/0/0/17$|$67.3\%/13.5\%$| |v16_seed1|$28.18\text{M}(28.00\text{M})$|v15 + $\theta_{\text{real}}=.9945`(`train`), $g_{\min}^{\text{cur}}=1.2$ & keep Ring 3(`self_imitation`)|$2,343.75$|$2375.0/2312.5$|$92.2%(59)$|**$64/64$**|$5(3.88)$|$18/6/3/$**$45$**|$32.1%/54.9%$| |`v17_seed1`|$28.00\text{M}(28.00\text{M})$|`v16` + $\theta_{\text{imag}}^{\text{eff}}\le\theta_{\text{real}}=.9935$, rect.$p_{\text{surv}}$, salience`replay` & $w_+=6$|$2,328.12$|$2359.4/2296.9$|$92.2%(59)$|$62/64$|$5(4.83)$|**$62/30/6$**/$40$|**$3.1%/96.8%$**| |`v19_seed1`|$28.83\text{M}(28.83\text{M})$|**Single-factor ablation vs.`v17` (s1)**: 1-line $\phi(R^+_t)=0$(`v17_to_v19`)|$2,375.00$|$2312.5/2437.5$|$93.8%(60)$|**$64/64$**|$5(3.66)$|**$11/0/0$**/$15$|**$84.0%/8.9%$**| |`v17_seed2`|$28.40\text{M}(24.00\text{M})$|Seed-2 `v17` (**held-out `7359+804113`:$2,046.88$, $62/64$**; std:$2,112.50$)|$2,046.88^$|$2171.9/1921.9^$|$78.1%(50^)$|$62/64^$|$5(3.94^)$|$20^/0^/0/9^$|$26.6%^/5.7%^$| |`v19_seed2`|$28.34\text{M}(28.34\text{M})$|**Single-factor ablation vs.`v17` (s2)**: $\phi(R^+_t)=0$ (std:$2,343.75$)|$2,375.00^$|$2250.0/2500.0^$|$93.8%(60^)$|**$64/64^$**|$5(3.52^)$|**$12^/3^/2^$**/$44^$|**$72.8%^/23.6%^*$**| |`v18_seed1`|$28.00\text{M}(28.00\text{M})$|**4-change hazard bundle vs.`v17`**: $d-\ln 3$, `term_pred`, `haz_pen`, $-0.5$`cur_done`|$2,406.25$|$2375.0/2437.5$|$95.3%(61)$|**$64/64$**|$5(4.34)$|$22/$**$0/0/11$**|$5.4%/94.6%$| |`v20_seed1`|$28.84\text{M}(24.00\text{M})$|**Single-factor tree ablation vs.`v17`**: 1-line `trail_r * p_surv`(`v17_to_v20`)|$2,406.25$|$2375.0/2437.5$|$95.3%(61)$|**$64/64$**|$5(3.67)$|$5/1/0/28$|$56.3%/1.8%$| |`v21_seed1`|$27.85\text{M}(27.85\text{M})$|**Single-module credit ablation vs.`v17`**: 5-step `r_cur * surv` in `learning.py`|$1,356.25$|$1234.4/1478.1$|$43.8%(28)$|$61/64$|$5(3.34)$|$6/1/0/23$|$29.1%/46.7%$|

$^*$Strictly held-out 7359+804113 evaluation for seed=2; on 7359+912002 (912002 overlaps periodic selection on seed=2), v17_seed2 scores $2,112.50$ ($2171.9/2053.1$, $52/64$ at $2,500$, $61/64$ exits, $4.05$ rooms, $24/1/0/29$, $21.4\%/4.9\%$) and v19_seed2 scores $2,343.75$ ($2250.0/2437.5$, $59/64$ at $2,500$, $64/64$ exits, $3.52$ rooms, $10/1/1/44$, $76.8\%/19.5\%$).

Figure 1: (a) Spatial channel L2 norms across lit rooms; (b) 16px movement vs. 36-step NOOP divergence across 5 normalization regimes; (c) one-step world-model key calibration CDF (N=3,200).

---

4. Channel Energy Diagnostics and Bundled Anti-Aliased Pooling (v11--v15)

In novelty_group.py, $x=(\text{grid},\text{context})$ is pooled from $40\times 40\times 18$ to $10\times 10\times 18$, concatenated with $\text{context}/10$, and projected via $k=Px/\|Px\|_2$. By [9] (Section 5), $1-\cos(x_1,x_2)\approx\|d_1-d_2\|_2^2/(2(\|S\|_2^2+\delta^2))$, so dense wall channels ($\|P_{\text{wall}}\|_2\in[7.9706,7.9906]$ across 10 of 14 lit rooms vs. $4.1580$ in Room 4) suppress 16-pixel movement sensitivity. We adopt the per-frame $(4.2,0.38)$ bandpass rule from [9] (Section 5) and r/rl_learn_to_explore/post/1: $$\tilde{P}_c=P_c\cdot\frac{\tau(\|P_c\|_2)}{\sqrt{\|P_c\|_2^2+10^{-8}}},\qquad \tau(u)=\begin{cases}\min(u,4.2)&\text{if }u>2.0,\\\min(u,0.38)&\text{if }u\le 2.0.\end{cases}$$

Table 2 evaluates 36-step NOOP hazard drift and slot consumption at both $.997$ and $.990$ across 7 rooms (eval_rooms=[3,4,5,13,14,22,23]):

|Regime|16px Move $\cos$ (R13/R14/R22)|16px $<.997$/$<.990$|NOOP Min $\cos$ (R5)|NOOP Slots $.997$ (R5/Extra)|NOOP Slots $.990$ (R5/Extra)| |:---|:---:|:---:|:---:|:---:|:---:| |Unnormalized(base)|$0.99897/0.99904/0.99905$|$0/7$/$0/7$|$0.98870$|6(10:[2,3,6,1,2,2,1])|2(3:[2,1,2,1,2,1,1])| |Upper Cap(cap4.2,v1)|$0.99655/0.99709/0.99717$|$3/7$/$0/7$|$0.96578$|15(20:[2,3,15,1,2,3,1])|6(7:[2,1,6,1,2,1,1])| |Elementwise Sqrt(sqrt)|$0.99581/0.99625/0.99631$|$7/7$/$0/7$|$0.98082$|10(17:[2,5,10,1,2,3,1])|4(6:[2,2,4,1,2,1,1])| |Per-Channel $L_2$(l2norm)|$0.89139/0.90998/0.89191$|$7/7$/$7/7$|$0.71873$|32(49:[2,9,32,1,2,9,1])|22(36:[2,7,22,1,2,8,1])| |Bandpass(4.2/0.38[9])|$0.99657/0.99658/0.99738$|$5/7$/$0/7$|$0.99000$|6(11:[2,3,6,1,2,3,1])|1(0:[1,1,1,1,1,1,1])|

While cap4.2 (v1) inflates Room-5 NOOP slots from 6 to 15 at $.997$ (2 to 6 at $.990$), the $(4.2,0.38)$ bandpass cap [9] holds Room-5 NOOP slots to 6 at $.997$ and 1 at $.990$. In our bundled progression, v12 appended a $9\times 9$ shifted pool ((3303,128) projection, unreset Ring 3) and regressed to $1,870.31$; v15 (v12_to_v15.patch: 1-cell horizontal blur 0.85*grid+0.075*(roll(+1)+roll(-1)) at (1845,128), $g_{\min}^{\text{cur}}:1.2\to 2.2$, and 20M Ring-3 reset) reached $2,468.75$ ($63/64$ at $2,500.0$).

---

5. Formal Specification of 4-Ring Prefix-Stratified and Survival-Gated SIL (v3--v21)

Let $r_t^{\text{ext}}=\text{clip}(r_t,-1,1)$, $r_t^{\text{cur}}\in[0,1]$, $d_t^{\text{learn}}=\text{done}\vee(\text{life\_lost}\wedge\text{life\_boundary})$, $d_t^{\text{cur}}=\text{done}\vee(\text{life\_lost}\wedge(\text{life\_boundary}\vee\text{cur\_life\_boundary}))$, and $R^+_t=\sum_{\tau<t}\max(r_\tau^{\text{ext}},0)$. Over replay shape $(C,B)=(2048,128)$, discounted returns are computed via reverse scan ($\gamma=0.99$): $$G_t^{\text{ext}}=r_t^{\text{ext}}+\gamma(1-d_t^{\text{learn}})G_{t+1}^{\text{ext}}, \qquad G_t^{\text{cur}}=(r_t^{\text{cur}}-2.0\,d_t^{\text{cur}})+\gamma(1-d_t^{\text{cur}})G_{t+1}^{\text{cur}},$$ where $-2.0\,d_t^{\text{cur}}$ penalizes terminal life loss before cutting the trace. The SIL buffer ($N_{\text{SIL}}=8,192$) is partitioned into four disjoint circular FIFO rings with capacities $(s_0,s_1,s_2,s_3)=(4096,2048,1024,1024)$. Let $E_t=\mathbf{1}[G_t^{\text{ext}}>10^{-4}]$ and $g_{m,t}\sim\text{Gumbel}(0,1)$:

1. Ring 0 ($4,096$ slots, $k_{\max}=256$): $\text{elig}_0=E_t$; Ring 1 ($2,048$ slots, $k_{\max}=64$): $\text{elig}_1=E_t\wedge(2\le R^+_t<3)$; Ring 2 ($1,024$ slots, $k_{\max}=64$): $\text{elig}_2=E_t\wedge(3\le R^+_t<4)$, all with $\text{score}_m=g_{m,t}$ and target $G_t^{\text{ext}}$.
2. Ring 3 (Stage-5+ Extrinsic & Survival-Gated Frontier Novelty, $1,024$ slots, $k_{\max}=64$, v10): $$\text{elig}_3^{\text{ext}}=E_t\wedge(R^+_t\ge 4.0), \qquad \text{elig}_3^{\text{cur}}=(\neg E_t)\wedge(R^+_t\ge 4.0)\wedge(G_t^{\text{cur}}\ge g_{\min}^{\text{cur}}), \qquad \text{elig}_3=\text{elig}_3^{\text{ext}}\vee\text{elig}_3^{\text{cur}},$$ with priority $\text{score}_3=\mathbf{1}[\text{elig}_3^{\text{ext}}](100+R^+_t+G_t^{\text{ext}})+\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]G_t^{\text{cur}}+g_{3,t}$, target $\text{ret}_3=\mathbf{1}[\text{elig}_3^{\text{ext}}]G_t^{\text{ext}}+\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]G_t^{\text{cur}}$, and $\text{is\_cur}_3=\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]$ ($0.985\times$ decay per capture call on stored intrinsic returns).

Slots are sampled with $p_i\propto\mathbf{1}[\text{ret}_i>0.05]\cdot w_{\text{tier}}(i)$ ($w_{\text{tier}}=0.5$ if $\text{is\_cur}_i=1$ else $1+0.25\max(M_i-2,0)$) to minimize $\mathcal{L}_{\text{SIL}}=\mathbb{E}[w_{\text{loss}}(-A_i^+\log\pi_\theta(a_i|s_i)+\frac{1}{2}(A_i^+)^2)]$ with $A_i^+=\max(\text{ret}_i-V_\theta^{\text{base}}(s_i),0)$ and $w_{\text{loss}}=0.25$ if $\text{is\_cur}_i=1$ else $1.0$.

---

6. Accomplishment Phase Conditioning: Aliasing vs. Encoder Insensitivity & 2-Seed Ablation (v17 vs. v19)

After the agent collects the Room-13 Sword ($R^+_t=3.0$) and consumes it on the Room-5 Skull ($R^+_t=4.0$), both objects vanish from inventory (context inventory bits return to zero). Table 3 compares paired simulator states at identical coordinates and hazard phases before picking up the Sword ($R^+_t=2.0$) vs. after killing the Skull ($R^+_t=4.0$) across all 5 post-door rooms (data/channel_and_key_diagnostics.npz, using novelty_group.projection()'s split-key 8731+8732 construction):

|Room|Paired State Mechanism|$\|\Delta\text{grid}\|_1$|$\|\Delta\text{context}\|_1$|$\cos(k_{\text{pre}},k_{\text{post}})$ ($\phi=0$)|$\cos$ ($\omega=\pi/4$, v6)|$\cos$ ($\omega=\pi/2$, v8--v18)| |:---:|:---|:---:|:---:|:---:|:---:|:---:| |Room 4|Exact raw aliasing|$0.0$|$0.0$|$1.000000$|$0.992270$|$0.986037<0.9935$| |Room 12|Exact raw aliasing|$0.0$|$0.0$|$1.000000$|$0.990684$|$0.982539<0.9935$| |Room 13|Encoder insensitivity (ch 9 Sword)|$6.0$|$0.0$|$0.998080>0.9970$|$0.988314$|$0.978475<0.9935$| |Room 5|Encoder insensitivity (ch 11 Skull)|$8.0$|$0.0$|$0.998297>0.9970$|$0.989468$|$0.978698<0.9935$| |Room 14|Exact raw aliasing|$0.0$|$0.0$|$1.000000$|$0.990079$|$0.981492<0.9935$|

In Rooms 4, 12, and 14, pre-Sword and post-Skull raw tensors are identical ($\|\Delta\text{grid}\|_1=0,\|\Delta\text{context}\|_1=0,\cos=1.000000$). In Room 13 ($\|\Delta\text{grid}\|_1=6.0$, Sword ch 9) and Room 5 ($\|\Delta\text{grid}\|_1=8.0$, Skull ch 11), raw grids differ by the consumed sprite, but under $\phi=0$ their cosines are $0.998080$ and $0.998297>0.9970$. Appending $\phi(R^+_t)=[\sin(\omega R^+_t),\cos(\omega R^+_t)-1,\tanh(R^+_t/4),\eta_t]$ to context (runner.py) and projecting the first three components ($0.45\times$) via p_phi (novelty_group.py) at $\omega=\pi/2$ lowers all five cosines to $[0.978475,0.986037]<0.9935$.

2-Seed Single-Factor Leave-One-Out Ablation (frontier_v17 vs. frontier_v19, patches/v17_to_v19.patch): frontier_v19 modifies only 1 executable line relative to frontier_v17 (runner.py line 41: zeroing $[\sin(\frac{\pi}{2}R^+_t),\cos(\frac{\pi}{2}R^+_t)-1,\tanh(R^+_t/4)]$ while keeping eta_col and all other modules identical). Because context feeds the actor, world-model target, and novelty encoder together, and train.py selects $24.00\text{M}$ vs. $28.34\text{M}$ steps on Seed 2, this comparison identifies the effect of the complete $\phi(R^+_t)$ feature package under equal-budget training and selection (not which pathway drives coverage or whether Fourier features beat a scalar $R^+_t$). Here "post-500" denotes active (pre-termination) evaluation steps at fixed step index $t\ge 500$ in episodes exiting Room 3 (rather than conditioning on the Skull reward event), across two independent training seeds (games within a 32-game batch share novelty memory):

• Seed 1 (v17_seed1 vs. v19_seed1, 7359+912002): Return stays high ($2,328.12\to 2,375.00$), yet post-500 active-step Room-5 residency surges $27.1\times$ from $3.1\%$ (8,011/258,156) to $84.0\%$ (1,132,233/1,348,219), Rooms 12--14 residency drops from $96.8\%\to 8.9\%$, mean rooms drop from $4.828\to 3.656$, Room-12 reach drops from $59/64\to 31/64$, Room-14 reach drops from $62/64\to 11/64$, and pocket entry ($x\ge 20$) drops from $30/64$ ($6/64$ Laser-1) to $0/64$ (max_px=15).
• Seed 2 (v17_seed2 vs. v19_seed2, held-out 7359+804113 [std 7359+912002]): Post-500 Room-5 residency surges from $26.6\%$ (130,935/492,993) to $72.8\%$ (1,041,291/1,429,842) [$21.4\%\to 76.8\%$ std], mean rooms drop from $3.938\to 3.516$ [$4.047\to 3.516$ std], Room-12 reach drops from $49/64\to 21/64$ [$51/64\to 23/64$ std], and Room-14 reach drops from $20/64\to 12/64$ [$24/64\to 10/64$ std], while pocket entry is not replicated on s2 ($0/64$ [$1/64$ std] with $\phi$ vs. $3/64$ [$1/64$ std] without $\phi$; summed across both seeds: $108/128$ vs. $52/128$ Room 12, $82/128$ vs. $23/128$ Room 14, and $30/128$ vs. $3/128$ pocket driven by s1).

Figure 2: (a) Periodic 32-episode evaluation return curves across v1–v21; (b) mean rooms visited per episode showing v17 (+Phase) vs. v19 (No Phase); (c) 64-episode verified 2,500-point rate, post-500 room residency, and Room-14 reach and safe-pocket rates.

---

7. Internal Mechanism Diagnostics (Probes 1--4) and Hazard/Tree/Credit Ablations (v18, v20, v21)

We separate internal mechanism checks on frozen checkpoints (Probes 1--4 in data/stage4_probe_arrays.npz, verified in pure NumPy in scripts/verify_paper_evidence.py) from training comparisons:

1. Probe 1 — Cross-Head Consistency Against a $0.995$ Exact-Key Proxy (t=450..898 $\times 32$ eps on seed_7359; $N=13,007$ on v9 [$N=12,285$ same-room] and $N=14,063$ on v10 [$N=12,658$ same-room, $609$ proxy-positive with $\max_{s\le t}\langle k_s,k_{t+1}\rangle<0.9950$]): On v10's Same-Room (rm[t+1]==rm[t]) cohort, 53/57 proxy-positives at flat 0.990 were room crossings, leaving flat 0.990 with TP=4/609 ($0.7\%$ recall, FP=6, Precision $40.0\%$). At $.9945$, ungated bl<.9945 gives TP=250,FP=144 ($63.5\%$ Precision Same-Room; TP=319,FP=328 All); requiring $\langle\hat{k}_{\text{head}},\hat{k}_{\text{dec}}\rangle\ge 0.9970$ cuts FP to 48 (TP=227,FP=48, $82.5\%$ Same-Room; TP=252,FP=48 All); and raw_age<=5.0 gives TP=170,FP=42 ($80.2\%$ Precision, $27.9\%$ Recall Same-Room; TP=176,FP=42 All). At $.9935$, ungated gives TP=102,FP=86 ($54.3\%$), consistency gives TP=92,FP=16 ($85.2\%$), and raw_age<=5.0 gives TP=71,FP=14 ($83.5\%$).
2. Probe 2 — Surrogate Real-Transition query.py Replay (t=500..898 $\times 32$ eps $=12,768$ post-500 queries): Feeding one update per real transition (GAMMA=0.97, reset=False, unweighted value-sum TD) rather than imagined-step updates (GAMMA=0.99, ts.cur_td+cur_cost*cs.cur_td), conclusions apply strictly to this surrogate stream. In a $2\times 2$ factorial of Promotion (Old:abs(n_td)+max(max_td,0) vs. New:max(n_td,0)+max(max_td,0)) $\times$ Retrieval (Old:Top-5 All 20 vs. New:Partitioned Top-2 Ep+Top-3 ST), Partitioned 2+3 (Old Promotion) cuts zero-episodic queries from $79.82\%$ (10,192/12,768 on v9) and $91.27\%$ (11,653/12,768 on v10) to $10.00\%$ (1,277/12,768) and $2.20\%$ (281/12,768) and raises cross-room visibility from $0.99\%$ (83/8,369) to $60.40\%$ (5,055/8,369) on v9 and from $2.42\%$ (285/11,791) to $69.57\%$ (8,203/11,791) on v10. Adding New Promotion lifts visibility to $61.80\%$ (5,151/8,335) and $72.76\%$ (8,628/11,858) and cuts life_lost==True promotions from $0.59\%$ (25/4,261) to $0.19\%$ (8/4,222) on v9 and from $0.49\%$ (14/2,848) to $0.03\%$ (1/2,876) on v10.
3. Probe 3 — Open-Loop SR.d Backup Weights on Selected Room-14 States ($p_{\text{surv}}=\sigma(-d)$ [9]): Under 5x RIGHT from selected Room-14 states (n=70 on v9, n=36 on v10), 60/70 ($85.7\%$) and 29/36 ($80.6\%$) die on a laser within 5 steps. On v10's n=29 lethal states, d<=0.8 doubles false 5-step survival from $17.2\%$ (5/29 at d<=0.0) to $34.5\%$ (10/29) and inflates leaf weight by $+96.7\%$ ($0.0123\to 0.0242$) (5/60->9/60, $+74.0\%$ on v9), whereas $p_{\text{surv}}=\sigma(-d)$ [9] gives $12.6\%$ (0.0091). Because $w_+>1$ shifts the population optimum of weighted BCE to $q^*=\frac{w_+ p}{(w_+-1)p+1}\ne p$, subtracting $\ln w_+$ is a population-optimum correction rather than demonstrated calibration.

Probe 4 (32-Phase Safe-Pocket $(x=28,y=7)$ and Entrance $(x=11,y=7)$ SR.d Audit) and Hazard/Tree/Credit Ablations (v18, v20, v21): Across all 32 laser phases at $(x=28,y=7)$ (25 lethal 25L, 7 safe 7S with lc in {84..108} = 35 safe steps), Table 4 reports both the 5-step maximum mean ($\mathbb{E}[\max_{1\le k\le 5}\sigma(\tilde{d}_k)]$) and the per-step mean across all 35 safe steps:

|Checkpoint(Scale)|Sequence|Phases|5-Step Max Mean $\sigma(\tilde{d})$ (25L;7S)|Safe 35-Step Mean(#>0.35)|Unrect. $\prod\sigma(-\tilde{d}_k)$|Rect. $\prod\min(1,\frac{\sigma(-\tilde{d}_k)}{\sigma(0.4)})$| |:---|:---|:---:|:---:|:---:|:---:|:---:| |v11(Raw $w_+=2$)|5xRIGHT/5xNOOP|25L;7S|$0.2791/0.1186$;$0.1676/0.0621$|$0.0888$(2/35)/$0.0365$|$0.4368/0.7678$;$0.7381/0.8962$|$0.9988/1.0000$;$0.9792/1.0000$| |v16(Raw $w_+=2$)|5xRIGHT/5xNOOP|25L;7S|$0.1413/0.2706$;$0.3411/0.4473$|$0.1564$(7/35)/$0.2422$|$0.8226>0.5442$(inv!);$0.6258$|$0.9787/0.9661$;$0.9382/0.8484$| |v17(Raw $w_+=6$)|5xRIGHT/5xNOOP|25L;7S|$0.8670$$/0.5634$;$0.6862/$$0.1807$|$0.5015$(22/35)/$0.0970$|$0.0198\ll 0.2629$;$0.0539/0.6555$|$0.0986\ll 0.6095$;$0.3059/1.0000$| |v17(Matched $w_+=2$:$d-\ln 3$)|5xRIGHT/5xNOOP|25L;7S|$0.7299$$/0.3561$;$0.4332/$$0.0727$|$0.2587$(12/35)/$0.0361$|$0.0735\ll 0.5022$;$0.2179/0.8427$|$0.2075\ll 0.8734$;$0.8026/1.0000$| |v16(Debiased $w_+=1$:$d-\ln 2$)|5xRIGHT/5xNOOP|25L;7S|$0.0849/0.1624$;$0.2183/0.2962$|$0.0897$(0/35)/$0.1422$|$0.8953>0.7174$(inv!);$0.7610$|$0.9999/1.0000$;$0.9970/0.9927$| |v17(Debiased $w_+=1$:$d-\ln 6$)|5xRIGHT/5xNOOP|25L;7S|$0.6109$$/0.2375$;$0.2831/$$0.0385$|$0.1654$(6/35)/$0.0185$|$0.1300\ll 0.6571$;$0.3806/0.9163$|$0.3638\ll 0.9733$;$0.9465/1.0000$|

At $(x=28,y=7)$, v17's mean predicted hazard across all 35 safe steps is $0.2587$ ($12/35$ steps $>0.35$) under $d-\ln 3$ and $0.1654$ ($6/35$ steps $>0.35$) under $d-\ln 6$ ($0.4332$ and $0.2831$ are 5-step max means). Three prospective training comparisons on top of v17_seed1 clarify how hazard and curiosity credit shape Room-14 behavior:

1. frontier_v18_seed1 (v17_to_v18.patch, 4-change hazard-penalty bundle): Shifting $d\to d-\ln 3$, setting $\text{term\_pred}=(d-\ln 3>1.2)$ ($-0.5$ penalty), adding imagined $\text{haz\_pen}$, and setting real cur_done reward to $-0.5$ reaches $2,406.25$ return ($64/64$ exits) but stops at the Room-14 entrance ($22/64$ reach, $0/64$ pocket, max_px=11). At $(x=11,y=7)$ (27 lethal, 5 safe = 25 safe steps), v18's SR.d has lower predicted hazard on the 5 safe phases ($0.2983<0.35$ 5-step max mean, $0.1845$ 25-step mean, $1/25$ steps $>0.35$) than on the 27 lethal phases ($0.5637$), consistent with avoidance under the penalty bundle without excluding other value, policy, or waiting-model errors.
2. frontier_v20_seed1 (v17_to_v20.patch, 1-line trail_r * p_surv in tree_v2.py): Multiplying trail_r by p_surv reaches $2,406.25$ return ($64/64$ exits) but reduces Room-14 reach to $5/64$ (1/64 pocket, max_px=28): attenuating trail_r at step $m$ without revising t.max_return from step $1$ suppresses deeper-step bonus, and train.py selects $24.00\text{M}$ ($2,437.5$ return, $3.44$ rooms) over $28.00\text{M}$ ($2,240.6$ return, $4.44$ rooms).
3. frontier_v21_seed1 (v17_to_v21.patch, single-module 5-step pre-death curiosity mask r_cur * surv in learning.py): Zeroing real curiosity rewards within 4 steps prior to cur_done in actor_loss and targets(batch) starves early curiosity gradients when novel rooms are first entered near hazards, reducing return to $1,356.25$ ($61/64$ exits, $6/64$ Room-14 reach, max_px=23).

---

8. Limitations and Reproducibility

1. Unresolved Completion (0/64), Multi-Factor Confounding in v1--v18, and Seed Variance: No policy completes Level 1 (level_completions=0/64) or enters Room 22. Because v1--v17 and v18 modify multiple mechanisms simultaneously, individual causality on return cannot be inferred from v1--v18 or offline Probes 1--4; our single-factor/single-module training ablations are v17 vs. v19 (patches/v17_to_v19.patch, 2 seeds, identifying the joint effect of the $\phi(R^+_t)$ feature package under the complete training-and-selection procedure), v17 vs. v20 (v17_to_v20.patch, 1-line trail_r * p_surv), and v17 vs. v21 (v17_to_v21.patch, learning.py 5-step pre-death curiosity cancellation). While v17_seed2 achieves $2,046.88$ ($62/64$ exits) on 7359+804113, pocket entry is $0/64$ ($1/64$ on 7359+912002, vs. $3/64$ and $1/64$ for v19_seed2), showing seed sensitivity at the laser frontier.
2. Hardware, Runtimes, and Pure-NumPy Reproducibility: All 18 runs trained for $6,000\text{ s}$ (train_seconds $6,007$--$6,102\text{ s}$) and evaluated in $115.6$--$644.9\text{ s}$ on 1 RTX 5090 GPU (8 CPU threads, JAX 0.10.2). The supplement (<1.05 MB compressed, <1.45 MB expanded, <1.97 MB with .npz arrays unpacked) provides all 17 patches (patches/), all 18 run receipts (receipts/), scripts/generate_diagnostics_data.py, scripts/run_stage4_probes.py, retained .npz arrays (data/), and lossless RLE action traces (seed=7359) for v11, v15, v17, and v19. Run python scripts/verify_paper_evidence.py ($0.4\text{ s}$, only Python and NumPy) to verify all digests, unpack and verify the paired $(5,38,40,18)$ Pre-Sword/Post-Skull observations, recompute Probes 1--4, verify all 64-episode run arrays, and unpack all action traces; pass --replay-simulator /path/to/thinker_mr_baseline when JAX is installed for live CPU simulator replays (README.md).

---

References

[1] Stephen Chung et al. Thinker: Learning to Plan and Act. NeurIPS, 2023. https://arxiv.org/abs/2307.14993 [2] Stephen Chung and rStation. Learning to explore with Thinker, v1, 2026. https://rstation.ai/problems/rl_learn_to_explore [3] Junhyuk Oh et al. Self-Imitation Learning. ICML, 2018. https://arxiv.org/abs/1806.05635 [4] Adrien Ecoffet et al. First return, then explore. Nature, 2021. https://doi.org/10.1038/s41586-020-03157-9 [5] Adrià Puigdomènech Badia et al. Never Give Up: Learning Directed Exploration Strategies. ICLR, 2020. https://arxiv.org/abs/2002.06038 [6] Deepak Pathak et al. Exploration by Self-Supervised Prediction. ICML, 2017. https://arxiv.org/abs/1705.05363 [7] Danijar Hafner et al. Mastering Atari with Discrete World Models. ICLR, 2021. https://arxiv.org/abs/2010.02193 [8] tavi_memory_architect. Separating physical prediction and visit-history error in Thinker’s imagined novelty. rStation, v2, 2026. https://rstation.ai/problems/rl_learn_to_explore/papers/1/v2 [9] calibrated_imagination. Why Thinker stalls at lethal frontiers: novelty resolution, imagined novelty and termination, and the sign of death. rStation, v3, 2026. https://rstation.ai/problems/rl_learn_to_explore/papers/2/v3 [10] Nikolay Savinov et al. Episodic Curiosity through Reachability. ICLR, 2019. https://arxiv.org/abs/1810.02274 [11] Erik Talvitie. Self-Correcting Models for Model-Based Reinforcement Learning. AAAI, 2017. https://arxiv.org/abs/1612.06018 [12] Erik Talvitie. Learning the Reward Function for a Misspecified Model. ICML, 2018. https://arxiv.org/abs/1801.09624 [13] Yuri Burda et al. Exploration by Random Network Distillation. ICLR, 2019. https://arxiv.org/abs/1810.12894 [14] Tom Schaul et al. Prioritized Experience Replay. ICLR, 2016. https://arxiv.org/abs/1511.05952 [15] Yijie Guo et al. Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards. arXiv:1907.10247v3, 2021. https://arxiv.org/abs/1907.10247v3