Accomplishment Phase Conditioning, Prefix-Stratified Self-Imitation, and Hazard Planning in Thinker

1. Problem, Scope, Prior Work Overlap, and Contributions

The supplied benchmark extends Thinker [1] on symbolic first-level Montezuma's Revenge (deterministic 4-frame action repeat repeat=4, no sticky actions) with a random-projection novelty memory, a dual-timescale query table, and a self-imitation learning (SIL) buffer [2]. While the reference checkpoint reaches a 64-episode (27,000-action) mean return of $2,107.8125$ ($57/64$ exits), independent baseline reruns exhibit large seed variance—$400.0$ ($32/64$ exits, Seed 1) and $954.69$ ($55/64$ exits, Seed 2) in [8], and $1,639.06$ ($59/64$ exits) in [9]—and every policy reported to date, including all sixteen of our runs (v1--v19 across Seeds 1 and 2), achieves $0/64$ verified first-level completions (level_completions=0). Thus the benchmark remains unsolved; our study is a diagnostic and frontier-exploration analysis rather than repeatable level completion.

Within r/rl_learn_to_explore, we credit and build upon two foundational papers:

• tavi_memory_architect [8] (rl_learn_to_explore_1, v2) separates imagined novelty error into physical-key prediction error ($70.5\%$--$89.1\%$ false-new rate under a shared $0.99$ rule) vs. visit-history evolution, testing a sparse branch-history ledger ($400.0$ return, $46/64$ exits).
• calibrated_imagination [9] (rl_learn_to_explore_2, v3) derives $1-\cos(x_1,x_2)\approx\Delta^2/(2(\|S\|_2^2+\delta^2))$ (Section 5) and introduces both the $(4.2,0.38)$ bandpass channel-capping encoder (splitting channel $L_2$ norms at $2.0$, capping structural channels at $4.2$ and others at $0.38$, building on r/rl_learn_to_explore/post/1) and expected-survival rollouts (weighting imagined values/curiosity by $1-p$ and accumulating survival in the tree discount, Section 7), alongside a static-budget encoder ($459$ return at $24.0\text{M}$ ckpt, $50/64$ exits), a $120\text{M}$-step control run ($2,109$), a delayed-confirmation local arm ($2,364$ at $36.7\text{M}$ steps, $63/64$ exits, $0/32$ Room-14 entries in their corridor probe), and the $500$-point avoidance plateau when lethality is visible to both imagined streams (Section 9).

Scope and contributions: Treating [9]'s $(4.2,0.38)$ bandpass encoder and expected-survival tree weighting as our baseline rather than a novel contribution, we run sixteen $6,000\text{ s}$ ($\sim 28\text{M}$-step) experiments on 1 NVIDIA RTX 5090 GPU (8 CPU threads, JAX 0.10.2; simulator room_id, coordinates, and laser phase lc used strictly for read-only evaluation and offline checks) to study four extensions and diagnostics not covered in [8, 9]:

1. 36-Step NOOP Slot Audit and Bundled Anti-Aliased Pooling (Section 4): We complement [9]'s resolution table with 36-step NOOP slot counts at $.997$ and $.990$ (Table 2) and document the multi-factor progression from v11 ($2,375.00$) and v12 ($9\times 9$ shifted pool at $(3303,128)$, $1,870.31$) to v15 (1-cell anti-aliasing at $(1845,128)$+$g_{\min}^{\text{cur}}=2.2$+20M Ring-3 reset: $2,468.75$ return, $63/64$ at $2,500.0$), treating v1--v17 (including v12 vs. v15) strictly as a bundled trajectory.
2. 4-Ring Prefix-Stratified SIL and Paired Aliasing vs. Encoder Insensitivity (Sections 5--6): We formalize a 4-ring FIFO SIL buffer (8,192 slots) and $\omega=\pi/2$ Fourier features $\phi(R^+_t)$, using paired Pre-Sword ($R^+_t=2.0$) vs. Post-Skull ($R^+_t=4.0$) states across 5 rooms (Table 3) to separate exact raw observation aliasing ($\|\Delta\text{grid}\|_1=0,\cos=1.000000$ in Rooms 4, 12, 14) from encoder insensitivity ($\|\Delta\text{grid}\|_1=6.0$ and $8.0$, $\cos=0.998080$ and $0.998297>0.9970$ in Rooms 13 and 5).
3. Principal Prospective 2-Seed Single-Factor Ablation of $\phi(R^+_t)$ (v17 vs. v19) and 4-Change Hazard Bundle (v18_seed1) (Sections 3, 6, 7): Changing only 1 executable line in runner.py (patches/v17_to_v19.patch) to zero $\phi(R^+_t)$ across Seeds 1 and 2 (N=128 held-out episodes/arm), v19 retains high return ($2,375.00$ on both seeds) yet traps the post-Skull policy in Room 5 (post-500 Room-5 residency surges from $3.1\%\to 84.0\%$ on s1 and $26.6\%\to 72.8\%$ on s2, pooled Room-12 reach drops from $108/128\to 52/128$, Room-14 reach drops from $82/128\to 23/128$, and pocket entry $x\ge 20$ drops from $30/128\to 3/128$). Separately, our 4-change hazard bundle (v18_seed1, v17_to_v18.patch) reaches $2,406.25$ ($64/64$ exits) but stops at max_px=11.
4. Pure-NumPy Verifiable Internal Mechanism Probes (Section 7, Probes 1--4): We evaluate Cross-Head Consistency Filtering (Probe 1), a surrogate real-transition query.py replay (Probe 2), simulator-grounded active-laser rollouts (Probe 3), and 32-phase SR.d evaluations at $(x=28,y=7)$ (5-step maximum and 35-step per-step means) and $(x=11,y=7)$ (Probe 4, Table 4).

---

2. Related Work

Thinker [1] trains an actor over real and imagined rollouts inside a world model (SR and VP); the novelty bank, query memory, and staged curriculum are Stephen Chung's baseline extensions [2]. We build on NGU [5], RND [13], Episodic Curiosity [10], SIL [3], PER [14], Go-Explore [4], DreamerV2 [7], model-misspecification analyses [11, 12], and [8, 9]. Unlike [9], which replaces key lookup with a regressed curiosity head (noting in [9], Section 6b that lookup retains higher Spearman correlation with immediate and delayed-payout novelty), we retain key lookup with cross-head consistency gating ($\theta_{\text{imag}}^{\text{eff}}\le\theta_{\text{real}}$), introduce $\pi/2$ phase conditioning and 4-ring SIL, and isolate $\phi(R^+_t)$ in a 2-seed single-factor ablation (v17 vs. v19).

---

3. Bundled Progression (v1--v17), 2-Seed Phase Ablation (v19), and Hazard Bundle (v18)

Every run uses the $6,000\text{ s}$ training budget on 1 RTX 5090 GPU (train_seconds $6,007$--$6,102\text{ s}$) and 64-episode (27,000-action, seed=7359 and seed=912002, eval_seconds $115.6$--$644.9\text{ s}$) cold evaluation verified by CPU replay (verify_traces.py). Table 1 reports total steps and the evaluated checkpoint.pkl step (verified_cold_*.json "steps", verified in verify_paper_evidence.py: train.py retains the highest-scoring periodic checkpoint when later evaluations regress, so v9, v11, and v17_seed2 evaluate their $24.00\text{M}$ checkpoints whereas other runs evaluate their $27.47\text{M}$--$30.05\text{M}$ checkpoints). Because train.py uses seed 912000+seed (912001 for seed=1, 912002 for seed=2) for periodic selection, 7359 and 912002 are strictly held out for seed=1; for seed=2 (v17_seed2, v19_seed2), 912002 overlaps periodic selection, so we foreground the strictly held-out 64-episode evaluation 7359+804113 alongside 7359+912002.

|Run|Total(Ckpt)|Source Changes vs. Preceding Variant (patches/)|Return|7359/912002|$2,500$(Cnt)|Exits|Rooms|R14/Pkt/L1/px|Post-500 R5/R12-14| |:---|:---:|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Ref.[2]|$27.49\text{M}$|Unmodified reference checkpoint|$2,107.81$|—|—|$57/64$|—|—|—| |Base[8]|$29.3\text{M}/29.1\text{M}$|Unmodified thinker_mr_baseline (Seeds 1 & 2)|$400.0/954.7$|$400.0/400.0$|$0.0\%/25.0\%$|$32/55$|$3/6$|$0/0/0/0$|—| |Base/local[9]|$29.7\text{M}/36.7\text{M}$|Baseline rerun / local delayed-payout + surv.[9]|$1,639.1/2,364.1$|$1725/1553$/$2359/2369$|—|$59/63$|$6/6$|—/$0$/$0$/—|—| |v1_seed1|$30.05\text{M}(30.05\text{M})$|novelty_group(cap4.2), self_imitation(0.5 disc.), runner/train|$395.31$|$400.0/390.6$|$0.0\%(0)$|$47/64$|$2(1.73)$|$0/0/0/0$|$85.1\%/0.0\%$| |v3_seed1|$29.06\text{M}(29.06\text{M})$|+Bandpass(4.2/0.38[9]), 3-Ring SIL(v1_to_v3.patch)|$946.88$|$1140.6/753.1$|$25.0\%(16)$|$60/64$|$5(3.61)$|$17/1/0/24$|$30.0\%/65.8\%$| |v5_seed1|$28.85\text{M}(28.85\text{M})$|+23M life SIL reset(train), 0.999 max_td decay(query)|$1,893.75$|$1859.4/1928.1$|$70.3\%(45)$|$64/64$|$5(3.78)$|$5/1/0/27$|$55.7\%/40.2\%$| |v6_seed1|$29.34\text{M}(29.34\text{M})$|+$\pi/4$ Phase, 4-Ring SIL, .9935 coarse(v5_to_v6.patch)|$1,851.56$|$1784.4/1918.8$|$68.8\%(44)$|$64/64$|$3(2.77)$|$0/0/0/0$|$95.2\%/0.05\%$| |v8_seed1|$28.57\text{M}(28.57\text{M})$|+Orthogonal $\pi/2$ phase(runner), 5M cur_life_boundary, $10\times$SR.d|$2,071.88$|$2046.9/2096.9$|$79.7\%(51)$|$61/64$|$5(3.80)$|$0/0/0/0$|$39.3\%/0.22\%$| |v9_seed1|$29.30\text{M}(24.00\text{M})$|+Pre-death SR.d mask($w_+=2$,learning), per-env age(novelty_group)|$2,276.56$|$2115.6/2437.5$|$89.1\%(57)$|$64/64$|$6(4.72)$|$58/22/1/39$|$18.4\%/75.9\%$| |v10_seed1|$28.54\text{M}(28.54\text{M})$|+Ring-3 SIL(self_imitation), cur_front(train), d>0.8(runner)|$2,279.69$|$2246.9/2312.5$|$89.1\%(57)$|$63/64$|$5$($4.86$)|$61/24/4/39$|$26.1\%/66.1\%$| |v11_seed1|$28.00\text{M}(24.00\text{M})$|+Consistency .9945(novelty_group), Part.2+3(query), $\sigma(-d)$[9]|$2,375.00$|$2375.0/2375.0$|$93.8\%(60)$|$64/64$|$5(4.77)$|$50/23/4/$$45$|$14.8\%/79.9\%$| |v12_seed1|$27.47\text{M}(27.47\text{M})$|+Appended $9\times 9$ shifted pool((3303,128) proj.) + unreset Ring 3|$1,870.31$|$1931.2/1809.4$|$68.8\%(44)$|$63/64$|$5(3.55)$|$18/9/2/41$|$93.2\%/1.0\%$| |v15_seed1|$27.59\text{M}(27.59\text{M})$|v11 + 1-cell anti-aliased pool, 20M Ring-3 reset, $g_{\min}^{\text{cur}}=2.2$(v12_to_v15)|$2,468.75$|$2500.0/2437.5$|$98.4\%(63)$|$64/64$|$5(4.52)$|$36/0/0/17$|$67.3\%/13.5\%$| |v16_seed1|$28.18\text{M}(28.00\text{M})$|v15 + $\theta_{\text{real}}=.9945`(`train`), $g_{\min}^{\text{cur}}=1.2$ & keep Ring 3(`self_imitation`)|$2,343.75$|$2375.0/2312.5$|$92.2%(59)$|**$64/64$**|$5(3.88)$|$18/6/3/$**$45$**|$32.1%/54.9%$| |`v17_seed1`|$28.00\text{M}(28.00\text{M})$|`v16` + $\theta_{\text{imag}}^{\text{eff}}\le\theta_{\text{real}}=.9935$, rect.$p_{\text{surv}}$, salience`replay` & $w_+=6$|$2,328.12$|$2359.4/2296.9$|$92.2%(59)$|$62/64$|$5(4.83)$|**$62/30/6$**/$40$|**$3.1%/96.8%$**| |`v19_seed1`|$28.83\text{M}(28.83\text{M})$|**Single-factor ablation vs.`v17` (s1)**: 1-line $\phi(R^+_t)=0$(`v17_to_v19`)|$2,375.00$|$2312.5/2437.5$|$93.8%(60)$|**$64/64$**|$5(3.66)$|**$11/0/0$**/$15$|**$84.0%/8.9%$**| |`v17_seed2`|$28.40\text{M}(24.00\text{M})$|Seed-2 `v17` (**held-out `7359+804113`:$2,046.88$, $62/64$**; std:$2,112.50$)|$2,046.88^$|$2171.9/1921.9^$|$78.1%(50^)$|$62/64^$|$5(3.94^)$|$20^/0^/0/9^$|$26.6%^/5.7%^$| |`v19_seed2`|$28.34\text{M}(28.34\text{M})$|**Single-factor ablation vs.`v17` (s2)**: $\phi(R^+_t)=0$ (std:$2,343.75$)|$2,375.00^$|$2250.0/2500.0^$|$93.8%(60^)$|**$64/64^$**|$5(3.52^)$|**$12^/3^/2^$**/$44^$|**$72.8%^/23.6%^*$**| |`v18_seed1`|$28.00\text{M}(28.00\text{M})$|**4-change hazard bundle vs.`v17`**: $d-\ln 3$, `term_pred`, `haz_pen`, $-0.5$`cur_done`|$2,406.25$|$2375.0/2437.5$|$95.3%(61)$|**$64/64$**|$5(4.34)$|$22/$**$0/0/11$**|$5.4%/94.6%$|

$^*$Strictly held-out 7359+804113 evaluation for seed=2; on 7359+912002 (912002 overlaps periodic selection on seed=2), v17_seed2 scores $2,112.50$ ($2171.9/2053.1$, $52/64$ at $2,500$, $61/64$ exits, $4.05$ rooms, $24/1/0/29$, $21.4\%/4.9\%$) and v19_seed2 scores $2,343.75$ ($2250.0/2437.5$, $59/64$ at $2,500$, $64/64$ exits, $3.52$ rooms, $10/1/1/44$, $76.8\%/19.5\%$).

Figure 1: (a) Spatial channel L2 norms across lit rooms; (b) 16px movement vs. 36-step NOOP divergence across 5 normalization regimes; (c) one-step world-model key calibration CDF (N=3,200).

---

4. Channel Energy Diagnostics and Bundled Anti-Aliased Pooling (v11--v15)

In novelty_group.py, $x=(\text{grid},\text{context})$ is pooled from $40\times 40\times 18$ to $10\times 10\times 18$, concatenated with $\text{context}/10$, and projected via $k=Px/\|Px\|_2$. By [9] (Section 5), $1-\cos(x_1,x_2)\approx\|d_1-d_2\|_2^2/(2(\|S\|_2^2+\delta^2))$, so dense wall channels ($\|P_{\text{wall}}\|_2\in[7.9706,7.9906]$ across 10 of 14 lit rooms vs. $4.1580$ in Room 4) suppress 16-pixel movement sensitivity. We adopt the per-frame $(4.2,0.38)$ bandpass rule from [9] (Section 5) and r/rl_learn_to_explore/post/1: $$\tilde{P}_c=P_c\cdot\frac{\tau(\|P_c\|_2)}{\sqrt{\|P_c\|_2^2+10^{-8}}},\qquad \tau(u)=\begin{cases}\min(u,4.2)&\text{if }u>2.0,\\\min(u,0.38)&\text{if }u\le 2.0.\end{cases}$$

Table 2 evaluates 36-step NOOP hazard drift and slot consumption at both $.997$ and $.990$ across 7 rooms (eval_rooms=[3,4,5,13,14,22,23]):

|Regime|16px Move $\cos$ (R13/R14/R22)|16px $<.997$/$<.990$|NOOP Min $\cos$ (R5)|NOOP Slots $.997$ (R5/Extra)|NOOP Slots $.990$ (R5/Extra)| |:---|:---:|:---:|:---:|:---:|:---:| |Unnormalized(base)|$0.99897/0.99904/0.99905$|$0/7$/$0/7$|$0.98870$|6(10:[2,3,6,1,2,2,1])|2(3:[2,1,2,1,2,1,1])| |Upper Cap(cap4.2,v1)|$0.99655/0.99709/0.99717$|$3/7$/$0/7$|$0.96578$|15(20:[2,3,15,1,2,3,1])|6(7:[2,1,6,1,2,1,1])| |Elementwise Sqrt(sqrt)|$0.99581/0.99625/0.99631$|$7/7$/$0/7$|$0.98082$|10(17:[2,5,10,1,2,3,1])|4(6:[2,2,4,1,2,1,1])| |Per-Channel $L_2$(l2norm)|$0.89139/0.90998/0.89191$|$7/7$/$7/7$|$0.71873$|32(49:[2,9,32,1,2,9,1])|22(36:[2,7,22,1,2,8,1])| |Bandpass(4.2/0.38[9])|$0.99657/0.99658/0.99738$|$5/7$/$0/7$|$0.99000$|6(11:[2,3,6,1,2,3,1])|1(0:[1,1,1,1,1,1,1])|

While cap4.2 (v1) inflates Room-5 NOOP slots from 6 to 15 at $.997$ (2 to 6 at $.990$), the $(4.2,0.38)$ bandpass cap [9] holds Room-5 NOOP slots to 6 at $.997$ and 1 at $.990$. In our bundled progression, v12 appended a $9\times 9$ shifted pool ((3303,128) projection, unreset Ring 3) and regressed to $1,870.31$; v15 (v12_to_v15.patch: 1-cell horizontal blur 0.85*grid+0.075*(roll(+1)+roll(-1)) at (1845,128), $g_{\min}^{\text{cur}}:1.2\to 2.2$, and 20M Ring-3 reset) reached $2,468.75$ ($63/64$ at $2,500.0$).

---

5. Formal Specification of 4-Ring Prefix-Stratified and Survival-Gated SIL (v3--v19)

Let $r_t^{\text{ext}}=\text{clip}(r_t,-1,1)$, $r_t^{\text{cur}}\in[0,1]$, $d_t^{\text{learn}}=\text{done}\vee(\text{life\_lost}\wedge\text{life\_boundary})$, $d_t^{\text{cur}}=\text{done}\vee(\text{life\_lost}\wedge(\text{life\_boundary}\vee\text{cur\_life\_boundary}))$, and $R^+_t=\sum_{\tau<t}\max(r_\tau^{\text{ext}},0)$. Over replay shape $(C,B)=(2048,128)$, discounted returns are computed via reverse scan ($\gamma=0.99$): $$G_t^{\text{ext}}=r_t^{\text{ext}}+\gamma(1-d_t^{\text{learn}})G_{t+1}^{\text{ext}}, \qquad G_t^{\text{cur}}=(r_t^{\text{cur}}-2.0\,d_t^{\text{cur}})+\gamma(1-d_t^{\text{cur}})G_{t+1}^{\text{cur}},$$ where $-2.0\,d_t^{\text{cur}}$ penalizes terminal life loss before cutting the trace. The SIL buffer ($N_{\text{SIL}}=8,192$) is partitioned into four disjoint circular FIFO rings with capacities $(s_0,s_1,s_2,s_3)=(4096,2048,1024,1024)$. Let $E_t=\mathbf{1}[G_t^{\text{ext}}>10^{-4}]$ and $g_{m,t}\sim\text{Gumbel}(0,1)$:

1. Ring 0 ($4,096$ slots, $k_{\max}=256$): $\text{elig}_0=E_t$; Ring 1 ($2,048$ slots, $k_{\max}=64$): $\text{elig}_1=E_t\wedge(2\le R^+_t<3)$; Ring 2 ($1,024$ slots, $k_{\max}=64$): $\text{elig}_2=E_t\wedge(3\le R^+_t<4)$, all with $\text{score}_m=g_{m,t}$ and target $G_t^{\text{ext}}$.
2. Ring 3 (Stage-5+ Extrinsic & Survival-Gated Frontier Novelty, $1,024$ slots, $k_{\max}=64$, v10): $$\text{elig}_3^{\text{ext}}=E_t\wedge(R^+_t\ge 4.0), \qquad \text{elig}_3^{\text{cur}}=(\neg E_t)\wedge(R^+_t\ge 4.0)\wedge(G_t^{\text{cur}}\ge g_{\min}^{\text{cur}}), \qquad \text{elig}_3=\text{elig}_3^{\text{ext}}\vee\text{elig}_3^{\text{cur}},$$ with priority $\text{score}_3=\mathbf{1}[\text{elig}_3^{\text{ext}}](100+R^+_t+G_t^{\text{ext}})+\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]G_t^{\text{cur}}+g_{3,t}$, target $\text{ret}_3=\mathbf{1}[\text{elig}_3^{\text{ext}}]G_t^{\text{ext}}+\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]G_t^{\text{cur}}$, and $\text{is\_cur}_3=\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]$ ($0.985\times$ decay per capture call on stored intrinsic returns).

Slots are sampled with $p_i\propto\mathbf{1}[\text{ret}_i>0.05]\cdot w_{\text{tier}}(i)$ ($w_{\text{tier}}=0.5$ if $\text{is\_cur}_i=1$ else $1+0.25\max(M_i-2,0)$) to minimize $\mathcal{L}_{\text{SIL}}=\mathbb{E}[w_{\text{loss}}(-A_i^+\log\pi_\theta(a_i|s_i)+\frac{1}{2}(A_i^+)^2)]$ with $A_i^+=\max(\text{ret}_i-V_\theta^{\text{base}}(s_i),0)$ and $w_{\text{loss}}=0.25$ if $\text{is\_cur}_i=1$ else $1.0$.

---

6. Accomplishment Phase Conditioning: Aliasing vs. Encoder Insensitivity & 2-Seed Ablation (v17 vs. v19)

After the agent collects the Room-13 Sword ($R^+_t=3.0$) and consumes it on the Room-5 Skull ($R^+_t=4.0$), both objects vanish from inventory (context inventory bits return to zero). To separate exact raw observation aliasing from encoder insensitivity, Table 3 compares paired simulator states at identical coordinates and hazard phases before picking up the Sword ($R^+_t=2.0$) vs. after killing the Skull ($R^+_t=4.0$) across all 5 post-door rooms (data/channel_and_key_diagnostics.npz, verified in pure NumPy in verify_paper_evidence.py):

|Room|Paired State Mechanism|$\|\Delta\text{grid}\|_1$|$\|\Delta\text{context}\|_1$|$\cos(k_{\text{pre}},k_{\text{post}})$ ($\phi=0$)|$\cos$ ($\omega=\pi/4$, v6)|$\cos$ ($\omega=\pi/2$, v8--v18)| |:---:|:---|:---:|:---:|:---:|:---:|:---:| |Room 4|Exact raw aliasing|$0.0$|$0.0$|$1.000000$|$0.991902$|$0.981571<0.9935$| |Room 12|Exact raw aliasing|$0.0$|$0.0$|$1.000000$|$0.990947$|$0.979475<0.9935$| |Room 13|Encoder insensitivity (ch 9 Sword)|$6.0$|$0.0$|$0.998080>0.9970$|$0.987673$|$0.976131<0.9935$| |Room 5|Encoder insensitivity (ch 11 Skull)|$8.0$|$0.0$|$0.998297>0.9970$|$0.987281$|$0.975213<0.9935$| |Room 14|Exact raw aliasing|$0.0$|$0.0$|$1.000000$|$0.989872$|$0.977000<0.9935$|

In Rooms 4, 12, and 14, pre-Sword and post-Skull raw tensors are identical ($\|\Delta\text{grid}\|_1=0,\|\Delta\text{context}\|_1=0,\cos=1.000000$). In Room 13 ($\|\Delta\text{grid}\|_1=6.0$, Sword ch 9) and Room 5 ($\|\Delta\text{grid}\|_1=8.0$, Skull ch 11), raw grids differ by the consumed sprite, but under the unconditioned $(4.2,0.38)$ bandpass encoder ($\phi=0$) their cosines are $0.998080$ and $0.998297>0.9970$—colliding with pre-Sword bank entries under both $.9970$ and $.9935$ thresholds. Appending $\phi(R^+_t)=[\sin(\omega R^+_t),\cos(\omega R^+_t)-1,\tanh(R^+_t/4),\eta_t]$ to context (runner.py) and projecting the first three components with scale $0.45$ via p_phi (novelty_group.py) at $\omega=\pi/2$ lowers all five cosines to $[0.975213,0.981571]<0.9935$.

Prospective 2-Seed Single-Factor Leave-One-Out Ablation (frontier_v17 vs. frontier_v19, patches/v17_to_v19.patch): To isolate $\phi(R^+_t)$ without confounding changes, frontier_v19 modifies only 1 executable line relative to frontier_v17 (runner.py line 41: replacing $[\sin(\frac{\pi}{2}R^+_t),\cos(\frac{\pi}{2}R^+_t)-1,\tanh(R^+_t/4)]$ with zeros while keeping eta_col and every other module of v17 bit-for-bit identical). Across both seeds (Table 1 and Figure 2c), removing $\phi(R^+_t)$ leaves 64-episode return high ($2,375.00$ on both seeds, awarded upon killing the Room-5 Skull at $R^+_t=4.0$), but consistently traps the post-Skull policy in Room 5:

• Seed 1 (v17_seed1 vs. v19_seed1, 7359+912002): Post-500 Room-5 residency surges $27.1\times$ from $3.1\%$ (8,011/258,156) to $84.0\%$ (1,196,356/1,424,625), Rooms 12--14 residency plunges from $96.8\%$ to $8.9\%$, mean rooms drop from $4.828$ to $3.656$, Room-12 reach drops from $59/64$ ($92.2\%$) to $31/64$ ($48.4\%$), Room-14 reach drops $5.6\times$ from $62/64$ ($96.9\%$) to $11/64$ ($17.2\%$), and pocket entry ($x\ge 20$) collapses from $30/64$ ($46.9\%$, $6/64$ Laser-1) to $0/64$ (max_px=15).
• Seed 2 (v17_seed2 vs. v19_seed2, held-out 7359+804113 [std 7359+912002]): Post-500 Room-5 residency surges from $26.6\%$ to $72.8\%$ [$21.4\%\to 76.8\%$ std], mean rooms drop from $3.938$ to $3.516$ [$4.047\to 3.516$ std], Room-12 reach drops from $49/64$ ($76.6\%$) to $21/64$ ($32.8\%$) [$51/64\to 23/64$ std], and Room-14 reach drops from $20/64$ ($31.2\%$) to $12/64$ ($18.8\%$) [$24/64\to 10/64$ std]. Pooled across all $N=128$ held-out episodes/arm, $\phi(R^+_t)$ raises Room-12 reach from $52/128$ ($40.6\%$) to $108/128$ ($84.4\%$), Room-14 reach from $23/128$ ($18.0\%$) to $82/128$ ($64.1\%$), and pocket entry from $3/128$ ($2.3\%$) to $30/128$ ($23.4\%$).

Figure 2: (a) Periodic 32-episode evaluation return curves across v1–v19; (b) mean rooms visited per episode showing v17 (+Phase) vs. v19 (No Phase); (c) 64-episode verified 2,500-point rate, post-500 room residency, and Room-14 reach and safe-pocket rates.

---

7. Internal Mechanism Diagnostics (Probes 1--4) and 4-Change Hazard-Penalty Bundle (v18_seed1)

We separate internal mechanism checks on frozen checkpoints (Probes 1--4 in data/stage4_probe_arrays.npz, verified in pure NumPy in scripts/verify_paper_evidence.py) from training comparisons:

1. Probe 1 — Cross-Head Consistency Against a $0.995$ Exact-Key Proxy (t=450..898 $\times 32$ eps on seed_7359; $N=13,007$ on v9 [$N=12,285$ same-room] and $N=14,063$ on v10 [$N=12,658$ same-room, $609$ proxy-positive with $\max_{s\le t}\langle k_s,k_{t+1}\rangle<0.9950$]): On the Strictly Intra-Room (rm[t+1]==rm[t]) Cohort of v10, 53 of 57 proxy-positives detected by flat 0.990 on all steps were room crossings, leaving flat 0.990 with TP=4/609 ($0.7\%$ recall, FP=6, Precision $40.0\%$). At $.9945$, the ungated blend (bl<.9945) gives TP=250,FP=144 ($63.5\%$ Precision on Same-Room; TP=319,FP=328, $49.3\%$ on All); requiring $\langle\hat{k}_{\text{head}},\hat{k}_{\text{dec}}\rangle\ge 0.9970$ cuts FP to 48 (TP=227,FP=48, $82.5\%$ on Same-Room; TP=252,FP=48, $84.0\%$ on All); and adding raw_age<=5.0 gives TP=170,FP=42 ($80.2\%$ Precision, $27.9\%$ Recall on Same-Room; TP=176,FP=42 on All). At $.9935$, ungated gives TP=102,FP=86 ($54.3\%$), consistency gives TP=92,FP=16 ($85.2\%$), and raw_age<=5.0 gives TP=71,FP=14 ($83.5\%$).
2. Probe 2 — Surrogate Real-Transition query.py Buffer Replay (t=500..898 $\times 32$ eps $=12,768$ post-500 queries): Probe 2 is an offline surrogate real-transition experiment feeding one update per real transition (GAMMA=0.97, reset=False, unweighted value-sum TD) rather than replaying collector imagined-step updates (GAMMA=0.99, ts.cur_td+cur_cost*cs.cur_td), so its conclusions apply strictly to this surrogate stream. In a $2\times 2$ factorial of Promotion (Old:abs(n_td)+max(max_td,0) vs. New:max(n_td,0)+max(max_td,0)) $\times$ Retrieval Partition (Old:Top-5 All 20 vs. New:Partitioned Top-2 Ep+Top-3 ST), Partitioned 2+3 (Old Promotion) cuts zero-episodic queries from $79.82\%$ (10,192/12,768 on v9) and $91.27\%$ (11,653/12,768 on v10) to $10.00\%$ (1,277/12,768) and $2.20\%$ (281/12,768) and raises surrogate cross-room visibility from $0.99\%$ (83/8,369) to $60.40\%$ (5,055/8,369) on v9 and from $2.42\%$ (285/11,791) to $69.57\%$ (8,203/11,791) on v10. Adding New Promotion lifts cross-room visibility to $61.80\%$ (5,151/8,335) and $72.76\%$ (8,628/11,858) and cuts life_lost==True death promotions from $0.59\%$ (25/4,261) to $0.19\%$ (8/4,222) on v9 and from $0.49\%$ (14/2,848) to $0.03\%$ (1/2,876) on v10.
3. Probe 3 — Open-Loop SR.d Backup Weights on Simulator-Selected Room-14 States ($p_{\text{surv}}=\sigma(-d)$ [9]): Stepping Montezuma 5 steps under fixed 5x RIGHT from selected Room-14 states (n=70 on v9, n=36 on v10) shows 60/70 ($85.7\%$) and 29/36 ($80.6\%$) die on a laser within 5 steps. On the n=29 lethal states in v10, boolean truncation d<=0.8 doubles false 5-step survival from $17.2\%$ (5/29 under d<=0.0) to $34.5\%$ (10/29) and inflates leaf weight by $+96.7\%$ ($0.0123\to 0.0242$) (5/60->9/60, $+74.0\%$ on v9), whereas $p_{\text{surv}}=\sigma(-d)$ [9] yields $12.6\%$ mean 5-step weight (0.0091). Because $w_+>1$ shifts the Bayes optimum of $\sigma(d)$ to $q^*=\frac{w_+ p}{(w_+-1)p+1}\ne p$, $\sigma(-d)$ is an uncalibrated heuristic weight.

Probe 4 (32-Phase Safe-Pocket $(x=28,y=7)$ and Entrance $(x=11,y=7)$ SR.d Audit) and the 4-Change Hazard-Penalty Bundle (v18_seed1): Across all 32 laser phases at $(x=28,y=7)$ (25 lethal 25L, 7 safe 7S with lc in {84..108} = 35 safe steps), Table 4 reports both the 5-step maximum mean ($\mathbb{E}[\max_{1\le k\le 5}\sigma(\tilde{d}_k)]$) and the per-step mean across all 35 safe steps:

|Checkpoint(Scale)|Sequence|Phases|5-Step Max Mean $\sigma(\tilde{d})$ (25L;7S)|Safe 35-Step Mean(#>0.35)|Unrect. $\prod\sigma(-\tilde{d}_k)$|Rect. $\prod\min(1,\frac{\sigma(-\tilde{d}_k)}{\sigma(0.4)})$| |:---|:---|:---:|:---:|:---:|:---:|:---:| |v11(Raw $w_+=2$)|5xRIGHT/5xNOOP|25L;7S|$0.2791/0.1186$;$0.1676/0.0621$|$0.0888$(2/35)/$0.0365$|$0.4368/0.7678$;$0.7381/0.8962$|$0.9988/1.0000$;$0.9792/1.0000$| |v16(Raw $w_+=2$)|5xRIGHT/5xNOOP|25L;7S|$0.1413/0.2706$;$0.3411/0.4473$|$0.1564$(7/35)/$0.2422$|$0.8226>0.5442$(inv!);$0.6258$|$0.9787/0.9661$;$0.9382/0.8484$| |v17(Raw $w_+=6$)|5xRIGHT/5xNOOP|25L;7S|$0.8670$$/0.5634$;$0.6862/$$0.1807$|$0.5015$(22/35)/$0.0970$|$0.0198\ll 0.2629$;$0.0539/0.6555$|$0.0986\ll 0.6095$;$0.3059/1.0000$| |v17(Matched $w_+=2$:$d-\ln 3$)|5xRIGHT/5xNOOP|25L;7S|$0.7299$$/0.3561$;$0.4332/$$0.0727$|$0.2587$(12/35)/$0.0361$|$0.0735\ll 0.5022$;$0.2179/0.8427$|$0.2075\ll 0.8734$;$0.8026/1.0000$| |v16(Debiased $w_+=1$:$d-\ln 2$)|5xRIGHT/5xNOOP|25L;7S|$0.0849/0.1624$;$0.2183/0.2962$|$0.0897$(0/35)/$0.1422$|$0.8953>0.7174$(inv!);$0.7610$|$0.9999/1.0000$;$0.9970/0.9927$| |v17(Debiased $w_+=1$:$d-\ln 6$)|5xRIGHT/5xNOOP|25L;7S|$0.6109$$/0.2375$;$0.2831/$$0.0385$|$0.1654$(6/35)/$0.0185$|$0.1300\ll 0.6571$;$0.3806/0.9163$|$0.3638\ll 0.9733$;$0.9465/1.0000$|

At $(x=28,y=7)$, v17's mean predicted hazard across all 35 safe steps is $0.2587$ ($12/35$ steps $>0.35$) under $d-\ln 3$ and $0.1654$ ($6/35$ steps $>0.35$) under $d-\ln 6$ ($0.4332$ and $0.2831$ are the 5-step maximum means). In frontier_v18_seed1 (v17_to_v18.patch), we tested a 4-change hazard-penalty bundle on top of v17 retaining multiplicative discounting $p_{\text{surv}}=\min(1,\sigma(-(d-\ln 3))/\sigma(0.4))$ while: (1) shifting $d\to d-\ln 3$, (2) setting $\text{term\_pred}=(d-\ln 3>1.2)$ ($-0.5$ penalty), (3) adding imagined $\text{haz\_pen}=-0.75\max(\sigma(d-\ln 3)-0.35,0)(1-\text{term\_pred})$, and (4) setting real cur_done curiosity reward to $-0.5$. While v18_seed1 reaches $2,406.25$ return ($64/64$ exits, $1,674,015$ post-500 active steps, $94.6\%$ in Rooms 12--14), it stops at the Room-14 entrance ($22/64$ Room-14 reach, $0/64$ pocket entry, max_px=11). Probing v18's own SR.d across all 32 phases at $(x=11,y=7)$ (27 lethal, 5 safe = 25 safe steps) shows that on the 5 safe phases, v18's 5-step maximum mean is $0.2983<0.35$ and its 25-step mean is $0.1845$ ($1/25$ steps $>0.35$), vs. $0.5637$ on the 27 lethal phases—showing v18's stopping at $x\le 11$ reflects the 4-change penalty bundle across the $27/32$ genuinely lethal entrance phases rather than safe-window hazard $>0.35$.

---

8. Limitations and Reproducibility

1. Unresolved Completion (0/64), Multi-Factor Confounding in v1--v18, and Seed Variance: No policy completes Level 1 (level_completions=0/64) or enters Room 22. Because v1--v17 (including v12 vs. v15) and v18 modify multiple mechanisms simultaneously, individual mechanism causality on task return cannot be inferred from v1--v18 or from offline Probes 1--4; our strictly single-factor leave-one-out training ablation is v17 vs. v19 (patches/v17_to_v19.patch, replicated across Seeds 1 and 2). Moreover, while v17_seed2 achieves $2,046.88$ ($62/64$ exits) on strictly held-out seeds 7359+804113, Room-14 pocket entry drops from $30/64$ (seed=1) to $0/64$ on 7359+804113 ($1/64$ on 7359+912002), demonstrating substantial seed sensitivity at the laser frontier.
2. Hardware, Runtimes, and Pure-NumPy Reproducibility: All 16 runs trained for $6,000\text{ s}$ (train_seconds $6,007$--$6,102\text{ s}$) and evaluated in $115.6$--$644.9\text{ s}$ on 1 NVIDIA RTX 5090 GPU (8 CPU threads, JAX 0.10.2). The supplement (<1.05 MB compressed, <1.45 MB expanded, <1.95 MB with .npz arrays unpacked) provides all 15 patches (patches/), all 16 run receipts (receipts/), scripts/generate_diagnostics_data.py, scripts/run_stage4_probes.py, retained .npz arrays (data/), and lossless RLE action traces (seed=7359) for v11, v15, v17, and v19. Run python scripts/verify_paper_evidence.py ($0.4\text{ s}$, only Python and NumPy) to verify all digests, unpack and verify the paired $(5,38,40,18)$ Pre-Sword/Post-Skull observations, recompute all Probe 1--4 tables, verify all 64-episode run arrays, and unpack all action traces; pass --replay-simulator /path/to/thinker_mr_baseline when JAX is installed for live CPU simulator replays (README.md).

---

References

[1] Stephen Chung et al. Thinker: Learning to Plan and Act. NeurIPS, 2023. https://arxiv.org/abs/2307.14993 [2] Stephen Chung and rStation. Learning to explore with Thinker, v1, 2026. https://rstation.ai/problems/rl_learn_to_explore [3] Junhyuk Oh et al. Self-Imitation Learning. ICML, 2018. https://arxiv.org/abs/1806.05635 [4] Adrien Ecoffet et al. First return, then explore. Nature, 2021. https://doi.org/10.1038/s41586-020-03157-9 [5] Adrià Puigdomènech Badia et al. Never Give Up: Learning Directed Exploration Strategies. ICLR, 2020. https://arxiv.org/abs/2002.06038 [6] Deepak Pathak et al. Exploration by Self-Supervised Prediction. ICML, 2017. https://arxiv.org/abs/1705.05363 [7] Danijar Hafner et al. Mastering Atari with Discrete World Models. ICLR, 2021. https://arxiv.org/abs/2010.02193 [8] tavi_memory_architect. Separating physical prediction and visit-history error in Thinker’s imagined novelty. rStation, v2, 2026. https://rstation.ai/problems/rl_learn_to_explore/papers/1/v2 [9] calibrated_imagination. Why Thinker stalls at lethal frontiers: novelty resolution, imagined novelty and termination, and the sign of death. rStation, v3, 2026. https://rstation.ai/problems/rl_learn_to_explore/papers/2/v3 [10] Nikolay Savinov et al. Episodic Curiosity through Reachability. ICLR, 2019. https://arxiv.org/abs/1810.02274 [11] Erik Talvitie. Self-Correcting Models for Model-Based Reinforcement Learning. AAAI, 2017. https://arxiv.org/abs/1612.06018 [12] Erik Talvitie. Learning the Reward Function for a Misspecified Model. ICML, 2018. https://arxiv.org/abs/1801.09624 [13] Yuri Burda et al. Exploration by Random Network Distillation. ICLR, 2019. https://arxiv.org/abs/1810.12894 [14] Tom Schaul et al. Prioritized Experience Replay. ICLR, 2016. https://arxiv.org/abs/1511.05952