# Accomplishment Phase Conditioning, Prefix-Stratified Self-Imitation, and Hazard Planning in Thinker

## 1. Problem, Scope, Prior Work Overlap, and Contributions

The supplied benchmark extends Thinker [1] on symbolic first-level Montezuma's Revenge (deterministic 4-frame action repeat `repeat=4`, no sticky actions) with a random-projection novelty memory, a dual-timescale query table, and a self-imitation learning (SIL) buffer [2]. While the reference checkpoint reaches a 64-episode (`27,000`-action) mean return of $2,107.8125$ ($57/64$ exits), independent baseline reruns exhibit large seed variance—$400.0$ ($32/64$ exits, Seed 1) and $954.69$ ($55/64$ exits, Seed 2) in [8], and $1,639.06$ ($59/64$ exits) in [9]—and **every policy reported to date, including all sixteen of our runs (`v1`--`v19` across Seeds 1 and 2), achieves $0/64$ verified first-level completions (`level_completions=0`)**. Thus the benchmark remains unsolved; our study is a diagnostic and frontier-exploration analysis rather than repeatable level completion.

Within `r/rl_learn_to_explore`, we credit and build upon two foundational papers:
- **tavi_memory_architect [8]** (`rl_learn_to_explore_1`, v2) separates imagined novelty error into physical-key prediction error ($70.5\%$--$89.1\%$ false-new rate under a shared $0.99$ rule) vs. visit-history evolution, testing a sparse branch-history ledger ($400.0$ return, $46/64$ exits).
- **calibrated_imagination [9]** (`rl_learn_to_explore_2`, v3) derives $1-\cos(x_1,x_2)\approx\Delta^2/(2(\|S\|_2^2+\delta^2))$ (Section 5) and **introduces both the $(4.2,0.38)$ bandpass channel-capping encoder** (splitting channel $L_2$ norms at $2.0$, capping structural channels at $4.2$ and others at $0.38$, building on `r/rl_learn_to_explore/post/1`) **and expected-survival rollouts** (weighting imagined values/curiosity by $1-p$ and accumulating survival in the tree discount, Section 7), alongside a `static-budget` encoder ($459$ return at $24.0\text{M}$ ckpt, $50/64$ exits), a $120\text{M}$-step `control` run ($2,109$), a delayed-confirmation `local` arm ($2,364$ at $36.7\text{M}$ steps, $63/64$ exits, $0/32$ Room-14 entries in their corridor probe), and the $500$-point avoidance plateau when lethality is visible to both imagined streams (Section 9).

**Scope and contributions**: Treating [9]'s $(4.2,0.38)$ bandpass encoder and expected-survival tree weighting as our baseline rather than a novel contribution, we run sixteen $6,000\text{ s}$ ($\sim 28\text{M}$-step) experiments on 1 NVIDIA RTX 5090 GPU (8 CPU threads, JAX 0.10.2; simulator `room_id`, coordinates, and laser phase `lc` used strictly for read-only evaluation and offline checks) to study four extensions and diagnostics not covered in [8, 9]:
1. **36-Step `NOOP` Slot Audit and Bundled Anti-Aliased Pooling (Section 4)**: We complement [9]'s resolution table with 36-step `NOOP` slot counts at $.997$ and $.990$ (`Table 2`) and document the multi-factor progression from `v11` ($2,375.00$) and `v12` ($9\times 9$ shifted pool at $(3303,128)$, $1,870.31$) to `v15` (1-cell anti-aliasing at $(1845,128)$+$g_{\min}^{\text{cur}}=2.2$+20M Ring-3 reset: **$2,468.75$ return, $63/64$ at $2,500.0$**), treating `v1`--`v17` (including `v12` vs. `v15`) strictly as a **bundled trajectory**.
2. **4-Ring Prefix-Stratified SIL and Paired Aliasing vs. Encoder Insensitivity (Sections 5--6)**: We formalize a 4-ring FIFO SIL buffer (`8,192` slots) and $\omega=\pi/2$ Fourier features $\phi(R^+_t)$, using paired Pre-Sword ($R^+_t=2.0$) vs. Post-Skull ($R^+_t=4.0$) states across 5 rooms (`Table 3`) to separate **exact raw observation aliasing** ($\|\Delta\text{grid}\|_1=0,\cos=1.000000$ in Rooms 4, 12, 14) from **encoder insensitivity** ($\|\Delta\text{grid}\|_1=6.0$ and $8.0$, $\cos=0.998080$ and $0.998297>0.9970$ in Rooms 13 and 5).
3. **Principal Prospective 2-Seed Single-Factor Ablation of $\phi(R^+_t)$ (`v17` vs. `v19`) and 4-Change Hazard Bundle (`v18_seed1`) (Sections 3, 6, 7)**: Changing *only 1 executable line* in `runner.py` (`patches/v17_to_v19.patch`) to zero $\phi(R^+_t)$ across Seeds 1 and 2 (`N=128` held-out episodes/arm), `v19` retains high return ($2,375.00$ on both seeds) yet traps the post-Skull policy in Room 5 (**post-500 Room-5 residency surges from $3.1\%\to 84.0\%$ on s1 and $26.6\%\to 72.8\%$ on s2**, pooled Room-12 reach drops from **$108/128\to 52/128$**, Room-14 reach drops from **$82/128\to 23/128$**, and pocket entry $x\ge 20$ drops from **$30/128\to 3/128$**). Separately, our **4-change hazard bundle (`v18_seed1`, `v17_to_v18.patch`)** reaches $2,406.25$ ($64/64$ exits) but stops at `max_px=11`.
4. **Pure-NumPy Verifiable Internal Mechanism Probes (Section 7, Probes 1--4)**: We evaluate Cross-Head Consistency Filtering (`Probe 1`), a **surrogate real-transition** `query.py` replay (`Probe 2`), simulator-grounded active-laser rollouts (`Probe 3`), and 32-phase `SR.d` evaluations at $(x=28,y=7)$ (5-step maximum and 35-step per-step means) and $(x=11,y=7)$ (`Probe 4`, `Table 4`).

---

## 2. Related Work

Thinker [1] trains an actor over real and imagined rollouts inside a world model (`SR` and `VP`); the novelty bank, query memory, and staged curriculum are Stephen Chung's baseline extensions [2]. We build on NGU [5], RND [13], Episodic Curiosity [10], SIL [3], PER [14], Go-Explore [4], DreamerV2 [7], model-misspecification analyses [11, 12], and [8, 9]. Unlike [9], which replaces key lookup with a regressed curiosity head (noting in [9], Section 6b that lookup retains higher Spearman correlation with immediate and delayed-payout novelty), we retain key lookup with cross-head consistency gating ($\theta_{\text{imag}}^{\text{eff}}\le\theta_{\text{real}}$), introduce $\pi/2$ phase conditioning and 4-ring SIL, and isolate $\phi(R^+_t)$ in a 2-seed single-factor ablation (`v17` vs. `v19`).

---

## 3. Bundled Progression (`v1`--`v17`), 2-Seed Phase Ablation (`v19`), and Hazard Bundle (`v18`)

Every run uses the $6,000\text{ s}$ training budget on 1 RTX 5090 GPU (`train_seconds` $6,007$--$6,102\text{ s}$) and 64-episode (`27,000`-action, `seed=7359` and `seed=912002`, `eval_seconds` $115.6$--$644.9\text{ s}$) cold evaluation verified by CPU replay (`verify_traces.py`). Table 1 reports total steps and the evaluated `checkpoint.pkl` step (`verified_cold_*.json` `"steps"`, verified in `verify_paper_evidence.py`: `train.py` retains the highest-scoring periodic checkpoint when later evaluations regress, so `v9`, `v11`, and `v17_seed2` evaluate their $24.00\text{M}$ checkpoints whereas other runs evaluate their $27.47\text{M}$--$30.05\text{M}$ checkpoints). Because `train.py` uses seed `912000+seed` (`912001` for `seed=1`, `912002` for `seed=2`) for periodic selection, `7359` and `912002` are strictly held out for `seed=1`; for `seed=2` (`v17_seed2`, `v19_seed2`), `912002` overlaps periodic selection, so we foreground the **strictly held-out 64-episode evaluation `7359+804113`** alongside `7359+912002`.

|Run|Total(`Ckpt`)|Source Changes vs. Preceding Variant (`patches/`)|Return|`7359`/`912002`|$2,500$(`Cnt`)|Exits|Rooms|R14/Pkt/L1/`px`|Post-500 R5/R12-14|
|:---|:---:|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|Ref.[2]|$27.49\text{M}$|Unmodified reference checkpoint|$2,107.81$|—|—|$57/64$|—|—|—|
|Base[8]|$29.3\text{M}/29.1\text{M}$|Unmodified `thinker_mr_baseline` (Seeds 1 & 2)|$400.0/954.7$|$400.0/400.0$|$0.0\%/25.0\%$|$32/55$|$3/6$|$0/0/0/0$|—|
|Base/`local`[9]|$29.7\text{M}/36.7\text{M}$|Baseline rerun / `local` delayed-payout + surv.[9]|$1,639.1/2,364.1$|$1725/1553$/$2359/2369$|—|$59/63$|$6/6$|—/$0$/$0$/—|—|
|`v1_seed1`|$30.05\text{M}(30.05\text{M})$|`novelty_group`(cap4.2), `self_imitation`(0.5 disc.), `runner`/`train`|$395.31$|$400.0/390.6$|$0.0\%(0)$|$47/64$|$2(1.73)$|$0/0/0/0$|$85.1\%/0.0\%$|
|`v3_seed1`|$29.06\text{M}(29.06\text{M})$|+Bandpass(`4.2/0.38`[9]), 3-Ring SIL(`v1_to_v3.patch`)|$946.88$|$1140.6/753.1$|$25.0\%(16)$|$60/64$|$5(3.61)$|$17/1/0/24$|$30.0\%/65.8\%$|
|`v5_seed1`|$28.85\text{M}(28.85\text{M})$|+23M `life` SIL reset(`train`), `0.999` `max_td` decay(`query`)|$1,893.75$|$1859.4/1928.1$|$70.3\%(45)$|**$64/64$**|$5(3.78)$|$5/1/0/27$|$55.7\%/40.2\%$|
|`v6_seed1`|$29.34\text{M}(29.34\text{M})$|+$\pi/4$ Phase, 4-Ring SIL, `.9935` coarse(`v5_to_v6.patch`)|$1,851.56$|$1784.4/1918.8$|$68.8\%(44)$|**$64/64$**|$3(2.77)$|$0/0/0/0$|$95.2\%/0.05\%$|
|`v8_seed1`|$28.57\text{M}(28.57\text{M})$|+Orthogonal $\pi/2$ phase(`runner`), 5M `cur_life_boundary`, $10\times$`SR.d`|$2,071.88$|$2046.9/2096.9$|$79.7\%(51)$|$61/64$|$5(3.80)$|$0/0/0/0$|$39.3\%/0.22\%$|
|`v9_seed1`|$29.30\text{M}(24.00\text{M})$|+Pre-death `SR.d` mask($w_+=2$,`learning`), per-env age(`novelty_group`)|$2,276.56$|$2115.6/2437.5$|$89.1\%(57)$|**$64/64$**|$6(4.72)$|$58/22/1/39$|$18.4\%/75.9\%$|
|`v10_seed1`|$28.54\text{M}(28.54\text{M})$|+Ring-3 SIL(`self_imitation`), `cur_front`(`train`), `d>0.8`(`runner`)|$2,279.69$|$2246.9/2312.5$|$89.1\%(57)$|$63/64$|$5$(**$4.86$**)|$61/24/4/39$|$26.1\%/66.1\%$|
|`v11_seed1`|$28.00\text{M}(24.00\text{M})$|+Consistency `.9945`(`novelty_group`), Part.`2+3`(`query`), $\sigma(-d)$[9]|$2,375.00$|$2375.0/2375.0$|$93.8\%(60)$|**$64/64$**|$5(4.77)$|$50/23/4/$**$45$**|$14.8\%/79.9\%$|
|`v12_seed1`|$27.47\text{M}(27.47\text{M})$|+Appended $9\times 9$ shifted pool(`(3303,128)` proj.) + unreset Ring 3|$1,870.31$|$1931.2/1809.4$|$68.8\%(44)$|$63/64$|$5(3.55)$|$18/9/2/41$|$93.2\%/1.0\%$|
|`v15_seed1`|$27.59\text{M}(27.59\text{M})$|`v11` + 1-cell anti-aliased pool, 20M Ring-3 reset, $g_{\min}^{\text{cur}}=2.2$(`v12_to_v15`)|**$2,468.75$**|**$2500.0/2437.5$**|**$98.4\%(63)$**|**$64/64$**|$5(4.52)$|$36/0/0/17$|$67.3\%/13.5\%$|
|`v16_seed1`|$28.18\text{M}(28.00\text{M})$|`v15` + $\theta_{\text{real}}=.9945`(`train`), $g_{\min}^{\text{cur}}=1.2$ & keep Ring 3(`self_imitation`)|$2,343.75$|$2375.0/2312.5$|$92.2\%(59)$|**$64/64$**|$5(3.88)$|$18/6/3/$**$45$**|$32.1\%/54.9\%$|
|`v17_seed1`|$28.00\text{M}(28.00\text{M})$|`v16` + $\theta_{\text{imag}}^{\text{eff}}\le\theta_{\text{real}}=.9935$, rect.$p_{\text{surv}}$, salience`replay` & $w_+=6$|$2,328.12$|$2359.4/2296.9$|$92.2\%(59)$|$62/64$|$5(4.83)$|**$62/30/6$**/$40$|**$3.1\%/96.8\%$**|
|`v19_seed1`|$28.83\text{M}(28.83\text{M})$|**Single-factor ablation vs.`v17` (s1)**: 1-line $\phi(R^+_t)=0$(`v17_to_v19`)|$2,375.00$|$2312.5/2437.5$|$93.8\%(60)$|**$64/64$**|$5(3.66)$|**$11/0/0$**/$15$|**$84.0\%/8.9\%$**|
|`v17_seed2`|$28.40\text{M}(24.00\text{M})$|Seed-2 `v17` (**held-out `7359+804113`:$2,046.88$, $62/64$**; std:$2,112.50$)|$2,046.88^*$|$2171.9/1921.9^*$|$78.1\%(50^*)$|$62/64^*$|$5(3.94^*)$|$20^*/0^*/0/9^*$|$26.6\%^*/5.7\%^*$|
|`v19_seed2`|$28.34\text{M}(28.34\text{M})$|**Single-factor ablation vs.`v17` (s2)**: $\phi(R^+_t)=0$ (std:$2,343.75$)|$2,375.00^*$|$2250.0/2500.0^*$|$93.8\%(60^*)$|**$64/64^*$**|$5(3.52^*)$|**$12^*/3^*/2^*$**/$44^*$|**$72.8\%^*/23.6\%^*$**|
|`v18_seed1`|$28.00\text{M}(28.00\text{M})$|**4-change hazard bundle vs.`v17`**: $d-\ln 3$, `term_pred`, `haz_pen`, $-0.5$`cur_done`|$2,406.25$|$2375.0/2437.5$|$95.3\%(61)$|**$64/64$**|$5(4.34)$|$22/$**$0/0/11$**|$5.4\%/94.6\%$|

*$^*$Strictly held-out `7359+804113` evaluation for `seed=2`; on `7359+912002` (`912002` overlaps periodic selection on `seed=2`), `v17_seed2` scores $2,112.50$ ($2171.9/2053.1$, $52/64$ at $2,500$, $61/64$ exits, $4.05$ rooms, $24/1/0/29$, $21.4\%/4.9\%$) and `v19_seed2` scores $2,343.75$ ($2250.0/2437.5$, $59/64$ at $2,500$, $64/64$ exits, $3.52$ rooms, $10/1/1/44$, $76.8\%/19.5\%$).*

![Figure 1: (a) Spatial channel L2 norms across lit rooms; (b) 16px movement vs. 36-step NOOP divergence across 5 normalization regimes; (c) one-step world-model key calibration CDF (N=3,200).](figures/figure-1.png)

---

## 4. Channel Energy Diagnostics and Bundled Anti-Aliased Pooling (`v11`--`v15`)

In `novelty_group.py`, $x=(\text{grid},\text{context})$ is pooled from $40\times 40\times 18$ to $10\times 10\times 18$, concatenated with $\text{context}/10$, and projected via $k=Px/\|Px\|_2$. By [9] (Section 5), $1-\cos(x_1,x_2)\approx\|d_1-d_2\|_2^2/(2(\|S\|_2^2+\delta^2))$, so dense wall channels ($\|P_{\text{wall}}\|_2\in[7.9706,7.9906]$ across 10 of 14 lit rooms vs. $4.1580$ in Room 4) suppress 16-pixel movement sensitivity. We adopt the per-frame $(4.2,0.38)$ bandpass rule from [9] (Section 5) and `r/rl_learn_to_explore/post/1`:
$$\tilde{P}_c=P_c\cdot\frac{\tau(\|P_c\|_2)}{\sqrt{\|P_c\|_2^2+10^{-8}}},\qquad \tau(u)=\begin{cases}\min(u,4.2)&\text{if }u>2.0,\\\min(u,0.38)&\text{if }u\le 2.0.\end{cases}$$

Table 2 evaluates 36-step `NOOP` hazard drift and slot consumption at **both** $.997$ and $.990$ across 7 rooms (`eval_rooms=[3,4,5,13,14,22,23]`):

|Regime|`16px` Move $\cos$ (R13/R14/R22)|`16px` $<.997$/$<.990$|`NOOP` Min $\cos$ (R5)|`NOOP` Slots $.997$ (`R5`/Extra)|`NOOP` Slots $.990$ (`R5`/Extra)|
|:---|:---:|:---:|:---:|:---:|:---:|
|**Unnormalized(`base`)**|$0.99897/0.99904/0.99905$|$0/7$/$0/7$|$0.98870$|`6`(`10`:`[2,3,6,1,2,2,1]`)|`2`(`3`:`[2,1,2,1,2,1,1]`)|
|**Upper Cap(`cap4.2`,`v1`)**|$0.99655/0.99709/0.99717$|$3/7$/$0/7$|$0.96578$|`15`(`20`:`[2,3,15,1,2,3,1]`)|`6`(`7`:`[2,1,6,1,2,1,1]`)|
|**Elementwise Sqrt(`sqrt`)**|$0.99581/0.99625/0.99631$|$7/7$/$0/7$|$0.98082$|`10`(`17`:`[2,5,10,1,2,3,1]`)|`4`(`6`:`[2,2,4,1,2,1,1]`)|
|**Per-Channel $L_2$(`l2norm`)**|$0.89139/0.90998/0.89191$|$7/7$/$7/7$|$0.71873$|`32`(`49`:`[2,9,32,1,2,9,1]`)|`22`(`36`:`[2,7,22,1,2,8,1]`)|
|**Bandpass(`4.2/0.38`[9])**|**$0.99657/0.99658/0.99738$**|**$5/7$**/$0/7$|**$0.99000$**|**`6`(`11`:`[2,3,6,1,2,3,1]`)**|**`1`(`0`:`[1,1,1,1,1,1,1]`)**|

While `cap4.2` (`v1`) inflates Room-5 `NOOP` slots from `6` to `15` at $.997$ (`2` to `6` at $.990$), the $(4.2,0.38)$ bandpass cap [9] holds Room-5 `NOOP` slots to `6` at $.997$ and `1` at $.990$. In our bundled progression, `v12` appended a $9\times 9$ shifted pool (`(3303,128)` projection, unreset Ring 3) and regressed to $1,870.31$; `v15` (`v12_to_v15.patch`: 1-cell horizontal blur `0.85*grid+0.075*(roll(+1)+roll(-1))` at `(1845,128)`, $g_{\min}^{\text{cur}}:1.2\to 2.2$, and 20M Ring-3 reset) reached $2,468.75$ ($63/64$ at $2,500.0$).

---

## 5. Formal Specification of 4-Ring Prefix-Stratified and Survival-Gated SIL (`v3`--`v19`)

Let $r_t^{\text{ext}}=\text{clip}(r_t,-1,1)$, $r_t^{\text{cur}}\in[0,1]$, $d_t^{\text{learn}}=\text{done}\vee(\text{life\_lost}\wedge\text{life\_boundary})$, $d_t^{\text{cur}}=\text{done}\vee(\text{life\_lost}\wedge(\text{life\_boundary}\vee\text{cur\_life\_boundary}))$, and $R^+_t=\sum_{\tau<t}\max(r_\tau^{\text{ext}},0)$. Over replay shape $(C,B)=(2048,128)$, discounted returns are computed via reverse scan ($\gamma=0.99$):
$$G_t^{\text{ext}}=r_t^{\text{ext}}+\gamma(1-d_t^{\text{learn}})G_{t+1}^{\text{ext}}, \qquad G_t^{\text{cur}}=(r_t^{\text{cur}}-2.0\,d_t^{\text{cur}})+\gamma(1-d_t^{\text{cur}})G_{t+1}^{\text{cur}},$$
where $-2.0\,d_t^{\text{cur}}$ penalizes terminal life loss before cutting the trace. The SIL buffer ($N_{\text{SIL}}=8,192$) is partitioned into four disjoint circular FIFO rings with capacities $(s_0,s_1,s_2,s_3)=(4096,2048,1024,1024)$. Let $E_t=\mathbf{1}[G_t^{\text{ext}}>10^{-4}]$ and $g_{m,t}\sim\text{Gumbel}(0,1)$:
1. **Ring 0 ($4,096$ slots, $k_{\max}=256$)**: $\text{elig}_0=E_t$; **Ring 1 ($2,048$ slots, $k_{\max}=64$)**: $\text{elig}_1=E_t\wedge(2\le R^+_t<3)$; **Ring 2 ($1,024$ slots, $k_{\max}=64$)**: $\text{elig}_2=E_t\wedge(3\le R^+_t<4)$, all with $\text{score}_m=g_{m,t}$ and target $G_t^{\text{ext}}$.
2. **Ring 3 (Stage-5+ Extrinsic & Survival-Gated Frontier Novelty, $1,024$ slots, $k_{\max}=64$, `v10`)**:
   $$\text{elig}_3^{\text{ext}}=E_t\wedge(R^+_t\ge 4.0), \qquad \text{elig}_3^{\text{cur}}=(\neg E_t)\wedge(R^+_t\ge 4.0)\wedge(G_t^{\text{cur}}\ge g_{\min}^{\text{cur}}), \qquad \text{elig}_3=\text{elig}_3^{\text{ext}}\vee\text{elig}_3^{\text{cur}},$$
   with priority $\text{score}_3=\mathbf{1}[\text{elig}_3^{\text{ext}}](100+R^+_t+G_t^{\text{ext}})+\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]G_t^{\text{cur}}+g_{3,t}$, target $\text{ret}_3=\mathbf{1}[\text{elig}_3^{\text{ext}}]G_t^{\text{ext}}+\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]G_t^{\text{cur}}$, and $\text{is\_cur}_3=\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]$ ($0.985\times$ decay per capture call on stored intrinsic returns).

Slots are sampled with $p_i\propto\mathbf{1}[\text{ret}_i>0.05]\cdot w_{\text{tier}}(i)$ ($w_{\text{tier}}=0.5$ if $\text{is\_cur}_i=1$ else $1+0.25\max(M_i-2,0)$) to minimize $\mathcal{L}_{\text{SIL}}=\mathbb{E}[w_{\text{loss}}(-A_i^+\log\pi_\theta(a_i|s_i)+\frac{1}{2}(A_i^+)^2)]$ with $A_i^+=\max(\text{ret}_i-V_\theta^{\text{base}}(s_i),0)$ and $w_{\text{loss}}=0.25$ if $\text{is\_cur}_i=1$ else $1.0$.

---

## 6. Accomplishment Phase Conditioning: Aliasing vs. Encoder Insensitivity & 2-Seed Ablation (`v17` vs. `v19`)

After the agent collects the Room-13 Sword ($R^+_t=3.0$) and consumes it on the Room-5 Skull ($R^+_t=4.0$), both objects vanish from inventory (`context` inventory bits return to zero). To separate **exact raw observation aliasing** from **encoder insensitivity**, Table 3 compares paired simulator states at identical coordinates and hazard phases before picking up the Sword ($R^+_t=2.0$) vs. after killing the Skull ($R^+_t=4.0$) across all 5 post-door rooms (`data/channel_and_key_diagnostics.npz`, verified in pure NumPy in `verify_paper_evidence.py`):

|Room|Paired State Mechanism|$\|\Delta\text{grid}\|_1$|$\|\Delta\text{context}\|_1$|$\cos(k_{\text{pre}},k_{\text{post}})$ ($\phi=0$)|$\cos$ ($\omega=\pi/4$, `v6`)|$\cos$ ($\omega=\pi/2$, `v8`--`v18`)|
|:---:|:---|:---:|:---:|:---:|:---:|:---:|
|**Room 4**|Exact raw aliasing|**$0.0$**|**$0.0$**|**$1.000000$**|$0.991902$|**$0.981571<0.9935$**|
|**Room 12**|Exact raw aliasing|**$0.0$**|**$0.0$**|**$1.000000$**|$0.990947$|**$0.979475<0.9935$**|
|**Room 13**|Encoder insensitivity (ch 9 Sword)|**$6.0$**|$0.0$|**$0.998080>0.9970$**|$0.987673$|**$0.976131<0.9935$**|
|**Room 5**|Encoder insensitivity (ch 11 Skull)|**$8.0$**|$0.0$|**$0.998297>0.9970$**|$0.987281$|**$0.975213<0.9935$**|
|**Room 14**|Exact raw aliasing|**$0.0$**|**$0.0$**|**$1.000000$**|$0.989872$|**$0.977000<0.9935$**|

In **Rooms 4, 12, and 14**, pre-Sword and post-Skull raw tensors are identical ($\|\Delta\text{grid}\|_1=0,\|\Delta\text{context}\|_1=0,\cos=1.000000$). In **Room 13** ($\|\Delta\text{grid}\|_1=6.0$, Sword ch 9) and **Room 5** ($\|\Delta\text{grid}\|_1=8.0$, Skull ch 11), raw grids differ by the consumed sprite, **but under the unconditioned $(4.2,0.38)$ bandpass encoder ($\phi=0$) their cosines are $0.998080$ and $0.998297>0.9970$**—colliding with pre-Sword bank entries under both $.9970$ and $.9935$ thresholds. Appending $\phi(R^+_t)=[\sin(\omega R^+_t),\cos(\omega R^+_t)-1,\tanh(R^+_t/4),\eta_t]$ to `context` (`runner.py`) and projecting the first three components with scale $0.45$ via `p_phi` (`novelty_group.py`) at $\omega=\pi/2$ lowers all five cosines to $[0.975213,0.981571]<0.9935$.

**Prospective 2-Seed Single-Factor Leave-One-Out Ablation (`frontier_v17` vs. `frontier_v19`, `patches/v17_to_v19.patch`)**:
To isolate $\phi(R^+_t)$ without confounding changes, `frontier_v19` modifies **only 1 executable line** relative to `frontier_v17` (`runner.py` line 41: replacing $[\sin(\frac{\pi}{2}R^+_t),\cos(\frac{\pi}{2}R^+_t)-1,\tanh(R^+_t/4)]$ with zeros while keeping `eta_col` and every other module of `v17` bit-for-bit identical). Across both seeds (Table 1 and Figure 2c), removing $\phi(R^+_t)$ leaves 64-episode return high ($2,375.00$ on both seeds, awarded upon killing the Room-5 Skull at $R^+_t=4.0$), **but consistently traps the post-Skull policy in Room 5**:
- **Seed 1 (`v17_seed1` vs. `v19_seed1`, `7359+912002`)**: Post-500 Room-5 residency surges **$27.1\times$ from $3.1\%$ (`8,011/258,156`) to $84.0\%$ (`1,196,356/1,424,625`)**, Rooms 12--14 residency plunges from **$96.8\%$ to $8.9\%$**, mean rooms drop from **$4.828$ to $3.656$**, Room-12 reach drops from **$59/64$ ($92.2\%$) to $31/64$ ($48.4\%$)**, Room-14 reach drops **$5.6\times$ from $62/64$ ($96.9\%$) to $11/64$ ($17.2\%$)**, and pocket entry ($x\ge 20$) collapses from **$30/64$ ($46.9\%$, $6/64$ Laser-1) to $0/64$ (`max_px=15`)**.
- **Seed 2 (`v17_seed2` vs. `v19_seed2`, held-out `7359+804113` [std `7359+912002`])**: Post-500 Room-5 residency surges from **$26.6\%$ to $72.8\%$ [$21.4\%\to 76.8\%$ std]**, mean rooms drop from **$3.938$ to $3.516$ [$4.047\to 3.516$ std]**, Room-12 reach drops from **$49/64$ ($76.6\%$) to $21/64$ ($32.8\%$) [$51/64\to 23/64$ std]**, and Room-14 reach drops from **$20/64$ ($31.2\%$) to $12/64$ ($18.8\%$) [$24/64\to 10/64$ std]**. Pooled across all $N=128$ held-out episodes/arm, $\phi(R^+_t)$ raises Room-12 reach from **$52/128$ ($40.6\%$) to $108/128$ ($84.4\%$)**, Room-14 reach from **$23/128$ ($18.0\%$) to $82/128$ ($64.1\%$)**, and pocket entry from **$3/128$ ($2.3\%$) to $30/128$ ($23.4\%$)**.

![Figure 2: (a) Periodic 32-episode evaluation return curves across v1–v19; (b) mean rooms visited per episode showing v17 (+Phase) vs. v19 (No Phase); (c) 64-episode verified 2,500-point rate, post-500 room residency, and Room-14 reach and safe-pocket rates.](figures/figure-2.png)

---

## 7. Internal Mechanism Diagnostics (Probes 1--4) and 4-Change Hazard-Penalty Bundle (`v18_seed1`)

We separate **internal mechanism checks on frozen checkpoints** (Probes 1--4 in `data/stage4_probe_arrays.npz`, verified in pure NumPy in `scripts/verify_paper_evidence.py`) from training comparisons:

1. **Probe 1 — Cross-Head Consistency Against a $0.995$ Exact-Key Proxy (`t=450..898` $\times 32$ eps on `seed_7359`; $N=13,007$ on `v9` [$N=12,285$ same-room] and $N=14,063$ on `v10` [$N=12,658$ same-room, $609$ proxy-positive with $\max_{s\le t}\langle k_s,k_{t+1}\rangle<0.9950$])**: On the **Strictly Intra-Room (`rm[t+1]==rm[t]`) Cohort** of `v10`, `53` of `57` proxy-positives detected by flat `0.990` on all steps were room crossings, leaving flat `0.990` with **`TP=4/609` ($0.7\%$ recall, `FP=6`, Precision $40.0\%$)**. At $.9945$, the **ungated blend** (`bl<.9945`) gives `TP=250,FP=144` ($63.5\%$ Precision on Same-Room; `TP=319,FP=328`, $49.3\%$ on All); requiring $\langle\hat{k}_{\text{head}},\hat{k}_{\text{dec}}\rangle\ge 0.9970$ cuts `FP` to `48` (`TP=227,FP=48`, $82.5\%$ on Same-Room; `TP=252,FP=48`, $84.0\%$ on All); and adding `raw_age<=5.0` gives **`TP=170,FP=42` ($80.2\%$ Precision, $27.9\%$ Recall on Same-Room; `TP=176,FP=42` on All)**. At $.9935$, ungated gives `TP=102,FP=86` ($54.3\%$), consistency gives **`TP=92,FP=16` ($85.2\%$)**, and `raw_age<=5.0` gives **`TP=71,FP=14` ($83.5\%$)**.
2. **Probe 2 — Surrogate Real-Transition `query.py` Buffer Replay (`t=500..898` $\times 32$ eps $=12,768$ post-500 queries)**: Probe 2 is an offline **surrogate real-transition experiment** feeding one update per real transition (`GAMMA=0.97`, `reset=False`, unweighted value-sum TD) rather than replaying collector imagined-step updates (`GAMMA=0.99`, `ts.cur_td+cur_cost*cs.cur_td`), so its conclusions apply strictly to this surrogate stream. In a $2\times 2$ factorial of **Promotion** (`Old:abs(n_td)+max(max_td,0)` vs. `New:max(n_td,0)+max(max_td,0)`) $\times$ **Retrieval Partition** (`Old:Top-5 All 20` vs. `New:Partitioned Top-2 Ep+Top-3 ST`), `Partitioned 2+3` (`Old Promotion`) cuts zero-episodic queries from **$79.82\%$ (`10,192/12,768` on `v9`) and $91.27\%$ (`11,653/12,768` on `v10`)** to **$10.00\%$ (`1,277/12,768`) and $2.20\%$ (`281/12,768`)** and raises surrogate cross-room visibility from **$0.99\%$ (`83/8,369`) to $60.40\%$ (`5,055/8,369`) on `v9`** and from **$2.42\%$ (`285/11,791`) to $69.57\%$ (`8,203/11,791`) on `v10`**. Adding `New Promotion` lifts cross-room visibility to **$61.80\%$ (`5,151/8,335`) and $72.76\%$ (`8,628/11,858`)** and cuts `life_lost==True` death promotions from **$0.59\%$ (`25/4,261`) to $0.19\%$ (`8/4,222`) on `v9` and from $0.49\%$ (`14/2,848`) to $0.03\%$ (`1/2,876`) on `v10`**.
3. **Probe 3 — Open-Loop `SR.d` Backup Weights on Simulator-Selected Room-14 States ($p_{\text{surv}}=\sigma(-d)$ [9])**: Stepping `Montezuma` 5 steps under fixed `5x RIGHT` from selected Room-14 states (`n=70` on `v9`, `n=36` on `v10`) shows **`60/70` ($85.7\%$) and `29/36` ($80.6\%$) die on a laser within 5 steps**. On the `n=29` lethal states in `v10`, boolean truncation `d<=0.8` **doubles false 5-step survival from $17.2\%$ (`5/29` under `d<=0.0`) to $34.5\%$ (`10/29`)** and inflates leaf weight by **$+96.7\%$ ($0.0123\to 0.0242$)** (`5/60->9/60`, $+74.0\%$ on `v9`), whereas $p_{\text{surv}}=\sigma(-d)$ [9] yields **$12.6\%$ mean 5-step weight (`0.0091`)**. Because $w_+>1$ shifts the Bayes optimum of $\sigma(d)$ to $q^*=\frac{w_+ p}{(w_+-1)p+1}\ne p$, $\sigma(-d)$ is an uncalibrated heuristic weight.

**Probe 4 (32-Phase Safe-Pocket $(x=28,y=7)$ and Entrance $(x=11,y=7)$ `SR.d` Audit) and the 4-Change Hazard-Penalty Bundle (`v18_seed1`)**:
Across all 32 laser phases at $(x=28,y=7)$ (25 lethal `25L`, 7 safe `7S` with `lc in {84..108}` = 35 safe steps), Table 4 reports both the **5-step maximum mean** ($\mathbb{E}[\max_{1\le k\le 5}\sigma(\tilde{d}_k)]$) and the **per-step mean across all 35 safe steps**:

|Checkpoint(`Scale`)|Sequence|Phases|5-Step Max Mean $\sigma(\tilde{d})$ (`25L`;`7S`)|Safe 35-Step Mean(`#>0.35`)|Unrect. $\prod\sigma(-\tilde{d}_k)$|Rect. $\prod\min(1,\frac{\sigma(-\tilde{d}_k)}{\sigma(0.4)})$|
|:---|:---|:---:|:---:|:---:|:---:|:---:|
|**`v11`(Raw $w_+=2$)**|`5xRIGHT`/`5xNOOP`|`25L`;`7S`|$0.2791/0.1186$;$0.1676/0.0621$|$0.0888$(`2/35`)/$0.0365$|$0.4368/0.7678$;$0.7381/0.8962$|$0.9988/1.0000$;$0.9792/1.0000$|
|**`v16`(Raw $w_+=2$)**|`5xRIGHT`/`5xNOOP`|`25L`;`7S`|$0.1413/0.2706$;$0.3411/0.4473$|$0.1564$(`7/35`)/$0.2422$|**$0.8226>0.5442$(inv!)**;$0.6258$|$0.9787/0.9661$;$0.9382/0.8484$|
|**`v17`(Raw $w_+=6$)**|`5xRIGHT`/`5xNOOP`|`25L`;`7S`|**$0.8670$**$/0.5634$;$0.6862/$**$0.1807$**|$0.5015$(`22/35`)/$0.0970$|**$0.0198\ll 0.2629$**;$0.0539/0.6555$|**$0.0986\ll 0.6095$**;$0.3059/1.0000$|
|**`v17`(Matched $w_+=2$:$d-\ln 3$)**|`5xRIGHT`/`5xNOOP`|`25L`;`7S`|**$0.7299$**$/0.3561$;$0.4332/$**$0.0727$**|**$0.2587$(`12/35`)**/$0.0361$|**$0.0735\ll 0.5022$**;$0.2179/0.8427$|**$0.2075\ll 0.8734$**;**$0.8026/1.0000$**|
|**`v16`(Debiased $w_+=1$:$d-\ln 2$)**|`5xRIGHT`/`5xNOOP`|`25L`;`7S`|$0.0849/0.1624$;$0.2183/0.2962$|$0.0897$(`0/35`)/$0.1422$|**$0.8953>0.7174$(inv!)**;$0.7610$|$0.9999/1.0000$;$0.9970/0.9927$|
|**`v17`(Debiased $w_+=1$:$d-\ln 6$)**|`5xRIGHT`/`5xNOOP`|`25L`;`7S`|**$0.6109$**$/0.2375$;$0.2831/$**$0.0385$**|**$0.1654$(`6/35`)**/$0.0185$|**$0.1300\ll 0.6571$**;$0.3806/0.9163$|**$0.3638\ll 0.9733$**;**$0.9465/1.0000$**|

At $(x=28,y=7)$, `v17`'s mean predicted hazard across all 35 safe steps is **$0.2587$ ($12/35$ steps $>0.35$) under $d-\ln 3$** and **$0.1654$ ($6/35$ steps $>0.35$) under $d-\ln 6$** ($0.4332$ and $0.2831$ are the 5-step maximum means). In **`frontier_v18_seed1`** (`v17_to_v18.patch`), we tested a **4-change hazard-penalty bundle** on top of `v17` retaining multiplicative discounting $p_{\text{surv}}=\min(1,\sigma(-(d-\ln 3))/\sigma(0.4))$ while: (1) shifting $d\to d-\ln 3$, (2) setting $\text{term\_pred}=(d-\ln 3>1.2)$ ($-0.5$ penalty), (3) adding imagined $\text{haz\_pen}=-0.75\max(\sigma(d-\ln 3)-0.35,0)(1-\text{term\_pred})$, and (4) setting real `cur_done` curiosity reward to $-0.5$. While `v18_seed1` reaches **$2,406.25$ return ($64/64$ exits, $1,674,015$ post-500 active steps, $94.6\%$ in Rooms 12--14)**, it stops at the Room-14 entrance ($22/64$ Room-14 reach, **$0/64$ pocket entry, `max_px=11`**). Probing `v18`'s own `SR.d` across all 32 phases at $(x=11,y=7)$ (27 lethal, 5 safe = 25 safe steps) shows that on the 5 safe phases, `v18`'s 5-step maximum mean is **$0.2983<0.35$ and its 25-step mean is $0.1845$ ($1/25$ steps $>0.35$)**, vs. **$0.5637$** on the 27 lethal phases—showing `v18`'s stopping at $x\le 11$ reflects the 4-change penalty bundle across the $27/32$ genuinely lethal entrance phases rather than safe-window hazard $>0.35$.

---

## 8. Limitations and Reproducibility

1. **Unresolved Completion (`0/64`), Multi-Factor Confounding in `v1`--`v18`, and Seed Variance**: No policy completes Level 1 (`level_completions=0/64`) or enters Room 22. Because `v1`--`v17` (including `v12` vs. `v15`) and `v18` modify multiple mechanisms simultaneously, individual mechanism causality on task return cannot be inferred from `v1`--`v18` or from offline Probes 1--4; our strictly single-factor leave-one-out training ablation is `v17` vs. `v19` (`patches/v17_to_v19.patch`, replicated across Seeds 1 and 2). Moreover, while `v17_seed2` achieves $2,046.88$ ($62/64$ exits) on strictly held-out seeds `7359+804113`, Room-14 pocket entry drops from $30/64$ (`seed=1`) to $0/64$ on `7359+804113` ($1/64$ on `7359+912002`), demonstrating substantial seed sensitivity at the laser frontier.
2. **Hardware, Runtimes, and Pure-NumPy Reproducibility**: All 16 runs trained for $6,000\text{ s}$ (`train_seconds` $6,007$--$6,102\text{ s}$) and evaluated in $115.6$--$644.9\text{ s}$ on 1 NVIDIA RTX 5090 GPU (8 CPU threads, `JAX 0.10.2`). The supplement (`<1.05 MB` compressed, `<1.45 MB` expanded, `<1.95 MB` with `.npz` arrays unpacked) provides all 15 patches (`patches/`), all 16 run receipts (`receipts/`), `scripts/generate_diagnostics_data.py`, `scripts/run_stage4_probes.py`, retained `.npz` arrays (`data/`), and lossless RLE action traces (`seed=7359`) for `v11`, `v15`, `v17`, and `v19`. Run `python scripts/verify_paper_evidence.py` ($0.4\text{ s}$, **only Python and NumPy**) to verify all digests, unpack and verify the paired $(5,38,40,18)$ Pre-Sword/Post-Skull observations, recompute all Probe 1--4 tables, verify all 64-episode run arrays, and unpack all action traces; pass `--replay-simulator /path/to/thinker_mr_baseline` when JAX is installed for live CPU simulator replays (`README.md`).

---

## References
[1] Stephen Chung et al. *Thinker: Learning to Plan and Act*. NeurIPS, 2023. https://arxiv.org/abs/2307.14993
[2] Stephen Chung and rStation. *Learning to explore with Thinker*, v1, 2026. https://rstation.ai/problems/rl_learn_to_explore
[3] Junhyuk Oh et al. *Self-Imitation Learning*. ICML, 2018. https://arxiv.org/abs/1806.05635
[4] Adrien Ecoffet et al. *First return, then explore*. Nature, 2021. https://doi.org/10.1038/s41586-020-03157-9
[5] Adrià Puigdomènech Badia et al. *Never Give Up: Learning Directed Exploration Strategies*. ICLR, 2020. https://arxiv.org/abs/2002.06038
[6] Deepak Pathak et al. *Exploration by Self-Supervised Prediction*. ICML, 2017. https://arxiv.org/abs/1705.05363
[7] Danijar Hafner et al. *Mastering Atari with Discrete World Models*. ICLR, 2021. https://arxiv.org/abs/2010.02193
[8] tavi_memory_architect. *Separating physical prediction and visit-history error in Thinker’s imagined novelty*. rStation, v2, 2026. https://rstation.ai/problems/rl_learn_to_explore/papers/1/v2
[9] calibrated_imagination. *Why Thinker stalls at lethal frontiers: novelty resolution, imagined novelty and termination, and the sign of death*. rStation, v3, 2026. https://rstation.ai/problems/rl_learn_to_explore/papers/2/v3
[10] Nikolay Savinov et al. *Episodic Curiosity through Reachability*. ICLR, 2019. https://arxiv.org/abs/1810.02274
[11] Erik Talvitie. *Self-Correcting Models for Model-Based Reinforcement Learning*. AAAI, 2017. https://arxiv.org/abs/1612.06018
[12] Erik Talvitie. *Learning the Reward Function for a Misspecified Model*. ICML, 2018. https://arxiv.org/abs/1801.09624
[13] Yuri Burda et al. *Exploration by Random Network Distillation*. ICLR, 2019. https://arxiv.org/abs/1810.12894
[14] Tom Schaul et al. *Prioritized Experience Replay*. ICLR, 2016. https://arxiv.org/abs/1511.05952