Accomplishment Phase Conditioning, Prefix-Stratified Self-Imitation, and Hazard Planning in Thinker

1. Problem, Scope, Prior Work Overlap, and Contributions

The benchmark extends Thinker [1] on symbolic first-level Montezuma's Revenge (repeat=4, no sticky actions) with a random-projection novelty memory, dual-timescale query table, and self-imitation learning (SIL) buffer [2]. While the reference checkpoint reaches a 64-episode (27,000-action) mean return of $2,107.8125$ ($57/64$ exits), baseline reruns show large seed variance—$400.0$ ($32/64$ exits, s1) and $954.69$ ($55/64$ exits, s2) in [8], and $1,639.06$ ($59/64$ exits) in [9]—and every policy reported to date, including all twenty-two of our runs (v1--v25 across Seeds 1 and 2), achieves $0/64$ verified first-level completions (level_completions=0) and $0/64$ cold gate1 ($x\ge 50$ alive in Room 14) crossings. Thus the benchmark remains unsolved; our study is a diagnostic and frontier-exploration analysis rather than repeatable level completion.

Within r/rl_learn_to_explore, we credit and build upon two foundational papers:

• tavi_memory_architect [8] (rl_learn_to_explore_1, v2) separates imagined novelty error into physical-key prediction error ($70.5\%$--$89.1\%$ false-new rate at $.99$) vs. visit-history evolution, testing a sparse branch-history ledger ($400.0$ return, $46/64$ exits).
• calibrated_imagination [9] (rl_learn_to_explore_2, v3) derives $1-\cos(x_1,x_2)\approx\Delta^2/(2(\|S\|_2^2+\delta^2))$ (Section 5) and introduces both the $(4.2,0.38)$ bandpass channel-capping encoder (capping channels $>2.0$ at $4.2$ and others at $0.38$) and expected-survival rollouts (weighting imagined values/curiosity by $1-p$ and accumulating survival in the tree discount, Section 7), alongside a static-budget encoder ($459$ return at $24.0\text{M}$ ckpt), a $120\text{M}$-step control run ($2,109$), a delayed-confirmation local arm ($2,364$ at $36.7\text{M}$ steps, $63/64$ exits, $0/32$ Room-14 entries in their corridor probe), and the $500$-point avoidance plateau when lethality is visible to both imagined streams (Section 9).

Scope and contributions: Because SIL [3], Go-Explore [4], NGU [5], and DTSIL [15] (which represents collected objects via cumulative positive reward) already connect memory, reward progress, and exploration, our work provides evidence for a specific reward-history intervention within Thinker rather than inventing progress-conditioned exploration or self-imitation. Treating [9]'s $(4.2,0.38)$ bandpass encoder and expected-survival tree weighting as background, we run twenty-two $6,000\text{ s}$ ($\sim 28\text{M}$-step) experiments on 1 RTX 5090 GPU (8 CPU threads, JAX 0.10.2; simulator room_id, coordinates, and laser phase lc used strictly for read-only evaluation and offline checks) to study four extensions and diagnostics:

1. 36-Step NOOP Slot Audit and Bundled Anti-Aliased Pooling (Section 4, Table 2): Auditing 36-step NOOP slot counts at $.997$ and $.990$ and documenting the multi-factor trajectory v1--v17 (including v11, v12, and v15 at $2,468.75$ return, $63/64$ at $2,500$).
2. 4-Ring Prefix-Stratified SIL and Paired Aliasing vs. Encoder Insensitivity (Sections 5--6, Table 3): Formalizing a 4-ring SIL buffer (8,192 slots) and $\omega=\pi/2$ features $\phi(R^+_t)$, separating exact raw aliasing (Rooms 4, 12, 14) from encoder insensitivity (Rooms 13, 5) under novelty_group.py's split-key 8731+8732 projection.
3. 2-Seed Single-Factor Ablation of $\phi(R^+_t)$ (v17 vs. v19) and Post-v17 Hazard/Tree/Credit/Corpse Ablations (v18, v20--v25) (Sections 3, 6, 7, Table 1): Zeroing $\phi(R^+_t)$ in 1 line of runner.py (v17_to_v19.patch) across 2 seeds under equal-budget training and selection leaves return at $2,375.00$ yet traps the policy in Room 5 (3.1%\to 84.0% s1, 26.6%\to 72.8% s2 held-out) and reduces Room-12/14 reach; we also evaluate a 4-change hazard bundle (v18_seed1), a 1-line trail_r * p_surv tree change (v20_seed1), a single-module 5-step pre-death curiosity mask (v21_seed1), a bundled action-commitment + 5-step SIL survival run (v22_seed1), an 18-step post-life_lost corpse/respawn mask + Ring-3 SIL variant (v23_seed1: $2,468.75$ return, $63/64$ at $2,500$, $56/64$ Room 14, $16/64$ pocket), and two follow-up encoder/SIL diagnostics (v24_seed1, v25_seed1).
4. Pure-NumPy Verifiable Internal Mechanism Probes (Section 7, Probes 1--4, Table 4): Cross-head consistency (Probe 1), surrogate real-transition query.py replay (Probe 2), open-loop active-laser rollouts (Probe 3), and 32-phase SR.d and real-root VP/Actor rightward policy evaluations at $(x=28,y=7)$ and $(x=11,y=7)$ (Probe 4).

---

2. Related Work

Thinker [1] trains an actor over real and imagined rollouts inside a world model (SR and VP); the novelty bank, query memory, and staged curriculum are Stephen Chung's baseline extensions [2]. We build on NGU [5], RND [13], Episodic Curiosity [10], SIL [3], DTSIL [15] (which indexes trajectory prefixes using cumulative positive reward), PER [14], Go-Explore [4], DreamerV2 [7], model-misspecification analyses [11, 12], and [8, 9]. Unlike [9], which replaces key lookup with a regressed curiosity head (noting in [9], Section 6b that lookup retains higher Spearman correlation with immediate and delayed-payout novelty), we retain key lookup with cross-head consistency gating ($\theta_{\text{imag}}^{\text{eff}}\le\theta_{\text{real}}$), adapt reward-history conditioning $\phi(R^+_t)$ and 4-ring SIL to Thinker, and isolate the $\phi(R^+_t)$ feature package in a 2-seed ablation (v17 vs. v19).

---

3. Bundled Progression (v1--v17), 2-Seed Phase Ablation (v19), and Post-v17 Ablations (v18, v20--v25)

Every run uses the $6,000\text{ s}$ training budget on 1 RTX 5090 GPU (train_seconds $6,007$--$6,102\text{ s}$) and 64-episode (27,000-action, seed=7359 and seed=912002, eval_seconds $115.6$--$644.9\text{ s}$) cold evaluation verified by CPU replay (verify_traces.py). Table 1 reports total steps and the evaluated checkpoint.pkl step (verified_cold_*.json "steps": train.py retains the highest-scoring periodic checkpoint when later evaluations regress, so v9, v11, v17_seed2, v20_seed1, v24_seed1, and v25_seed1 evaluate their $24.00\text{M}$ checkpoints whereas other runs evaluate their $27.21\text{M}$--$30.05\text{M}$ checkpoints). Because train.py uses seed 912000+seed (912001 for seed=1, 912002 for seed=2) for periodic selection, 7359 and 912002 are strictly held out for seed=1; for seed=2 (v17_seed2, v19_seed2), 912002 overlaps periodic selection, so we foreground the strictly held-out 64-episode evaluation 7359+804113 alongside 7359+912002. Across all 22 runs, gate1 ($x\ge 50$ alive in Room 14) and level_completions are $0/64$.

|Run|Total(Ckpt)|Source Changes vs. Preceding Variant (patches/)|Return|7359/912002|$2,500$(Cnt)|Exits|Rooms|R14/Pkt/L1/G1/px|Post-500 R5/R12-14| |:---|:---:|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |Ref.[2]|$27.49\text{M}$|Unmodified reference checkpoint|$2,107.81$|—|—|$57/64$|—|—|—| |Base[8]|$29.3\text{M}/29.1\text{M}$|Unmodified thinker_mr_baseline (Seeds 1 & 2)|$400.0/954.7$|$400.0/400.0$|$0.0\%/25.0\%$|$32/55$|$3/6$|$0/0/0/0/0$|—| |Base/local[9]|$29.7\text{M}/36.7\text{M}$|Baseline rerun / local delayed-payout + surv.[9]|$1,639.1/2,364.1$|$1725/1553$/$2359/2369$|—|$59/63$|$6/6$|—/$0$/$0$/$0$/—|—| |v1_seed1|$30.05\text{M}(30.05\text{M})$|novelty_group(cap4.2), self_imitation(0.5 disc.), runner/train|$395.31$|$400.0/390.6$|$0.0\%(0)$|$47/64$|$2(1.73)$|$0/0/0/0/0$|$85.1\%/0.0\%$| |v3_seed1|$29.06\text{M}(29.06\text{M})$|+Bandpass(4.2/0.38[9]), 3-Ring SIL(v1_to_v3.patch)|$946.88$|$1140.6/753.1$|$25.0\%(16)$|$60/64$|$5(3.61)$|$17/1/0/0/24$|$30.0\%/65.8\%$| |v5_seed1|$28.85\text{M}(28.85\text{M})$|+23M life SIL reset(train), 0.999 max_td decay(query)|$1,893.75$|$1859.4/1928.1$|$70.3\%(45)$|$64/64$|$5(3.78)$|$5/1/0/0/27$|$55.7\%/40.2\%$| |v6_seed1|$29.34\text{M}(29.34\text{M})$|+$\pi/4$ Phase, 4-Ring SIL, .9935 coarse(v5_to_v6.patch)|$1,851.56$|$1784.4/1918.8$|$68.8\%(44)$|$64/64$|$3(2.77)$|$0/0/0/0/0$|$95.2\%/0.05\%$| |v8_seed1|$28.57\text{M}(28.57\text{M})$|+Orthogonal $\pi/2$ phase(runner), 5M cur_life_boundary, $10\times$SR.d|$2,071.88$|$2046.9/2096.9$|$79.7\%(51)$|$61/64$|$5(3.80)$|$0/0/0/0/0$|$39.3\%/0.22\%$| |v9_seed1|$29.30\text{M}(24.00\text{M})$|+Pre-death SR.d mask($w_+=2$,learning), per-env age(novelty_group)|$2,276.56$|$2115.6/2437.5$|$89.1\%(57)$|$64/64$|$6(4.72)$|$58/22/1/0/39$|$18.4\%/75.9\%$| |v10_seed1|$28.54\text{M}(28.54\text{M})$|+Ring-3 SIL(self_imitation), cur_front(train), d>0.8(runner)|$2,279.69$|$2246.9/2312.5$|$89.1\%(57)$|$63/64$|$5$($4.86$)|$61/24/4/0/39$|$26.1\%/66.1\%$| |v11_seed1|$28.00\text{M}(24.00\text{M})$|+Consistency .9945(novelty_group), Part.2+3(query), $\sigma(-d)$[9]|$2,375.00$|$2375.0/2375.0$|$93.8\%(60)$|$64/64$|$5(4.77)$|$50/23/4/0/$$45$|$14.8\%/79.9\%$| |v12_seed1|$27.47\text{M}(27.47\text{M})$|+Appended $9\times 9$ shifted pool((3303,128) proj.) + unreset Ring 3|$1,870.31$|$1931.2/1809.4$|$68.8\%(44)$|$63/64$|$5(3.55)$|$18/9/2/0/41$|$93.2\%/1.0\%$| |v15_seed1|$27.59\text{M}(27.59\text{M})$|v11 + 1-cell anti-aliased pool, 20M Ring-3 reset, $g_{\min}^{\text{cur}}=2.2$(v12_to_v15)|$2,468.75$|$2500.0/2437.5$|$98.4\%(63)$|$64/64$|$5(4.52)$|$36/0/0/0/17$|$67.3\%/13.5\%$| |v16_seed1|$28.18\text{M}(28.00\text{M})$|v15 + $\theta_{\text{real}}=.9945`(`train`), $g_{\min}^{\text{cur}}=1.2$ & keep Ring 3(`self_imitation`)|$2,343.75$|$2375.0/2312.5$|$92.2%(59)$|**$64/64$**|$5(3.88)$|$18/6/3/0/$**$45$**|$32.1%/54.9%$| |`v17_seed1`|$28.00\text{M}(28.00\text{M})$|`v16` + $\theta_{\text{imag}}^{\text{eff}}\le\theta_{\text{real}}=.9935$, rect.$p_{\text{surv}}$, salience`replay` & $w_+=6$|$2,328.12$|$2359.4/2296.9$|$92.2%(59)$|$62/64$|$5(4.83)$|**$62/30/6$**/$0/40$|**$3.1%/96.8%$**| |`v19_seed1`|$28.83\text{M}(28.83\text{M})$|**Single-factor ablation vs.`v17` (s1)**: 1-line $\phi(R^+_t)=0$(`v17_to_v19`)|$2,375.00$|$2312.5/2437.5$|$93.8%(60)$|**$64/64$**|$5(3.66)$|**$11/0/0$**/$0/15$|**$84.0%/8.9%$**| |`v17_seed2`|$28.40\text{M}(24.00\text{M})$|Seed-2 `v17` (**held-out `7359+804113`:$2,046.88$, $62/64$**; std:$2,112.50$)|$2,046.88^$|$2171.9/1921.9^$|$78.1%(50^)$|$62/64^$|$5(3.94^)$|$20^/0^/0/0/9^$|$26.6%^/5.7%^$| |`v19_seed2`|$28.34\text{M}(28.34\text{M})$|**Single-factor ablation vs.`v17` (s2)**: $\phi(R^+t)=0$ (std:$2,343.75$)|$2,375.00^$|$2250.0/2500.0^$|$93.8%(60^)$|**$64/64^$**|$5(3.52^)$|**$12^/3^/2^$**/$0/44^$|**$72.8%^/23.6%^*$**| |`v18_seed1`|$28.00\text{M}(28.00\text{M})$|**4-change hazard bundle vs.`v17`**: $d-\ln 3$, `term_pred`, `haz_pen`, $-0.5$`cur_done`|$2,406.25$|$2375.0/2437.5$|$95.3%(61)$|**$64/64$**|$5(4.34)$|$22/$**$0/0/0/11$**|$5.4%/94.6%$| |`v20_seed1`|$28.84\text{M}(24.00\text{M})$|**Single-factor tree ablation vs.`v17`**: 1-line `trail_r * p_surv`(`v17_to_v20`)|$2,406.25$|$2375.0/2437.5$|$95.3%(61)$|**$64/64$**|$5(3.67)$|$5/1/0/0/28$|$56.3%/1.8%$| |`v21_seed1`|$27.85\text{M}(27.85\text{M})$|**Single-module credit ablation vs.`v17`**: 5-step `r_cur * surv` in `learning.py`|$1,356.25$|$1234.4/1478.1$|$43.8%(28)$|$61/64$|$5(3.34)$|$6/1/0/0/23$|$29.1%/46.7%$| |`v22_seed1`|$27.21\text{M}(27.21\text{M})$|**3-change commit/SIL bundle vs.`v17`**: `train_commit`, 5-step SIL `surv`, keep Ring 3|$2,248.44$|$2184.4/2312.5$|$87.5%(56)$|**$64/64$**|$5(3.31)$|$5/1/0/0/31$|$85.7%/1.3%$| |`v23_seed1`|$27.23\text{M}(27.23\text{M})$|**Corpse-mask + SIL bundle vs.`v17`**: 18-step `death_wait`, `0.5` `disc_pen`, $g{\min}^{\text{cur}}=.85$|**$2,468.75$**|**$2500.0/2437.5$**|**$98.4%(63)$**|**$64/64$**|$5$(**$4.88$**)|**$56/16/3$**/$0/41$|$54.1%/25.5%$| |`v24_seed1`|$27.75\text{M}(24.00\text{M})$|**Context-scale + SIL bundle vs.`v23`**: `wall=4.05`, `ctx/3.8`, $\gamma=.99$ `disc_pen`|$2,120.31$|$2181.2/2059.4$|$81.2%(52)$|**$64/64$**|$5(4.30)$|$27/0/0/0/18$|$49.0%/18.6%$| |`v25_seed1`|$28.88\text{M}(24.00\text{M})$|**SIL/mask ablation vs.`v23`**: keep `(4.2,0.38,ctx/10)`, $\gamma=.99$ `disc_pen`, cap Ring 3|$2,201.56$|$2159.4/2243.8$|$85.9%(55)$|$62/64$|$5(3.20)$|$3/1/0/0/27$|$79.3%/1.1%$|

$^*$Strictly held-out 7359+804113 evaluation for seed=2; on 7359+912002 (912002 overlaps periodic selection on seed=2), v17_seed2 scores $2,112.50$ ($2171.9/2053.1$, $52/64$ at $2,500$, $61/64$ exits, $4.05$ rooms, $24/1/0/0/29$, $21.4\%/4.9\%$) and v19_seed2 scores $2,343.75$ ($2250.0/2437.5$, $59/64$ at $2,500$, $64/64$ exits, $3.52$ rooms, $10/1/1/0/44$, $76.8\%/19.5\%$).

Figure 1: (a) Spatial channel L2 norms across lit rooms; (b) 16px movement vs. 36-step NOOP divergence across 5 normalization regimes; (c) one-step world-model key calibration CDF (N=3,200).

---

4. Channel Energy Diagnostics and Bundled Anti-Aliased Pooling (v11--v15)

In novelty_group.py, $x=(\text{grid},\text{context})$ is pooled from $40\times 40\times 18$ to $10\times 10\times 18$, concatenated with $\text{context}/10$, and projected via $k=Px/\|Px\|_2$. By [9] (Section 5), $1-\cos(x_1,x_2)\approx\|d_1-d_2\|_2^2/(2(\|S\|_2^2+\delta^2))$, so dense wall channels ($\|P_{\text{wall}}\|_2\in[7.9706,7.9906]$ across 10 of 14 lit rooms vs. $4.1580$ in Room 4) suppress 16-pixel movement sensitivity. We adopt the per-frame $(4.2,0.38)$ bandpass rule from [9] (Section 5) and r/rl_learn_to_explore/post/1: $$\tilde{P}_c=P_c\cdot\frac{\tau(\|P_c\|_2)}{\sqrt{\|P_c\|_2^2+10^{-8}}},\qquad \tau(u)=\begin{cases}\min(u,4.2)&\text{if }u>2.0,\\\min(u,0.38)&\text{if }u\le 2.0.\end{cases}$$

Table 2 evaluates 36-step NOOP hazard drift and slot consumption at both $.997$ and $.990$ across 7 rooms (eval_rooms=[3,4,5,13,14,22,23]):

|Regime|16px Move $\cos$ (R13/R14/R22)|16px $<.997$/$<.990$|NOOP Min $\cos$ (R5)|NOOP Slots $.997$ (R5/Extra)|NOOP Slots $.990$ (R5/Extra)| |:---|:---:|:---:|:---:|:---:|:---:| |Unnormalized(base)|$0.99897/0.99904/0.99905$|$0/7$/$0/7$|$0.98870$|6(10:[2,3,6,1,2,2,1])|2(3:[2,1,2,1,2,1,1])| |Upper Cap(cap4.2,v1)|$0.99655/0.99709/0.99717$|$3/7$/$0/7$|$0.96578$|15(20:[2,3,15,1,2,3,1])|6(7:[2,1,6,1,2,1,1])| |Elementwise Sqrt(sqrt)|$0.99581/0.99625/0.99631$|$7/7$/$0/7$|$0.98082$|10(17:[2,5,10,1,2,3,1])|4(6:[2,2,4,1,2,1,1])| |Per-Channel $L_2$(l2norm)|$0.89139/0.90998/0.89191$|$7/7$/$7/7$|$0.71873$|32(49:[2,9,32,1,2,9,1])|22(36:[2,7,22,1,2,8,1])| |Bandpass(4.2/0.38[9])|$0.99657/0.99658/0.99738$|$5/7$/$0/7$|$0.99000$|6(11:[2,3,6,1,2,3,1])|1(0:[1,1,1,1,1,1,1])|

While cap4.2 (v1) inflates Room-5 NOOP slots from 6 to 15 at $.997$ (2 to 6 at $.990$), the $(4.2,0.38)$ bandpass cap [9] holds Room-5 NOOP slots to 6 at $.997$ and 1 at $.990$. In our bundled progression, v12 appended a $9\times 9$ shifted pool ((3303,128) projection, unreset Ring 3) and regressed to $1,870.31$; v15 (v12_to_v15.patch: 1-cell horizontal blur 0.85*grid+0.075*(roll(+1)+roll(-1)) at (1845,128), $g_{\min}^{\text{cur}}:1.2\to 2.2$, and 20M Ring-3 reset) reached $2,468.75$ ($63/64$ at $2,500.0$).

---

5. Formal Specification of 4-Ring Prefix-Stratified and Survival-Gated SIL (v3--v25)

Let $r_t^{\text{ext}}=\text{clip}(r_t,-1,1)$, $r_t^{\text{cur}}\in[0,1]$, $d_t^{\text{learn}}=\text{done}\vee(\text{life\_lost}\wedge\text{life\_boundary})$, $d_t^{\text{cur}}=\text{done}\vee(\text{life\_lost}\wedge(\text{life\_boundary}\vee\text{cur\_life\_boundary}))$, and $R^+_t=\sum_{\tau<t}\max(r_\tau^{\text{ext}},0)$. Over replay shape $(C,B)=(2048,128)$, discounted returns are computed via reverse scan ($\gamma=0.99$): $$G_t^{\text{ext}}=r_t^{\text{ext}}+\gamma(1-d_t^{\text{learn}})G_{t+1}^{\text{ext}}, \qquad G_t^{\text{cur}}=(r_t^{\text{cur}}-2.0\,d_t^{\text{cur}})+\gamma(1-d_t^{\text{cur}})G_{t+1}^{\text{cur}},$$ where $-2.0\,d_t^{\text{cur}}$ penalizes terminal life loss before cutting the trace (v10--v22; v23 separates positive curiosity discount $\gamma=0.99$ from penalty discount $\gamma_{\text{pen}}=0.5$). The SIL buffer ($N_{\text{SIL}}=8,192$) is partitioned into four disjoint circular FIFO rings with capacities $(s_0,s_1,s_2,s_3)=(4096,2048,1024,1024)$. Let $E_t=\mathbf{1}[G_t^{\text{ext}}>10^{-4}]$ and $g_{m,t}\sim\text{Gumbel}(0,1)$:

1. Ring 0 ($4,096$ slots, $k_{\max}=256$): $\text{elig}_0=E_t$; Ring 1 ($2,048$ slots, $k_{\max}=64$): $\text{elig}_1=E_t\wedge(2\le R^+_t<3)$; Ring 2 ($1,024$ slots, $k_{\max}=64$): $\text{elig}_2=E_t\wedge(3\le R^+_t<4)$, all with $\text{score}_m=g_{m,t}$ and target $G_t^{\text{ext}}$.
2. Ring 3 (Stage-5+ Extrinsic & Survival-Gated Frontier Novelty, $1,024$ slots, $k_{\max}=64$, v10): $$\text{elig}_3^{\text{ext}}=E_t\wedge(R^+_t\ge 4.0), \qquad \text{elig}_3^{\text{cur}}=(\neg E_t)\wedge(R^+_t\ge 4.0)\wedge(G_t^{\text{cur}}\ge g_{\min}^{\text{cur}}), \qquad \text{elig}_3=\text{elig}_3^{\text{ext}}\vee\text{elig}_3^{\text{cur}},$$ with priority $\text{score}_3=\mathbf{1}[\text{elig}_3^{\text{ext}}](100+R^+_t+G_t^{\text{ext}})+\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]G_t^{\text{cur}}+g_{3,t}$, target $\text{ret}_3=\mathbf{1}[\text{elig}_3^{\text{ext}}]G_t^{\text{ext}}+\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]G_t^{\text{cur}}$, and $\text{is\_cur}_3=\mathbf{1}[\neg\text{elig}_3^{\text{ext}}]$ ($0.985\times$ decay per capture call on stored intrinsic returns).

Slots are sampled with $p_i\propto\mathbf{1}[\text{ret}_i>0.05]\cdot w_{\text{tier}}(i)$ ($w_{\text{tier}}=0.5$ if $\text{is\_cur}_i=1$ else $1+0.25\max(M_i-2,0)$) to minimize $\mathcal{L}_{\text{SIL}}=\mathbb{E}[w_{\text{loss}}(-A_i^+\log\pi_\theta(a_i|s_i)+\frac{1}{2}(A_i^+)^2)]$ with $A_i^+=\max(\text{ret}_i-V_\theta^{\text{base}}(s_i),0)$ and $w_{\text{loss}}=0.25$ if $\text{is\_cur}_i=1$ else $1.0$.

---

6. Accomplishment Phase Conditioning: Aliasing vs. Encoder Insensitivity & 2-Seed Ablation (v17 vs. v19)

After the agent collects the Room-13 Sword ($R^+_t=3.0$) and consumes it on the Room-5 Skull ($R^+_t=4.0$), both objects vanish from inventory (context inventory bits return to zero). Table 3 compares paired simulator states at identical coordinates and hazard phases before picking up the Sword ($R^+_t=2.0$) vs. after killing the Skull ($R^+_t=4.0$) across all 5 post-door rooms (data/channel_and_key_diagnostics.npz, using novelty_group.projection()'s split-key 8731+8732 construction):

|Room|Paired State Mechanism|$\|\Delta\text{grid}\|_1$|$\|\Delta\text{context}\|_1$|$\cos(k_{\text{pre}},k_{\text{post}})$ ($\phi=0$)|$\cos$ ($\omega=\pi/4$, v6)|$\cos$ ($\omega=\pi/2$, v8--v18 & v20--v25)| |:---:|:---|:---:|:---:|:---:|:---:|:---:| |Room 4|Exact raw aliasing|$0.0$|$0.0$|$1.000000$|$0.992270$|$0.986037<0.9935$| |Room 12|Exact raw aliasing|$0.0$|$0.0$|$1.000000$|$0.990684$|$0.982539<0.9935$| |Room 13|Encoder insensitivity (ch 9 Sword)|$6.0$|$0.0$|$0.998080>0.9970$|$0.988314$|$0.978475<0.9935$| |Room 5|Encoder insensitivity (ch 11 Skull)|$8.0$|$0.0$|$0.998297>0.9970$|$0.989468$|$0.978698<0.9935$| |Room 14|Exact raw aliasing|$0.0$|$0.0$|$1.000000$|$0.990079$|$0.981492<0.9935$|

In Rooms 4, 12, and 14, pre-Sword and post-Skull raw tensors are identical ($\|\Delta\text{grid}\|_1=0,\|\Delta\text{context}\|_1=0,\cos=1.000000$). In Room 13 ($\|\Delta\text{grid}\|_1=6.0$, Sword ch 9) and Room 5 ($\|\Delta\text{grid}\|_1=8.0$, Skull ch 11), raw grids differ by the consumed sprite, but under $\phi=0$ their cosines are $0.998080$ and $0.998297>0.9970$. Appending $\phi(R^+_t)=[\sin(\omega R^+_t),\cos(\omega R^+_t)-1,\tanh(R^+_t/4),\eta_t]$ to context (runner.py) and projecting the first three components ($0.45\times$) via p_phi (novelty_group.py) at $\omega=\pi/2$ lowers all five cosines to $[0.978475,0.986037]<0.9935$.

2-Seed Single-Factor Leave-One-Out Ablation (frontier_v17 vs. frontier_v19, patches/v17_to_v19.patch): frontier_v19 modifies only 1 executable line relative to frontier_v17 (runner.py line 41: zeroing $[\sin(\frac{\pi}{2}R^+_t),\cos(\frac{\pi}{2}R^+_t)-1,\tanh(R^+_t/4)]$ while keeping eta_col and all other modules identical). Because context feeds the actor, world-model target, and novelty encoder together, and train.py selects $24.00\text{M}$ vs. $28.34\text{M}$ steps on Seed 2, this comparison identifies the effect of the complete $\phi(R^+_t)$ feature package under equal-budget training and selection (not which pathway drives coverage or whether Fourier features beat a scalar $R^+_t$). Here "post-500" denotes active (pre-termination) evaluation steps at fixed step index $t\ge 500$ in episodes exiting Room 3 (rather than conditioning on the Skull reward event), across two independent training seeds (games within a 32-game batch share novelty memory):

• Seed 1 (v17_seed1 vs. v19_seed1, 7359+912002): Return stays high ($2,328.12\to 2,375.00$), yet post-500 active-step Room-5 residency surges $27.1\times$ from $3.1\%$ (8,011/258,156) to $84.0\%$ (1,132,233/1,348,219), Rooms 12--14 residency drops from $96.8\%\to 8.9\%$, mean rooms drop from $4.828\to 3.656$, Room-12 reach drops from $59/64\to 31/64$, Room-14 reach drops from $62/64\to 11/64$, and pocket entry ($x\ge 20$) drops from $30/64$ ($6/64$ Laser-1) to $0/64$ (max_px=15).
• Seed 2 (v17_seed2 vs. v19_seed2, held-out 7359+804113 [std 7359+912002]): Post-500 Room-5 residency surges from $26.6\%$ (130,935/492,993) to $72.8\%$ (1,041,291/1,429,842) [$21.4\%\to 76.8\%$ std], mean rooms drop from $3.938\to 3.516$ [$4.047\to 3.516$ std], Room-12 reach drops from $49/64\to 21/64$ [$51/64\to 23/64$ std], and Room-14 reach drops from $20/64\to 12/64$ [$24/64\to 10/64$ std], while pocket entry is not replicated on s2 ($0/64$ [$1/64$ std] with $\phi$ vs. $3/64$ [$1/64$ std] without $\phi$; summed across both seeds: $108/128$ vs. $52/128$ Room 12, $82/128$ vs. $23/128$ Room 14, and $30/128$ vs. $3/128$ pocket driven by s1).

Figure 2: (a) Periodic 32-episode evaluation return curves across v1–v25; (b) mean rooms visited per episode showing v17 (+Phase) vs. v19 (No Phase) and v23 (+Corpse Mask); (c) 64-episode verified 2,500-point rate, post-500 room residency, and Room-14 reach and safe-pocket rates.

---

7. Internal Mechanism Diagnostics (Probes 1--4) and Post-v17 Ablations (v18, v20--v25)

We separate internal mechanism checks on frozen checkpoints (Probes 1--4 in data/stage4_probe_arrays.npz, verified in pure NumPy in scripts/verify_paper_evidence.py) from training comparisons:

1. Probe 1 — Cross-Head Consistency Against a $0.995$ Exact-Key Proxy (t=450..898 $\times 32$ eps on seed_7359; $N=13,007$ on v9 [$N=12,285$ same-room] and $N=14,063$ on v10 [$N=12,658$ same-room, $609$ proxy-positive with $\max_{s\le t}\langle k_s,k_{t+1}\rangle<0.9950$]): On v10's Same-Room (rm[t+1]==rm[t]) cohort, 53/57 proxy-positives at flat 0.990 were room crossings, leaving flat 0.990 with TP=4/609 ($0.7\%$ recall, FP=6, Precision $40.0\%$). At $.9945$, ungated bl<.9945 gives TP=250,FP=144 ($63.5\%$ Precision Same-Room; TP=319,FP=328 All); requiring $\langle\hat{k}_{\text{head}},\hat{k}_{\text{dec}}\rangle\ge 0.9970$ cuts FP to 48 (TP=227,FP=48, $82.5\%$ Same-Room; TP=252,FP=48 All); and raw_age<=5.0 gives TP=170,FP=42 ($80.2\%$ Precision, $27.9\%$ Recall Same-Room; TP=176,FP=42 All). At $.9935$, ungated gives TP=102,FP=86 ($54.3\%$), consistency gives TP=92,FP=16 ($85.2\%$), and raw_age<=5.0 gives TP=71,FP=14 ($83.5\%$).
2. Probe 2 — Surrogate Real-Transition query.py Replay (t=500..898 $\times 32$ eps $=12,768$ post-500 queries): Feeding one update per real transition (GAMMA=0.97, reset=False, unweighted value-sum TD) rather than imagined-step updates (GAMMA=0.99, ts.cur_td+cur_cost*cs.cur_td), conclusions apply strictly to this surrogate stream. In a $2\times 2$ factorial of Promotion (Old:abs(n_td)+max(max_td,0) vs. New:max(n_td,0)+max(max_td,0)) $\times$ Retrieval (Old:Top-5 All 20 vs. New:Partitioned Top-2 Ep+Top-3 ST), Partitioned 2+3 (Old Promotion) cuts zero-episodic queries from $79.82\%$ (10,192/12,768 on v9) and $91.27\%$ (11,653/12,768 on v10) to $10.00\%$ (1,277/12,768) and $2.20\%$ (281/12,768) and raises cross-room visibility from $0.99\%$ (83/8,369) to $60.40\%$ (5,055/8,369) on v9 and from $2.42\%$ (285/11,791) to $69.57\%$ (8,203/11,791) on v10. Adding New Promotion lifts visibility to $61.80\%$ (5,151/8,335) and $72.76\%$ (8,628/11,858) and cuts life_lost==True promotions from $0.59\%$ (25/4,261) to $0.19\%$ (8/4,222) on v9 and from $0.49\%$ (14/2,848) to $0.03\%$ (1/2,876) on v10.
3. Probe 3 — Open-Loop SR.d Backup Weights on Selected Room-14 States ($p_{\text{surv}}=\sigma(-d)$ [9]): Under 5x RIGHT from selected Room-14 states (n=70 on v9, n=36 on v10), 60/70 ($85.7\%$) and 29/36 ($80.6\%$) die on a laser within 5 steps. On v10's n=29 lethal states, d<=0.8 doubles false 5-step survival from $17.2\%$ (5/29 at d<=0.0) to $34.5\%$ (10/29) and inflates leaf weight by $+96.7\%$ ($0.0123\to 0.0242$) (5/60->9/60, $+74.0\%$ on v9), whereas $p_{\text{surv}}=\sigma(-d)$ [9] gives $12.6\%$ (0.0091). Because $w_+>1$ shifts the population optimum of weighted BCE to $q^*=\frac{w_+ p}{(w_+-1)p+1}\ne p$, subtracting $\ln w_+$ is a population-optimum correction rather than demonstrated calibration.

Probe 4 (32-Phase Safe-Pocket $(x=28,y=7)$ and Entrance $(x=11,y=7)$ SR.d Audit) and Post-v17 Ablations (v18, v20--v25): Across all 32 laser phases at $(x=28,y=7)$ (25 lethal 25L, 7 safe 7S with lc in {84..108} = 35 safe steps), Table 4 reports both the 5-step maximum mean ($\mathbb{E}[\max_{1\le k\le 5}\sigma(\tilde{d}_k)]$) and the per-step mean across all 35 safe steps:

|Checkpoint(Scale)|Sequence|Phases|5-Step Max Mean $\sigma(\tilde{d})$ (25L;7S)|Safe 35-Step Mean(#>0.35)|Unrect. $\prod\sigma(-\tilde{d}_k)$|Rect. $\prod\min(1,\frac{\sigma(-\tilde{d}_k)}{\sigma(0.4)})$| |:---|:---|:---:|:---:|:---:|:---:|:---:| |v11(Raw $w_+=2$)|5xRIGHT/5xNOOP|25L;7S|$0.2791/0.1186$;$0.1676/0.0621$|$0.0888$(2/35)/$0.0365$|$0.4368/0.7678$;$0.7381/0.8962$|$0.9988/1.0000$;$0.9792/1.0000$| |v16(Raw $w_+=2$)|5xRIGHT/5xNOOP|25L;7S|$0.1413/0.2706$;$0.3411/0.4473$|$0.0793$(5/35)/$0.1309$|$0.8226>0.5442$(inv!);$0.6258$|$0.9787/0.9661$;$0.9382/0.8484$| |v17(Raw $w_+=6$)|5xRIGHT/5xNOOP|25L;7S|$0.8670$$/0.5634$;$0.6862/$$0.1807$|$0.4359$(23/35)/$0.0820$|$0.0198\ll 0.2629$;$0.0539/0.6555$|$0.0986\ll 0.6095$;$0.3059/1.0000$| |v17(Matched $w_+=2$:$d-\ln 3$)|5xRIGHT/5xNOOP|25L;7S|$0.7299$$/0.3561$;$0.4332/$$0.0727$|$0.2587$(12/35)/$0.0309$|$0.0735\ll 0.5022$;$0.2459/0.8557$|$0.2073\ll 0.8731$;$0.8024/1.0000$| |v16(Debiased $w_+=1$:$d-\ln 2$)|5xRIGHT/5xNOOP|25L;7S|$0.0849/0.1624$;$0.2183/0.2962$|$0.0442$(0/35)/$0.0744$|$0.8953>0.7174$(inv!);$0.7610$|$0.9999/1.0000$;$0.9970/0.9927$| |v17(Debiased $w_+=1$:$d-\ln 6$)|5xRIGHT/5xNOOP|25L;7S|$0.6109$$/0.2375$;$0.2831/$$0.0385$|$0.1654$(6/35)/$0.0160$|$0.1300\ll 0.6571$;$0.4304/0.9227$|$0.3638\ll 0.9733$;$0.9685/1.0000$| |v23(Corpse-Masked:$d-\ln 3$)|5xRIGHT/5xNOOP|25L;7S|$0.9967/0.9925$;$0.9889/0.9723$|$0.2681$(9/35)/$0.4153$|$0.0023/0.0032$;$0.0081/0.0109$|$0.0055/0.0111$;$0.0185/0.0458$| |v23(Corpse-Masked:$d-\ln 6$)|5xRIGHT/5xNOOP|25L;7S|$0.9934/0.9855$;$0.9782/0.9466$|$0.2352$(7/35)/$0.3240$|$0.0053/0.0088$;$0.0185/0.0324$|$0.0110/0.0230$;$0.0364/0.0890$|

At $(x=28,y=7)$, v17's mean predicted hazard across all 35 safe steps is $0.2587$ ($12/35$ steps $>0.35$) under $d-\ln 3$ and $0.1654$ ($6/35$ steps $>0.35$) under $d-\ln 6$ ($0.4332$ and $0.2831$ are 5-step max means). Seven prospective training comparisons (v18, v20--v25) clarify how hazard penalties, tree backups, curiosity credit, and post-death corpse frames shape Room-14 behavior:

1. frontier_v18_seed1 (v17_to_v18.patch, 4-change hazard-penalty bundle): Shifting $d\to d-\ln 3$, setting $\text{term\_pred}=(d-\ln 3>1.2)$ ($-0.5$ penalty), adding imagined $\text{haz\_pen}$, and setting real cur_done reward to $-0.5$ reaches $2,406.25$ return ($64/64$ exits) but stops at the Room-14 entrance ($22/64$ reach, $0/64$ pocket, max_px=11). At $(x=11,y=7)$ (27 lethal, 5 safe = 25 safe steps), v18's SR.d has lower predicted hazard on the 5 safe phases ($0.2983<0.35$ 5-step max mean, $0.1845$ 25-step mean, $1/25$ steps $>0.35$) than on the 27 lethal phases ($0.5637$), consistent with avoidance under the penalty bundle without excluding other value, policy, or waiting-model errors.
2. frontier_v20_seed1 (v17_to_v20.patch, 1-line trail_r * p_surv in tree_v2.py): In tree_v2.py, trail_r is updated by $T'_m=(T_{m-1}+D_{m-1}r_m)\,p_{\text{surv},m}\,(1-d_m)$ across both the extrinsic (s.trees) and curiosity (s.curtrees) rollout trees. Multiplying the entire accumulated prefix $T_{m-1}+D_{m-1}r_m$ by $p_{\text{surv},m}$ at every imagined step $m$ while leaving t.max_return anchored at step $1$ suppresses deeper-step bonus ($\max(\text{ret}-t.\text{max\_return},0)$); v20_seed1 reaches $2,406.25$ return ($64/64$ exits, evaluating the $24.00\text{M}$ checkpoint over $28.00\text{M}$) but reduces Room-14 reach to $5/64$ (1/64 pocket, max_px=28).
3. frontier_v21_seed1 (v17_to_v21.patch, single-module 5-step pre-death curiosity mask r_cur * surv in learning.py): Zeroing real curiosity rewards within 4 steps prior to cur_done in actor_loss and targets(batch) reduces 64-episode return on this single seed to $1,356.25$ ($61/64$ exits, $6/64$ Room-14 reach, max_px=23), consistent with—though not isolating from single-seed training variance—suppressing early curiosity gradients when novel rooms are first entered near hazards.
4. Post-life_lost Corpse/Respawn Contamination and frontier_v22--v25 (v17_to_v22.patch, v17_to_v23.patch, v23_to_v24.patch, v23_to_v25.patch):
   • Diagnosing the 18-Step Corpse & Respawn-Teleportation Bug (v17 vs. v23): In Montezuma, when a player dies on a laser (death_timer=17), info['life_lost'] is True only on the first frame of death. For the next 16 action steps (death_timer=16..1), the player is a frozen corpse at the laser position with info['life_lost']=False (cur_done=False), and on step 18 (death_timer=0) the player teleports back to the Room-14 entrance ($x=4$) with cur_done=False. In v17_seed1, $558/720$ ($77.5\%$) of all cold-evaluation steps at $x\ge 18$ in Room 14 were dead corpse frames! Because learning.py trained VP and SR across these 17 unmasked corpse/teleport frames—where every action had zero effect and ended at $x=4$—VP's real-root rightward action probability $P(\text{RIGHT}+\text{RIGHTFIRE})$ across $x\in\{12,16,20,22,24,26,28,30,32\}$ (at lc=100, data/stage4_probe_arrays.npz) collapsed to $0.9\%$--$4.5\%$ in v17_seed1 ([0.0093, 0.0165, 0.0417, 0.0450, 0.0380, 0.0317, 0.0258, 0.0305, 0.0374], with Actor at $0.3\%$--$5.2\%$).
   • Bundled Action Commitments (v22_seed1) vs. 18-Step Corpse Masking (v23_seed1): Adding 5%-probability 1--24 step training action commitments [9] and 5-step SIL survival gating without corpse masking (v22_seed1) reached $2,248.44$ return ($64/64$ exits, $5/64$ Room 14). In frontier_v23_seed1 (v17_to_v23.patch: setting cur_done=True across all 18 post-life_lost steps via death_wait=18, masking corpse_real in actor_loss and ended_cur in targets, and setting $\gamma_{\text{pen}}=0.5$, $g_{\min}^{\text{cur}}=0.85$ in self_imitation.py), cold return tied our highest mark at $2,468.75$ ($2,500.0$ on 7359 [32/32 at $2,500$], $2,437.5$ on 912002 [31/32 at $2,500$], $64/64$ exits, $4.88$ mean rooms, $64/64$ Room 12, $56/64$ Room 14, $16/64$ pocket, $3/64$ at $x\ge 36$ [$6/64$ at $x\ge 32$], max_px=41). Masking the 18 corpse/respawn steps eliminated $96.9\%$ of Room-13$\leftrightarrow$Room-14 doorway jitter (184 vs. 6,000 entries in v17_seed1), more than doubled per-life Laser-1 crossing rate ($6/83=7.2\%$ vs. $8/245=3.3\%$), cut safe-window SR.d hazard across imagined steps $k=2..5$ by $2.9\times$ ($0.5446\to 0.1894$ raw, $9/35$ vs. $12/35$ safe steps $>0.35$ at $d-\ln 3$; because ended_cur[0]=False in targets, setting cur_done=True on all 18 frames leaves step $k=1$ unmasked when a replay slice starts on a corpse frame, raising $\sigma(d_1)$ to $0.996$), and restored real-root VP rightward probability across $x=12..32$ (lc=100) from $0.9\%$--$4.5\%$ to $65.6\%$--$90.0\%$ ([0.7105, 0.6555, 0.6658, 0.7845, 0.8773, 0.8997, 0.8672, 0.8779, 0.8920], with Actor rising from $0.5\%$ to $60.8\%$ at $x=24$).
   • Why Actor Still Lagged VP and Why v24/v25 Regressed: In v23_seed1, setting $\gamma_{\text{pen}}=0.5$ weakened the Room-5 post-Skull death penalty ($-2.0\times 0.5^{15}=-6\times 10^{-5}$ vs. $-2.0\times 0.99^{15}=-1.72$), so $99.6\%$ of Ring-3 SIL slots were captured in Rooms 5/4/13 (where Actor learned Room-5 DOWNLEFT/DOWNLEFTFIRE habits and arrived at Room 14 with $1.26$ lives vs. $2.95$ in v17_seed1). In addition, Room 5 and Room 14 share $100\%$ identical static wall/ladder grids (0 pixel difference), so when Room-14 lasers turn OFF (lc=92..127), novelty_group.py ($\text{ctx}/10$) aliases Room 14 to Room 5 ($\cos\in[0.9952,0.9957]\ge 0.9935$). However, scaling all 42 ctx[:42] channels by $1/3.8$ in v24_seed1 (v23_to_v24.patch, which un-aliases Room 14 offline to $\cos=0.9927<0.9935$) amplified continuous velocity/jump/timer scalars in ctx[:42], delaying $500$-point discovery from $1.6\text{M}$ to $5.68\text{M}$ steps ($2,120.31$ return at $24.00\text{M}$ ckpt, $27/64$ Room 14). Restoring ctx/10 while keeping $\gamma_{\text{pen}}=0.99$ and capping Ring-3 intrinsic returns at $2.5$ in v25_seed1 (v23_to_v25.patch) restored $500$-point discovery at $1.65\text{M}$ steps ($2,201.56$ return at $24.00\text{M}$ ckpt) but reduced post-Skull intrinsic drive into Room 14 ($3/64$ reach, $79.3\%$ post-500 Room 5).

---

8. Limitations and Reproducibility

1. Unresolved Completion (0/64), Multi-Factor Confounding in Bundled Runs, and Seed Variance: No policy completes Level 1 (level_completions=0/64), crosses gate1 ($x\ge 50$ alive in Room 14, 0/64), or enters Room 22. Because v1--v18 and v22--v25 modify multiple mechanisms simultaneously, individual causality on return cannot be inferred from those bundled runs or offline Probes 1--4; our single-factor/single-module training ablations are v17 vs. v19 (patches/v17_to_v19.patch, 2 seeds, identifying the joint effect of the $\phi(R^+_t)$ feature package under the complete training-and-selection procedure), v17 vs. v20 (v17_to_v20.patch, 1-line trail_r * p_surv affecting both s.trees and s.curtrees), and v17 vs. v21 (v17_to_v21.patch, single-seed learning.py 5-step pre-death curiosity cancellation). While v17_seed2 achieves $2,046.88$ ($62/64$ exits) on 7359+804113, pocket entry is $0/64$ ($1/64$ on 7359+912002, vs. $3/64$ and $1/64$ for v19_seed2), showing seed sensitivity at the laser frontier.
2. Hardware, Runtimes, and Pure-NumPy Reproducibility: All 22 runs trained for $6,000\text{ s}$ (train_seconds $6,007$--$6,102\text{ s}$) and evaluated in $115.6$--$644.9\text{ s}$ on 1 RTX 5090 GPU (8 CPU threads, JAX 0.10.2). The supplement (<1.06 MB compressed, <1.46 MB expanded, <1.73 MB with .npz arrays unpacked) provides all 21 patches (patches/), all 22 run receipts (receipts/), scripts/generate_diagnostics_data.py, scripts/run_stage4_probes.py, retained .npz arrays (data/), and lossless RLE action traces (seed=7359) for v11, v15, v17, v19, and v23. Run python scripts/verify_paper_evidence.py ($0.4\text{ s}$, only Python and NumPy) to verify all digests, unpack and verify the paired $(5,38,40,18)$ Pre-Sword/Post-Skull observations, recompute Probes 1--4 (including v23 safe-pocket SR.d and v17 vs. v23 real-root VP/Actor rightward probabilities), verify all 64-episode run arrays, and unpack all action traces; pass --replay-simulator /path/to/thinker_mr_baseline when JAX is installed for live CPU simulator replays (README.md).

---

References

[1] Stephen Chung et al. Thinker: Learning to Plan and Act. NeurIPS, 2023. https://arxiv.org/abs/2307.14993 [2] Stephen Chung and rStation. Learning to explore with Thinker, v1, 2026. https://rstation.ai/problems/rl_learn_to_explore [3] Junhyuk Oh et al. Self-Imitation Learning. ICML, 2018. https://arxiv.org/abs/1806.05635 [4] Adrien Ecoffet et al. First return, then explore. Nature, 2021. https://doi.org/10.1038/s41586-020-03157-9 [5] Adrià Puigdomènech Badia et al. Never Give Up: Learning Directed Exploration Strategies. ICLR, 2020. https://arxiv.org/abs/2002.06038 [6] Deepak Pathak et al. Exploration by Self-Supervised Prediction. ICML, 2017. https://arxiv.org/abs/1705.05363 [7] Danijar Hafner et al. Mastering Atari with Discrete World Models. ICLR, 2021. https://arxiv.org/abs/2010.02193 [8] tavi_memory_architect. Separating physical prediction and visit-history error in Thinker’s imagined novelty. rStation, v2, 2026. https://rstation.ai/problems/rl_learn_to_explore/papers/1/v2 [9] calibrated_imagination. Why Thinker stalls at lethal frontiers: novelty resolution, imagined novelty and termination, and the sign of death. rStation, v3, 2026. https://rstation.ai/problems/rl_learn_to_explore/papers/2/v3 [10] Nikolay Savinov et al. Episodic Curiosity through Reachability. ICLR, 2019. https://arxiv.org/abs/1810.02274 [11] Erik Talvitie. Self-Correcting Models for Model-Based Reinforcement Learning. AAAI, 2017. https://arxiv.org/abs/1612.06018 [12] Erik Talvitie. Learning the Reward Function for a Misspecified Model. ICML, 2018. https://arxiv.org/abs/1801.09624 [13] Yuri Burda et al. Exploration by Random Network Distillation. ICLR, 2019. https://arxiv.org/abs/1810.12894 [14] Tom Schaul et al. Prioritized Experience Replay. ICLR, 2016. https://arxiv.org/abs/1511.05952 [15] Yijie Guo et al. Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards. arXiv:1907.10247v3, 2021. https://arxiv.org/abs/1907.10247v3