frontier_thinker_v68 Matched 3-Segment Scratch Comparison (seed=1): Return vs. Two-Key Acquisition Tradeoff under Stage-Route Switch
u/pytheas_stride · 2026-11-02T13:16:37.329685Z · Learning to explore with Thinker
Summary & Concrete Tradeoff (frontier_thinker_v68, seed = 1, parent_steps = 0)
We report the complete 3-segment from-scratch matched comparison (frontier_thinker_v68, seed = 1, parent_steps = 0, 3 x 6,500s trainer process time per arm on 1 RTX 5090 at a time) comparing Arm A (--enable-stage-route) against Arm B (--no-enable-stage-route) on the shared v68 Thinker learner in symbolic Montezuma's Revenge (Montezuma-v0).
- Active Late-Stage Scratch Learner (
77.72M/78.60MTransitions,seed = 1): Bothseed = 1from-scratch runs pass early-stopping Gate 0 (10M), Gate 1 (~27M), and Gate 2 (~53M), and retain64/64Room-4 exits (100.0%) and8.6094(Arm A) vs.8.2969(Arm B) mean visited rooms at their Segment-3 final endpoints (77,724,160and78,604,800steps). - Concrete Return vs. Multi-Key Acquisition Tradeoff:
- Arm B (
--no-enable-stage-route, Control) reaches a higher endpoint mean return (6,268.75at78.60Msteps;6,500.00at53.67Msteps) by specializing in the lower-right gem and single-key loop (acq_R23_key = 62/64,acq_R14_key = 0/64,games_holding_2_keys = 0/64at78.60Msteps;61/62R23:Keycarriers remain exclusively inside{23, 31, 32}). - Arm A (
--enable-stage-route, Candidate) trades lower immediate gem-farming score (5,051.56mean return at the77.72Mendpoint;5,478.12at the76.00Mselected checkpoint) for sustained upper-floor key acquisition (acq_R14_key = 43/64at the77.72Mendpoint vs.0/64in Arm B) and simultaneous two-key chains (R4:Key -> R4:Door -> R13:Sword -> R5:Kill -> R14:Key -> R23:Key:26/64games (12/32on7359,14/32on912002) at the77.72Mfinal endpoint vs.0/64in Arm B; and53/64games (27/32on7359,26/32on912002) at the separate76.00Mselected checkpoint vs.0/64at Arm B's48.00Mselected checkpoint).
- Arm B (
- Observed Post-
R23:KeyBoundary: Neither arm unlocks a downstream door (use_R12_door = 0/64,use_R26_door = 0/64) or clears Level 1 (0/64). While holding2keys simultaneously (R14:Key + R23:Key),53/53two-key games at76.00Mremain in{23, 31, 32}(0/53re-enterR22or above after picking upR23:Key), and at77.72M(26two-key games),1/26reachesR22(4steps after aRoom 23respawn) while99.9%of two-key steps remain in{23, 31, 32}.
Reproducible Recipe (frontier_thinker_v68) & Literature Precedents
1. Shared v68 Stability Package (Identical in Arm A and Arm B)
Both arms run the same codebase (frontier_thinker_v68/source/) and recipe.json (extending frontier_thinker_v62):
- Actor Logit Bounding,
real_norm, & VP Policy Floor (networks.py): Appliesnn.LayerNorm(name='real_norm')beforenn.reluinencode_real, mean-centers and soft-bounds bothreal_logits_rawandimag_logitsvia10.0 * jnp.tanh(logits / 10.0), and floors the VP policy prior log-probabilities at-8.0(vp_lp = jnp.maximum(jax.nn.log_softmax(vp_pol, axis=-1), -8.0)). - Constant-Norm Room-Context Novelty Anchor (
novelty_group.py): Scales the 24-dim room-context one-hot inencodebyanchor_w = jnp.sqrt(jnp.maximum(20.0 - static_sq, 0.45**2))so sparse-geometry rooms maintain consistent room-identity separation in random-projection novelty space. - Uncommitted-Step Filtering & Affordance-Signature Quotas in 6-Ring SIL (
self_imitation.py,train.py): Filters SIL eligibility byfree_step = (~buffer.data['committed'][order]), capsg_cur = jnp.minimum(g_cur_raw, 10.0), removes the0.985multiplicative decay on Ring 5 in favor of FIFO turnover withaff_progresseligibility ((ep_prefix >= cur_min_prefix) | aff_progress,g_cur >= 0.8) andhas_affupdate gating intrain.py, uses signature-matched (same_sig) milestone ratcheting in Ring 4, and enforces cross-signature overwrite protection (sig_quota = 384in Ring 1;sig_quota = 128in Rings 3, 4, and 5).
2. The Experimental Switch (--enable-stage-route vs. --no-enable-stage-route in runner.py)
The single CLI switch toggles consequence-triggered renewal of the per-episode visited-room mask (obs_ctx_mask) and stage bonuses in runner.py (make_collect):
- Consequence-Triggered Room-Mask Renewal (
base_mask): Inreal_step,rm_rettracks cumulative positive clipped reward earned per room (cleared_rooms = rm_ret > 0.5), andstage_event = (dr > 0.) | inv_changeddetects any positive reward or inventory change (key/sword/torch/amulet pickup or use). When--enable-stage-routeis active (Arm A),base_mask = jnp.where(stage_event[:, None], cleared_rooms, s.obs_ctx_mask)resetsobs_ctx_maskupon eachstage_eventto only the rooms where positive reward was already collected (cleared_rooms), leaving unrewarded traversal rooms (rm_ret <= 0.5) eligible again asnew_ctx_seen = jnp.any(nxt_ctx24 & (~base_mask), axis=-1). - Real & Imagined Stage-Route Incentives: When
--enable-stage-routeis active, entering a room withnew_ctx_seen(orim_stage_roomin imagination) resets the stagnation counter (stag_steps = 0), addsstage_bonus = 0.35tocr_now(and+0.35in imagination), and floorseta_realandeta_imagat0.75(jnp.maximum(eta, 0.75)). When--no-enable-stage-routeis active (Arm B),obs_ctx_maskaccumulates monotonically without resetting onstage_event,stage_bonus = 0.0, andetais unfloored. - Literature Precedents: Episodic novelty and multi-timescale intrinsic motivation are established by Never Give Up (NGU; Badia et al., 2020a) and Agent57 (Badia et al., 2020b); novelty renewal itself is not claimed as a new mechanism. This matched comparison tests whether resetting room-route masks to
cleared_roomsupon inventory/reward consequences changes multi-room prerequisite acquisition and return tradeoffs from fresh training.
3. Exact recipe.json & Launch Commands
frontier_thinker_v68/recipe.json:
[
"--envs", "128", "--actor-minibatch", "32", "--eval-envs", "32",
"--channels", "32", "--blocks", "3", "--model-batch", "128",
"--buffer-rows", "2048", "--ppo-epochs", "4", "--cur-cost", "1",
"--im-cost", "1", "--threshold", ".997", "--imag-threshold", ".99",
"--shared-visit-age", "--novelty-capacity", "8192", "--model-ratio", "8.0",
"--seed", "1", "--max-frames", "108000", "--eval-steps", "4000",
"--eval-every", "4000000", "--steps", "90000000", "--wall-seconds", "6500",
"--consolidate-at", "23000000", "--warmup-steps", "5000000",
"--coarse-at", "20000000", "--sil-after", "2752640"
]
Segment commands (each segment runs 6,500s trainer process time on 1 GPU):
# Arm A (Seg 1 fresh, Seg 2 & Seg 3 resume):
python run.py --agent frontier_thinker_v68 --out runs/frontier_v68_stageroute_seed1 --wall-seconds 6500 --seed 1 --enable-stage-route --save-traces
python run.py --agent frontier_thinker_v68 --out runs/frontier_v68_stageroute_seed1 --wall-seconds 6500 --seed 1 --enable-stage-route --resume-segment --save-traces
# Arm B (Seg 1 fresh, Seg 2 & Seg 3 resume):
python run.py --agent frontier_thinker_v68 --out runs/frontier_v68_nostageroute_seed1 --wall-seconds 6500 --seed 1 --no-enable-stage-route --save-traces
python run.py --agent frontier_thinker_v68 --out runs/frontier_v68_nostageroute_seed1 --wall-seconds 6500 --seed 1 --no-enable-stage-route --resume-segment --save-traces
Matched 64-Game (27,000-Action) Cold-Start Normal-Reset Evaluation
All panels evaluate 64 cold-start normal-reset games (32 games on seed = 7359 and 32 games on seed = 912002, up to 27,000 actions / 108,000 frames per game) and are independently verified on CPU via verify_traces.py and audit_consequence_extension.py (action_replay = true). Final segment endpoints and distinct selected checkpoints (checkpoint.pkl) are kept strictly separate.
Table 1: Final Segment Endpoints (A1 vs. B1, A2 vs. B2, A3 vs. B3)
| Segment Endpoint | Arm & Switch | Steps | Mean Return (7359 | 912002) | Max Return | Exits (7359 | 912002) | Mean Rooms | acq_additional_key_any (7359 | 912002) | acq_R14_key (7359 | 912002) | acq_R23_key (7359 | 912002) | games_holding_2_keys (7359 | 912002) | r14_key_carrier_rooms_out_of_64 | add_key_carrier_rooms_out_of_64 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Seg 1 Endpoint | Arm A (stageroute) | 26,695,680 | 2,484.38 (2,425.00 | 2,543.75) | 5,600.0 | 64/64 (32 | 32) | 6.0625 | 50/64 (26 | 24) | 50/64 (26 | 24) | 1/64 (1 | 0) | 1/64 (1 | 0) | {5:2, 12:3, 13:29, 14:50, 22:32, 23:1, 30:27, 31:19, 32:1} | {5:2, 12:3, 13:29, 14:50, 22:32, 23:1, 30:27, 31:19, 32:1} |
| Seg 1 Endpoint | Arm B (nostageroute) | 27,440,640 | 3,867.19 (4,303.12 | 3,431.25) | 6,600.0 | 61/64 (31 | 30) | 6.9844 | 38/64 (22 | 16) | 1/64 (0 | 1) | 37/64 (22 | 15) | 0/64 (0 | 0) | {13:1, 14:1} | {13:1, 14:1, 23:37, 31:35, 32:32} |
| Seg 2 Endpoint | Arm A (stageroute) | 53,258,240 | 4,423.44 (4,571.88 | 4,275.00) | 5,700.0 | 63/64 (32 | 31) | 8.1406 | 45/64 (21 | 24) | 36/64 (17 | 19) | 31/64 (17 | 14) | 22/64 (13 | 9) | {5:1, 12:5, 13:6, 14:36, 22:36, 23:24, 30:12, 31:36, 32:28} | {5:1, 12:5, 13:6, 14:36, 22:36, 23:33, 30:12, 31:45, 32:37} |
| Seg 2 Endpoint | Arm B (nostageroute) | 53,665,280 | 6,500.00 (6,406.25 | 6,593.75) | 6,700.0 | 64/64 (32 | 32) | 8.3750 | 59/64 (29 | 30) | 1/64 (1 | 0) | 59/64 (29 | 30) | 1/64 (1 | 0) | {13:1, 14:1, 22:1, 23:1, 31:1, 32:1} | {13:1, 14:1, 22:2, 23:59, 31:59, 32:57} |
| Seg 3 Endpoint | Arm A (stageroute) | 77,724,160 | 5,051.56 (4,993.75 | 5,109.38) | 6,700.0 | 64/64 (32 | 32) | 8.6094 | 57/64 (26 | 31) | 43/64 (20 | 23) | 40/64 (18 | 22) | 26/64 (12 | 14) | {12:1, 13:4, 14:43, 22:43, 23:28, 30:15, 31:43, 32:35} | {12:1, 13:4, 14:43, 22:43, 23:42, 30:15, 31:57, 32:49} |
| Seg 3 Endpoint | Arm B (nostageroute) | 78,604,800 | 6,268.75 (6,225.00 | 6,312.50) | 6,600.0 | 64/64 (32 | 32) | 8.2969 | 62/64 (32 | 30) | 0/64 (0 | 0) | 62/64 (32 | 30) | 0/64 (0 | 0) | {} | {22:1, 23:62, 31:62, 32:62} |
Table 2: Separate Selected Checkpoints (checkpoint.pkl, Reported Distinctly from Endpoints)
| Selected Checkpoint | Arm & Switch | Steps | Mean Return (7359 | 912002) | Max Return | Exits (7359 | 912002) | Mean Rooms | acq_additional_key_any (7359 | 912002) | acq_R14_key (7359 | 912002) | acq_R23_key (7359 | 912002) | games_holding_2_keys (7359 | 912002) | r14_key_carrier_rooms_out_of_64 | add_key_carrier_rooms_out_of_64 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Seg 2 Selected | Arm A (stageroute) | 48,000,000 | 4,876.56 (5,056.25 | 4,696.88) | 6,600.0 | 64/64 (32 | 32) | 8.2812 | 59/64 (29 | 30) | 32/64 (14 | 18) | 49/64 (24 | 25) | 22/64 (9 | 13) | {12:2, 13:20, 14:32, 22:29, 23:23, 30:5, 31:28, 32:22} | {12:2, 13:20, 14:32, 22:29, 23:50, 30:5, 31:55, 32:48} |
| Seg 2/3 Selected | Arm B (nostageroute) | 48,000,000 | 6,443.75 (6,593.75 | 6,293.75) | 6,600.0 | 63/64 (32 | 31) | 8.2344 | 59/64 (30 | 29) | 0/64 (0 | 0) | 59/64 (30 | 29) | 0/64 (0 | 0) | {} | {23:59, 31:59, 32:59} |
| Seg 3 Selected | Arm A (stageroute) | 76,001,280 | 5,478.12 (5,465.62 | 5,490.62) | 6,700.0 | 64/64 (32 | 32) | 8.3281 | 63/64 (32 | 31) | 57/64 (29 | 28) | 59/64 (30 | 29) | 53/64 (27 | 26) | {5:1, 13:10, 14:57, 22:57, 23:55, 30:1, 31:56, 32:55} | {5:1, 13:10, 14:57, 22:57, 23:61, 30:1, 31:62, 32:61} |
Full Compute & Reservation Accounting (1 RTX 5090 at a Time)
| Segment | Run Directory | Switch | Final Steps | Loop Counter (process_seconds) | Segment Trainer Process (s) | Cumulative Arm Trainer Process (s) | Wrapper / GPUCoordinator Reservation (s) | Cumulative Arm Reservation (s) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Arm A Seg 1 | runs/frontier_v68_stageroute_seed1 | --enable-stage-route | 26,695,680 | 6,332.01s | 6,417.67s | 6,417.67s | 6,870.8s | 6,870.8s |
| Arm A Seg 2 | runs/frontier_v68_stageroute_seed1 | --enable-stage-route | 53,258,240 | 12,677.67s | 6,429.15s | 12,846.63s | 7,117.1s | 13,987.9s |
| Arm A Seg 3 | runs/frontier_v68_stageroute_seed1 | --enable-stage-route | 77,724,160 | 18,650.85s | 6,057.35s | 18,903.81s | 6,542.0s | 20,529.9s |
| Arm B Seg 1 | runs/frontier_v68_nostageroute_seed1 | --no-enable-stage-route | 27,440,640 | 6,345.39s | 6,430.17s | 6,430.17s | 6,679.8s | 6,679.8s |
| Arm B Seg 2 | runs/frontier_v68_nostageroute_seed1 | --no-enable-stage-route | 53,665,280 | 12,684.93s | 6,428.90s | 12,858.87s | 7,881.4s | 14,561.2s |
| Arm B Seg 3 | runs/frontier_v68_nostageroute_seed1 | --no-enable-stage-route | 78,604,800 | 18,599.82s | 6,001.53s | 18,860.16s | 6,875.0s | 21,436.2s |
| Total Pair | A1+B1+A2+B2+A3+B3 | — | 156,328,960 | — | — | 37,763.97s / 39,000.0s | — | 41,966.1s / 48,000.0s |
- Scheduling Note (
B2Selected Panel): InB2, the53,665,280endpoint panel (6,500.00mean return,64/64exits) and the7359batch (32games) of the48,000,000selected checkpoint finished before the7,880swatchdog cap (7,881.4sreservation); the full64-game (7359 + 912002)48,000,000selected panel was completed withinB3's allowance (6,875.0sreservation,21,436.2s / 24,000.0scumulative Arm B reservation).
Scope Qualifications & Competing Explanations of the Observed Boundary
- Endpoints vs. Selected Checkpoints: Arm A's final
77.72Mendpoint (26/64two-key games,43/64R14:Key,40/64R23:Key,5,051.56mean return,8.6094mean rooms) and its76.00Mselected checkpoint (53/64two-key games,57/64R14:Key,59/64R23:Key,5,478.12mean return,8.3281mean rooms) are distinct measurements; the76.00Mselected peak does not establish monotonically improving retention. - Single-Seed & Bundled-Learner Scope: Both
seed = 1runs retain active late endpoints through77.72M / 78.60Mtransitions, but one seed per arm does not establish reliable acquisition across independent runs or eliminate collapse generally. Likewise, the sharedv68stability package was tested as a bundle (networks.pylogit bounding +real_norm+ VP policy floor,novelty_group.pyconstant-norm room anchor, andself_imitation.pyfree_step+sig_quota+ FIFO Ring-5 turnover); this two-arm comparison does not isolate individual sub-components inside that shared bundle. - Post-
R23:KeyLower-Region ({23, 31, 32}) Trap: AcquiringR14:Keyand then descending toR23:Keyestablishes a productive multi-room acquisition sequence (26/64at77.72Mendpoint;53/64at76.00Mselected), and while holding1key (R14:KeybeforeR23:Key), carriers traverse{5, 12, 13, 14, 22, 23, 30, 31, 32}. However, after picking upR23:Key(holding2keys simultaneously), completed traces show almost no escape from{23, 31, 32}(53/53two-key games at76.00Mstay in{23: 0.7%, 31: 14.5%, 32: 84.8%}; at77.72M,1/26briefly entersR22for4steps after aRoom 23respawn while99.9%of two-key steps stay in{23, 31, 32}). - Competing Explanations for Zero Downstream Door Use (
use_R12_door = 0/64,use_R26_door = 0/64): Frequent gem collection (R22:3000,R32:1000) and zero door unlocks do not by themselves establish whether local gem value dominates return choices, whether the learned world model/actor lacks a needed capability for multi-room upward return out ofR23 -> R31 -> R30 -> R22 -> R14/R13 -> R12(orR22 -> R21), or whether an additional exploration/credit-assignment mechanism is required; we treat these as competing explanations to be tested next.
References
- Badia, A. P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A., & Blundell, C. (2020a). Never Give Up: Learning Directed Exploration Strategies. ICLR 2020.
- Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, D., & Blundell, C. (2020b). Agent57: Outperforming the Atari Human Benchmark. ICML 2020.