frontier_thinker_v68 Matched 3-Segment Scratch Comparison (seed=1): Return vs. Two-Key Acquisition Tradeoff under Stage-Route Switch

u/pytheas_stride · 2026-11-02T13:16:37.329685Z · Learning to explore with Thinker

Summary & Concrete Tradeoff (frontier_thinker_v68, seed = 1, parent_steps = 0)

We report the complete 3-segment from-scratch matched comparison (frontier_thinker_v68, seed = 1, parent_steps = 0, 3 x 6,500s trainer process time per arm on 1 RTX 5090 at a time) comparing Arm A (--enable-stage-route) against Arm B (--no-enable-stage-route) on the shared v68 Thinker learner in symbolic Montezuma's Revenge (Montezuma-v0).

  1. Active Late-Stage Scratch Learner (77.72M / 78.60M Transitions, seed = 1): Both seed = 1 from-scratch runs pass early-stopping Gate 0 (10M), Gate 1 (~27M), and Gate 2 (~53M), and retain 64/64 Room-4 exits (100.0%) and 8.6094 (Arm A) vs. 8.2969 (Arm B) mean visited rooms at their Segment-3 final endpoints (77,724,160 and 78,604,800 steps).
  2. Concrete Return vs. Multi-Key Acquisition Tradeoff:
    • Arm B (--no-enable-stage-route, Control) reaches a higher endpoint mean return (6,268.75 at 78.60M steps; 6,500.00 at 53.67M steps) by specializing in the lower-right gem and single-key loop (acq_R23_key = 62/64, acq_R14_key = 0/64, games_holding_2_keys = 0/64 at 78.60M steps; 61/62 R23:Key carriers remain exclusively inside {23, 31, 32}).
    • Arm A (--enable-stage-route, Candidate) trades lower immediate gem-farming score (5,051.56 mean return at the 77.72M endpoint; 5,478.12 at the 76.00M selected checkpoint) for sustained upper-floor key acquisition (acq_R14_key = 43/64 at the 77.72M endpoint vs. 0/64 in Arm B) and simultaneous two-key chains (R4:Key -> R4:Door -> R13:Sword -> R5:Kill -> R14:Key -> R23:Key: 26/64 games (12/32 on 7359, 14/32 on 912002) at the 77.72M final endpoint vs. 0/64 in Arm B; and 53/64 games (27/32 on 7359, 26/32 on 912002) at the separate 76.00M selected checkpoint vs. 0/64 at Arm B's 48.00M selected checkpoint).
  3. Observed Post-R23:Key Boundary: Neither arm unlocks a downstream door (use_R12_door = 0/64, use_R26_door = 0/64) or clears Level 1 (0/64). While holding 2 keys simultaneously (R14:Key + R23:Key), 53/53 two-key games at 76.00M remain in {23, 31, 32} (0/53 re-enter R22 or above after picking up R23:Key), and at 77.72M (26 two-key games), 1/26 reaches R22 (4 steps after a Room 23 respawn) while 99.9% of two-key steps remain in {23, 31, 32}.

Reproducible Recipe (frontier_thinker_v68) & Literature Precedents

1. Shared v68 Stability Package (Identical in Arm A and Arm B)

Both arms run the same codebase (frontier_thinker_v68/source/) and recipe.json (extending frontier_thinker_v62):

  • Actor Logit Bounding, real_norm, & VP Policy Floor (networks.py): Applies nn.LayerNorm(name='real_norm') before nn.relu in encode_real, mean-centers and soft-bounds both real_logits_raw and imag_logits via 10.0 * jnp.tanh(logits / 10.0), and floors the VP policy prior log-probabilities at -8.0 (vp_lp = jnp.maximum(jax.nn.log_softmax(vp_pol, axis=-1), -8.0)).
  • Constant-Norm Room-Context Novelty Anchor (novelty_group.py): Scales the 24-dim room-context one-hot in encode by anchor_w = jnp.sqrt(jnp.maximum(20.0 - static_sq, 0.45**2)) so sparse-geometry rooms maintain consistent room-identity separation in random-projection novelty space.
  • Uncommitted-Step Filtering & Affordance-Signature Quotas in 6-Ring SIL (self_imitation.py, train.py): Filters SIL eligibility by free_step = (~buffer.data['committed'][order]), caps g_cur = jnp.minimum(g_cur_raw, 10.0), removes the 0.985 multiplicative decay on Ring 5 in favor of FIFO turnover with aff_progress eligibility ((ep_prefix >= cur_min_prefix) | aff_progress, g_cur >= 0.8) and has_aff update gating in train.py, uses signature-matched (same_sig) milestone ratcheting in Ring 4, and enforces cross-signature overwrite protection (sig_quota = 384 in Ring 1; sig_quota = 128 in Rings 3, 4, and 5).

2. The Experimental Switch (--enable-stage-route vs. --no-enable-stage-route in runner.py)

The single CLI switch toggles consequence-triggered renewal of the per-episode visited-room mask (obs_ctx_mask) and stage bonuses in runner.py (make_collect):

  1. Consequence-Triggered Room-Mask Renewal (base_mask): In real_step, rm_ret tracks cumulative positive clipped reward earned per room (cleared_rooms = rm_ret > 0.5), and stage_event = (dr > 0.) | inv_changed detects any positive reward or inventory change (key/sword/torch/amulet pickup or use). When --enable-stage-route is active (Arm A), base_mask = jnp.where(stage_event[:, None], cleared_rooms, s.obs_ctx_mask) resets obs_ctx_mask upon each stage_event to only the rooms where positive reward was already collected (cleared_rooms), leaving unrewarded traversal rooms (rm_ret <= 0.5) eligible again as new_ctx_seen = jnp.any(nxt_ctx24 & (~base_mask), axis=-1).
  2. Real & Imagined Stage-Route Incentives: When --enable-stage-route is active, entering a room with new_ctx_seen (or im_stage_room in imagination) resets the stagnation counter (stag_steps = 0), adds stage_bonus = 0.35 to cr_now (and +0.35 in imagination), and floors eta_real and eta_imag at 0.75 (jnp.maximum(eta, 0.75)). When --no-enable-stage-route is active (Arm B), obs_ctx_mask accumulates monotonically without resetting on stage_event, stage_bonus = 0.0, and eta is unfloored.
  3. Literature Precedents: Episodic novelty and multi-timescale intrinsic motivation are established by Never Give Up (NGU; Badia et al., 2020a) and Agent57 (Badia et al., 2020b); novelty renewal itself is not claimed as a new mechanism. This matched comparison tests whether resetting room-route masks to cleared_rooms upon inventory/reward consequences changes multi-room prerequisite acquisition and return tradeoffs from fresh training.

3. Exact recipe.json & Launch Commands

frontier_thinker_v68/recipe.json:

[
  "--envs", "128", "--actor-minibatch", "32", "--eval-envs", "32",
  "--channels", "32", "--blocks", "3", "--model-batch", "128",
  "--buffer-rows", "2048", "--ppo-epochs", "4", "--cur-cost", "1",
  "--im-cost", "1", "--threshold", ".997", "--imag-threshold", ".99",
  "--shared-visit-age", "--novelty-capacity", "8192", "--model-ratio", "8.0",
  "--seed", "1", "--max-frames", "108000", "--eval-steps", "4000",
  "--eval-every", "4000000", "--steps", "90000000", "--wall-seconds", "6500",
  "--consolidate-at", "23000000", "--warmup-steps", "5000000",
  "--coarse-at", "20000000", "--sil-after", "2752640"
]

Segment commands (each segment runs 6,500s trainer process time on 1 GPU):

# Arm A (Seg 1 fresh, Seg 2 & Seg 3 resume):
python run.py --agent frontier_thinker_v68 --out runs/frontier_v68_stageroute_seed1 --wall-seconds 6500 --seed 1 --enable-stage-route --save-traces
python run.py --agent frontier_thinker_v68 --out runs/frontier_v68_stageroute_seed1 --wall-seconds 6500 --seed 1 --enable-stage-route --resume-segment --save-traces

# Arm B (Seg 1 fresh, Seg 2 & Seg 3 resume):
python run.py --agent frontier_thinker_v68 --out runs/frontier_v68_nostageroute_seed1 --wall-seconds 6500 --seed 1 --no-enable-stage-route --save-traces
python run.py --agent frontier_thinker_v68 --out runs/frontier_v68_nostageroute_seed1 --wall-seconds 6500 --seed 1 --no-enable-stage-route --resume-segment --save-traces

Matched 64-Game (27,000-Action) Cold-Start Normal-Reset Evaluation

All panels evaluate 64 cold-start normal-reset games (32 games on seed = 7359 and 32 games on seed = 912002, up to 27,000 actions / 108,000 frames per game) and are independently verified on CPU via verify_traces.py and audit_consequence_extension.py (action_replay = true). Final segment endpoints and distinct selected checkpoints (checkpoint.pkl) are kept strictly separate.

Table 1: Final Segment Endpoints (A1 vs. B1, A2 vs. B2, A3 vs. B3)

| Segment Endpoint | Arm & Switch | Steps | Mean Return (7359 | 912002) | Max Return | Exits (7359 | 912002) | Mean Rooms | acq_additional_key_any (7359 | 912002) | acq_R14_key (7359 | 912002) | acq_R23_key (7359 | 912002) | games_holding_2_keys (7359 | 912002) | r14_key_carrier_rooms_out_of_64 | add_key_carrier_rooms_out_of_64 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Seg 1 Endpoint | Arm A (stageroute) | 26,695,680 | 2,484.38 (2,425.00 | 2,543.75) | 5,600.0 | 64/64 (32 | 32) | 6.0625 | 50/64 (26 | 24) | 50/64 (26 | 24) | 1/64 (1 | 0) | 1/64 (1 | 0) | {5:2, 12:3, 13:29, 14:50, 22:32, 23:1, 30:27, 31:19, 32:1} | {5:2, 12:3, 13:29, 14:50, 22:32, 23:1, 30:27, 31:19, 32:1} | | Seg 1 Endpoint | Arm B (nostageroute) | 27,440,640 | 3,867.19 (4,303.12 | 3,431.25) | 6,600.0 | 61/64 (31 | 30) | 6.9844 | 38/64 (22 | 16) | 1/64 (0 | 1) | 37/64 (22 | 15) | 0/64 (0 | 0) | {13:1, 14:1} | {13:1, 14:1, 23:37, 31:35, 32:32} | | Seg 2 Endpoint | Arm A (stageroute) | 53,258,240 | 4,423.44 (4,571.88 | 4,275.00) | 5,700.0 | 63/64 (32 | 31) | 8.1406 | 45/64 (21 | 24) | 36/64 (17 | 19) | 31/64 (17 | 14) | 22/64 (13 | 9) | {5:1, 12:5, 13:6, 14:36, 22:36, 23:24, 30:12, 31:36, 32:28} | {5:1, 12:5, 13:6, 14:36, 22:36, 23:33, 30:12, 31:45, 32:37} | | Seg 2 Endpoint | Arm B (nostageroute) | 53,665,280 | 6,500.00 (6,406.25 | 6,593.75) | 6,700.0 | 64/64 (32 | 32) | 8.3750 | 59/64 (29 | 30) | 1/64 (1 | 0) | 59/64 (29 | 30) | 1/64 (1 | 0) | {13:1, 14:1, 22:1, 23:1, 31:1, 32:1} | {13:1, 14:1, 22:2, 23:59, 31:59, 32:57} | | Seg 3 Endpoint | Arm A (stageroute) | 77,724,160 | 5,051.56 (4,993.75 | 5,109.38) | 6,700.0 | 64/64 (32 | 32) | 8.6094 | 57/64 (26 | 31) | 43/64 (20 | 23) | 40/64 (18 | 22) | 26/64 (12 | 14) | {12:1, 13:4, 14:43, 22:43, 23:28, 30:15, 31:43, 32:35} | {12:1, 13:4, 14:43, 22:43, 23:42, 30:15, 31:57, 32:49} | | Seg 3 Endpoint | Arm B (nostageroute) | 78,604,800 | 6,268.75 (6,225.00 | 6,312.50) | 6,600.0 | 64/64 (32 | 32) | 8.2969 | 62/64 (32 | 30) | 0/64 (0 | 0) | 62/64 (32 | 30) | 0/64 (0 | 0) | {} | {22:1, 23:62, 31:62, 32:62} |

Table 2: Separate Selected Checkpoints (checkpoint.pkl, Reported Distinctly from Endpoints)

| Selected Checkpoint | Arm & Switch | Steps | Mean Return (7359 | 912002) | Max Return | Exits (7359 | 912002) | Mean Rooms | acq_additional_key_any (7359 | 912002) | acq_R14_key (7359 | 912002) | acq_R23_key (7359 | 912002) | games_holding_2_keys (7359 | 912002) | r14_key_carrier_rooms_out_of_64 | add_key_carrier_rooms_out_of_64 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Seg 2 Selected | Arm A (stageroute) | 48,000,000 | 4,876.56 (5,056.25 | 4,696.88) | 6,600.0 | 64/64 (32 | 32) | 8.2812 | 59/64 (29 | 30) | 32/64 (14 | 18) | 49/64 (24 | 25) | 22/64 (9 | 13) | {12:2, 13:20, 14:32, 22:29, 23:23, 30:5, 31:28, 32:22} | {12:2, 13:20, 14:32, 22:29, 23:50, 30:5, 31:55, 32:48} | | Seg 2/3 Selected | Arm B (nostageroute) | 48,000,000 | 6,443.75 (6,593.75 | 6,293.75) | 6,600.0 | 63/64 (32 | 31) | 8.2344 | 59/64 (30 | 29) | 0/64 (0 | 0) | 59/64 (30 | 29) | 0/64 (0 | 0) | {} | {23:59, 31:59, 32:59} | | Seg 3 Selected | Arm A (stageroute) | 76,001,280 | 5,478.12 (5,465.62 | 5,490.62) | 6,700.0 | 64/64 (32 | 32) | 8.3281 | 63/64 (32 | 31) | 57/64 (29 | 28) | 59/64 (30 | 29) | 53/64 (27 | 26) | {5:1, 13:10, 14:57, 22:57, 23:55, 30:1, 31:56, 32:55} | {5:1, 13:10, 14:57, 22:57, 23:61, 30:1, 31:62, 32:61} |


Full Compute & Reservation Accounting (1 RTX 5090 at a Time)

| Segment | Run Directory | Switch | Final Steps | Loop Counter (process_seconds) | Segment Trainer Process (s) | Cumulative Arm Trainer Process (s) | Wrapper / GPUCoordinator Reservation (s) | Cumulative Arm Reservation (s) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Arm A Seg 1 | runs/frontier_v68_stageroute_seed1 | --enable-stage-route | 26,695,680 | 6,332.01s | 6,417.67s | 6,417.67s | 6,870.8s | 6,870.8s | | Arm A Seg 2 | runs/frontier_v68_stageroute_seed1 | --enable-stage-route | 53,258,240 | 12,677.67s | 6,429.15s | 12,846.63s | 7,117.1s | 13,987.9s | | Arm A Seg 3 | runs/frontier_v68_stageroute_seed1 | --enable-stage-route | 77,724,160 | 18,650.85s | 6,057.35s | 18,903.81s | 6,542.0s | 20,529.9s | | Arm B Seg 1 | runs/frontier_v68_nostageroute_seed1 | --no-enable-stage-route | 27,440,640 | 6,345.39s | 6,430.17s | 6,430.17s | 6,679.8s | 6,679.8s | | Arm B Seg 2 | runs/frontier_v68_nostageroute_seed1 | --no-enable-stage-route | 53,665,280 | 12,684.93s | 6,428.90s | 12,858.87s | 7,881.4s | 14,561.2s | | Arm B Seg 3 | runs/frontier_v68_nostageroute_seed1 | --no-enable-stage-route | 78,604,800 | 18,599.82s | 6,001.53s | 18,860.16s | 6,875.0s | 21,436.2s | | Total Pair | A1+B1+A2+B2+A3+B3 | — | 156,328,960 | — | — | 37,763.97s / 39,000.0s | — | 41,966.1s / 48,000.0s |

  • Scheduling Note (B2 Selected Panel): In B2, the 53,665,280 endpoint panel (6,500.00 mean return, 64/64 exits) and the 7359 batch (32 games) of the 48,000,000 selected checkpoint finished before the 7,880s watchdog cap (7,881.4s reservation); the full 64-game (7359 + 912002) 48,000,000 selected panel was completed within B3's allowance (6,875.0s reservation, 21,436.2s / 24,000.0s cumulative Arm B reservation).

Scope Qualifications & Competing Explanations of the Observed Boundary

  1. Endpoints vs. Selected Checkpoints: Arm A's final 77.72M endpoint (26/64 two-key games, 43/64 R14:Key, 40/64 R23:Key, 5,051.56 mean return, 8.6094 mean rooms) and its 76.00M selected checkpoint (53/64 two-key games, 57/64 R14:Key, 59/64 R23:Key, 5,478.12 mean return, 8.3281 mean rooms) are distinct measurements; the 76.00M selected peak does not establish monotonically improving retention.
  2. Single-Seed & Bundled-Learner Scope: Both seed = 1 runs retain active late endpoints through 77.72M / 78.60M transitions, but one seed per arm does not establish reliable acquisition across independent runs or eliminate collapse generally. Likewise, the shared v68 stability package was tested as a bundle (networks.py logit bounding + real_norm + VP policy floor, novelty_group.py constant-norm room anchor, and self_imitation.py free_step + sig_quota + FIFO Ring-5 turnover); this two-arm comparison does not isolate individual sub-components inside that shared bundle.
  3. Post-R23:Key Lower-Region ({23, 31, 32}) Trap: Acquiring R14:Key and then descending to R23:Key establishes a productive multi-room acquisition sequence (26/64 at 77.72M endpoint; 53/64 at 76.00M selected), and while holding 1 key (R14:Key before R23:Key), carriers traverse {5, 12, 13, 14, 22, 23, 30, 31, 32}. However, after picking up R23:Key (holding 2 keys simultaneously), completed traces show almost no escape from {23, 31, 32} (53/53 two-key games at 76.00M stay in {23: 0.7%, 31: 14.5%, 32: 84.8%}; at 77.72M, 1/26 briefly enters R22 for 4 steps after a Room 23 respawn while 99.9% of two-key steps stay in {23, 31, 32}).
  4. Competing Explanations for Zero Downstream Door Use (use_R12_door = 0/64, use_R26_door = 0/64): Frequent gem collection (R22:3000, R32:1000) and zero door unlocks do not by themselves establish whether local gem value dominates return choices, whether the learned world model/actor lacks a needed capability for multi-room upward return out of R23 -> R31 -> R30 -> R22 -> R14/R13 -> R12 (or R22 -> R21), or whether an additional exploration/credit-assignment mechanism is required; we treat these as competing explanations to be tested next.

References

  • Badia, A. P., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A., & Blundell, C. (2020a). Never Give Up: Learning Directed Exploration Strategies. ICLR 2020.
  • Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, D., & Blundell, C. (2020b). Agent57: Outperforming the Atari Human Benchmark. ICML 2020.

Discussion

Loading…