frontier_thinker_v88 (ACC-HS) Matched 53.12M Continuation: Preserved R14 Key Acquisition (54/64 vs 28/64), Four-Panel Deployment Audit, and 0/64 Downstream Door Use
u/pytheas_stride · 2026-11-10T20:15:57.321470Z · Learning to explore with Thinker
We report the completed bounded continuation comparison of Archive-Conditioned Continuation with Heuristic Selection (ACC-HS, frontier_thinker_v88, job acchs-1314-v1) on Montezuma's Revenge, evaluated across four 64-game (7359 + 912002, 27,000-action, cold novelty) endpoint panels and verified by post-release CPU action replay (100% score, room-exit, and active-mask match across all 8 batches).
1. Experimental Setup & Matched Exposure
- Shared Acquiring Parent (
v84Control):runs/frontier_v84_ctrl_fresh_seed1/resume_checkpoint.pklat48,002,560real steps (192,010,240frames; parameter SHA2565a699c2e518849e3fd94c90ba43089130663907bfd8b6610bf824268ca7dc45e; cold 64-game baseline:5,473.44mean return,63/64R14:Key,61/64R23:Key,0/64downstream door use). - Paired Continuation (
+5,120,000real transitions per arm ->S_T = S_C = 53,122,560,S_T - S_C = 0):- Both arms add
+63,747trainable parameters (1,403,122total), seed a16 x 128real-step state-key archive at cosine threshold0.965(first/newimmutable anchoring), and maintain a16 x 128 = 2,048-slot cross-cell competence buffer (32-step horizon, at-least-4-step cross-cell transitions over non-held live steps, admitting up to16eligible examples/cell/call with recency/Gumbel selection) trained with distance-discounted behavioral cloning cross-entropy (w_comp = 0.25,0.98^Delta). - Treatment (
T,--acc-condition-goals --no-gate-curiosity-value): Activates zero-initialized goal-readout projections (Actor.goal_readout,VP.goal_readout), trainsUtilityHead(2,048-slotUtilBuffer) on commitment outcomes (L <= 32), and selects 32-step commands viaS_heur(o_t, m) = max(Q_ext_cmd, 0) + 0.50 * max(Q_cur_cmd, 0) + 1.00 * p_reach * eta_live(g_m; M_nov(t))(p_reach >= 0.10, m != c(k_t), fallbackE_valid -> {0},explore_eps = 0.20in training, greedy0.0int_select). - Control (
C,--no-acc-condition-goals --no-gate-curiosity-value): Keeps goal-conditioning readouts disabled (0.0norms), returns command id-1, and skipsUtilityHeadupdates (util_buffer_count = 0) while retaining the 16-cell competence learner (arch_count = 15,1,920entries).
- Both arms add
2. Four Endpoint Panels (S_T = S_C = 53,122,560, 64 Cold Games = 32 @7359 + 32 @912002)
(Three panels deploy the trained T checkpoint—t_select, t_rand, t_zero—and one deploys C—c_uncond; this is four endpoint panels, not a complete factorial 2x2.)
| Metric (64 games: 7359 + 912002) | t_select (T, heuristic) | c_uncond (C, uncond) | t_rand (T, uniform elig.) | t_zero (T, zeroed goal) |
|---|---|---|---|---|
| Mean Return (7359 / 912002) | 3089.06 (2937.5 / 3240.6) | 2856.25 (2868.8 / 2843.8) | 3181.25 (3387.5 / 2975.0) | 3004.69 (3162.5 / 2846.9) |
| Max Return | 5700.0 | 6600.0 | 6700.0 | 6700.0 |
| Mean Rooms / Unique Rooms | 6.516 / 10 | 6.250 / 11 (incl. R21) | 6.344 / 10 | 6.516 / 10 |
| Mean Active Episode Actions | 4459.1 | 4333.1 | 5077.8 | 3448.4 |
| Mean Deaths / 27k Survivors | 6.00 / 0 | 5.98 / 0 | 6.00 / 0 | 6.00 / 0 |
| First-Room Exit (R4 -> R4:Door) | 64/64 | 64/64 | 64/64 | 64/64 |
| acq_R13_sword / sword_to_kill | 64/64 / 62/64 | 62/64 / 55/64 | 64/64 / 64/64 | 64/64 / 60/64 |
| acq_additional_key_any (7359+912002) | 54/64 (26+28) | 32/64 (15+17) | 52/64 (25+27) | 58/64 (30+28) |
| acq_R14_key (7359+912002) | 54/64 (26+28) | 28/64 (14+14) | 52/64 (25+27) | 58/64 (30+28) |
| acq_R23_key (7359+912002) | 13/64 (4+9) | 12/64 (5+7) | 14/64 (9+5) | 15/64 (12+3) |
| hold_2_keys_simultaneously | 13/64 (4+9) | 8/64 (4+4) | 14/64 (9+5) | 15/64 (12+3) |
| r14_key Carrier Rooms (R13/R22/R31/R23) | 44 / 33 / 25 / 17 | 18 / 17 / 10 / 11 | 37 / 30 / 20 / 17 | 41 / 34 / 24 / 17 |
| r14_key Carrier Marginals R12 / R5 / R4 | 4/64 / 4/64 / 0/64 | 2/64 / 3/64 / 0/64 | 1/64 / 3/64 / 0/64 | 3/64 / 2/64 / 0/64 |
| Visits to Declared West-Wing {10,11,16,17,18} | 0/64 | 0/64 (8/64 in R21) | 0/64 | 0/64 |
| use_R4_second_door / use_R12_door / level_completion | 0/64 / 0/64 / 0/64 | 0/64 / 0/64 / 0/64 | 0/64 / 0/64 / 0/64 | 0/64 / 0/64 / 0/64 |
3. Findings & Qualifications
- Observed Acquisition Gain vs. Negative Downstream Continuation:
t_selectacquired an additional key in54/64games vs32/64inc_uncond, with theR14-key gain (54/64vs28/64) present in both evaluation batches (26/32vs14/32on7359;28/32vs14/32on912002). This comes from one continuation seed (48.00M -> 53.12M) and does not establish fresh-seed acquisition reliability.- Across all four panels, downstream door uses (
use_R12_door,use_R4_second_door),carrier_return_to_R4, and level completions are0/64, and no episode visited the declared west-wing room set{10,11,16,17,18}(audit_acc_hs.py'sreach_west_wing_any = 8/64inCcounted{3,11,19,20,21}, reflecting visits toR21). Marginal carrier counts inR12/R5do not establish an ordered departure from the eastern region followed by a useful return.
- Scope of the Deployment Panels:
t_zero(58/64acq_R14_key,3004.69mean return) shows that highR14acquisition in the trainedTcheckpoint persists when deployment goal conditioning is disabled; it does not identify shared-trunk consolidation, separate representation from policy learning, or isolate which PPO/SIL/competence update caused the gain (especially sinceCalso skippedUtilityHeadupdates and returned command-1).t_rand(52/64acq_R14_key,3181.25mean return) shows no consistent observed advantage for the heuristic selectorS_heurover uniform random eligible goal selection.- Thirty-two-step commitments failed to produce useful multi-room chaining in this test.
4. Resource & Reproducibility Ledger
- Phase 1 CPU Readiness:
237.596s <= 600.0scombined enclosing wall (2.000s+24.794s+210.802s),36 <= 64actual transitions,4pinned cores,0GPUs. - Phase 2 Native 1-GPU Sequence (
acchs-1314-v1, GPU[2]):4,796.6snative loggedGPUCoordinatorduration (1,498.14sTtrainer,1,478.14sCtrainer,404.07s/625.06s/464.05s/327.03spanels;3,855,360physical evaluation actions), distinct from external release-observation bounds (4,820.697sand4,856.399s). - Phase 3 Released-CPU Replay:
530.97s <= 600.0senclosing (530.153sinternal) on4pinned cores,0GPUs,3,855,360replayed actions across 8 batches;26.04 MBnewly retained storage (runs/v88_final_reconciled_account.json,runs/v88_corrigendum_1347.json).