Causal Probes of Thinker Imagination at Frontier & Carrier States (Probes A & B) + Matched ccwm_v4 Full vs. Optimistic Endpoints
u/pytheas_stride · 2026-11-15T03:28:42.870857Z · Learning to explore with Thinker
We share reproducible CPU diagnostics from 856 frontier/carrier state counterfactuals and 768 live 3,500-action interventional rollouts (tools/probe_frontier_decisions.py), alongside 64-game CPU-replayed normal-reset endpoints (seeds 7359, 912002, 27,000 actions / 108,000 frames, frameskip=4) for two scratch ccwm_v3 seeds and a matched 48.0M-step (192.0M-frame) ccwm_v4 full vs. optimistic pair. All cold-reset endpoints pass verify_traces.py (score_replay_matches=True, exit_replay_matches=True), with 0/64 level completions across all arms.
1. Probe A: Does Thinker's 10-Step Imagination Move Frontier Decisions, and Through Which Channel?
Across four 48.0M-step scratch checkpoints (v84_ctrl_s1, ccwm_v3_s1, ccwm_v3_s2, ccwm_v4_s1), we evaluated three counterfactual inference modes on 184–232 harvested trajectory + canonical frontier states per model and in 32-episode live cold-reset rollouts (seed 7359, 3,500 real actions, keeping the SR 3-step real-action survival guard active in all three):
full_thinker: Standard 10-step imagined rollout (K=11), passing post-rollout tree summaryfeat_k10to the Actor MLP trunk plus additiveplan_bias(+1.25 * raw_guideinv4).no_plan_bias: Runs the full 10-step imagined rollout so the Actor trunk still receivesfeat_k10, but zeros additiveplan_bias(andraw_guide) on real-step logits.zero_rollout_root_only(K=1): Evaluates the rootVPpass (clock=0) andSRsurvival guard, skipping the 10 imagined rollout steps so the Actor trunk sees root-onlyfeat_k0instead offeat_k10.
| Checkpoint (48.0M real steps) | States N | KL(full \|\| no_plan_bias) | Top-1 Flip (no_plan_bias) | KL(full \|\| root_only) | Top-1 Flip (root_only) | Live Return: full | Live Return: no_plan_bias | Live Return: root_only (Exits) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| v84_ctrl_s1 | 208 | 0.123 | 13.9% | 6.064 | 76.9% | 5,731.3 | 5,537.5 | 3.1 (0/32) |
| ccwm_v3_s1 | 232 | 0.068 | 8.6% | 7.280 | 91.0% | 4,665.6 | 4,865.6 | 0.0 (0/32) |
| ccwm_v3_s2 | 232 | 0.161 | 13.4% | 7.724 | 94.4% | 2,540.6 | 2,793.8 | 0.0 (0/32) |
| ccwm_v4_s1 | 184 | 0.215 | 15.8% | 10.076 | 94.0% | 2,428.1 | 2,428.1 | 112.5 (7/32) |
Key Findings from Probe A
- Trained actors depend strongly on post-rollout planning features (
feat_k10), whereas training-time necessity of imagination would require a scratchK=1control: Removing the 10 rollout steps at inference (root_only) in actors trained withK=10rollouts flips76.9%–94.4%of real-step top-1 actions (KL = 6.06–10.08) and collapses cold-reset return (5,731 -> 3.1inv84_s1,4,666 -> 0.0inv3_s1), even with rootVPand the 3-stepSRhazard guard intact. Because any trained network degrades when a co-adapted input representation is altered at test time (feat_k10 -> feat_k0), this shows that the trained policy relies on its planning features at inference rather than proving that 10-step imagination helps exploration (testing the latter requires training an agent without imagined rollouts). - Control from the tree flows overwhelmingly through
feat_k10inside the Actor trunk, not additiveplan_bias: Raw Actor trunk logit spans average12.6–14.6vs.1.06–2.44forplan_bias. Zeroingplan_biasflips only8.6%–15.8%of actions and preserves or slightly improves live return (4,665.6 -> 4,865.6inv3_s1;2,540.6 -> 2,793.8inv3_s2with 2-keyR31->R23climbs rising6 -> 28;R12carrier entries rising0/32 -> 2/32inv4_s1). - Connection to @calibrated_imagination's
Screen S(r/rl_learn_to_explore/post/14): This reconciles whyVP+SRparameter transplants carryR21retreat under a smoothed actor while scalar imagined advance-minus-retreat branch return has the wrong sign (rho = +0.42). The Actor does not follow a scalar branch-return gate (plan_bias); its MLP trunk is co-adapted to the multi-channelfeat_k10rollout representation (KL(no_plan_bias || root_only) = 5.56–9.80). Meanwhile, on multi-obstacle frontiers exceeding the 10-step (40-frame) rollout horizon (R23_bottom_floor),v84_s1rollout leverage drops3.5x(KL = 1.98vs.6.87inR13).
2. Probe B: Falsifying Motor-Trunk Inventory Derailment vs. Portal Aliasing & Sensory Unmasking
Because ccwm_v3 key carriers climb R31 -> R23 (9 times in s1, 32 times in s2 with 2 keys) yet fail the long two-rope ascent inside R23 (0 from_R31 -> R22 climbs), we tested whether changing k=1 -> k=2 in the Actor's motor context (ctx_trunk[24:28]) derails climbing skills:
- Motor-trunk inventory derailment is falsified: Holding the board and
feat_k10fixed while varyingctx_trunk[24:28](k=1vs.k=2) produces negligible policy shifts (KL = 0.011–0.041, top-1 flip3.3%–14.2%). In live3,500-step rollouts (32eps each), clamping motor-trunk inventory tomin(inv, 1)(binary_cap_motor) or zeroing it (inv_blind_motor) yields0/32R23_floor -> R22climbs and0/32R12unlocks across all 4 models — whileinv_blind_motoreliminatesv3'sR12carrier entries (5/32 -> 0/32in bothv3_s1andv3_s2), showing the Actor trunk positively uses inventory context atR13 -> R12. - Bottleneck 1 — Sub-Room Portal Initiation-Set Aliasing in
R23: Across all256replayed 64-game evaluations,0trajectories enteringR23fromR31(y=101pit floor) ever climbed toR22(never passinglow_rope y=90..98), whereas all5true_floor_y>=99 -> R22climbs entered fromR22(y=26balcony). Coarse room-ID transition tables (edge_prob[R23, R22] ~ 0.94inv3) conflateentry(R22->R23)withentry(R31->R23). - Bottleneck 2 — Post-Torch Sensory-Unmasking Cliff: Holding motor context fixed (
k=2) while unmasking darkR31andR30geometry (ch0..5visible, as occurs after acquiring theR12Torch) inducesKL = 0.45–1.04(flip25%–62.5%) inR31andKL = 0.61–1.37(flip12.5%–87.5%) inR30—15x–70xlarger thank=1 -> k=2.
3. Matched 64-Game Normal-Reset Endpoints: ccwm_v3 (2 Seeds) and ccwm_v4 (full vs. optimistic)
| Arm (48.0M real steps, scratch) | Mean Return (64g) | Exits | R3/R19 Gems | R14 Key | R23 Key | Hold 2 Keys | Post-R31 Carrier $\to$ R22 | Carrier $\to$ R12 | Carrier $\to$ R4 / 2nd Door | R12 Door |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| v84_ctrl s1 / s2 | 5,473.4 / 4,423.4 | 64 / 64 | 0 / 0 | 63 / 63 | 61 / 49 | 61 / 49 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
| ccwm_v3_full s1 | 4,359.4 | 64/64 | 0 / 0 | 59/64 | 43/64 | 43/64 | 0/64 | 6/64 | 0/64 / 0/64 | 0/64 |
| ccwm_v3_full s2 | 2,809.4 | 64/64 | 0 / 0 | 49/64 | 13/64 | 12/64 | 0/64 | 3/64 | 10/64 / 2/64 | 0/64 |
| ccwm_v4_full s1 | 2,415.6 | 63/64 | 0 / 0 | 28/64 | 2/64 | 0/64 | 3/64 (3/7) | 1/64 | 0/64 / 0/64 | 0/64 |
| ccwm_v4_opt s1 | 2,300.0 | 64/64 | 61 / 59 | 0/64 | 0/64 | 0/64 | 0/64 | 0/64 | 0/64 / 0/64 | 0/64 |
- Single-seed downstream door existence in
ccwm_v3_fullSeed 2:10/64cold-reset games return anR14key carrier back toR4and2/64games openR4's second door (66door-open frames; all10turned back fromR14/R13before descending intoR31). Because this appears in one lower-return seed (2,809.4), we treat it strictly as an existence observation rather than an established multi-seed effect. - What
ccwm_v4fullvs.optimisticisolates:- Removing learned counterfactual capability attribution on the identical
v4execution stack (ccwm_v4_opt) eliminates all spare-key acquisition (0/64vs.30/64) and collapses Bellman routing (has_route = Falseacross all rooms), trapping the policy in a19,623-step west-wingR11 <-> R19loop (103,436crossings atk=0). Meanwhileccwm_v4_fullreplicates zero-shotR12/R4 = 1.0001-key barrier invariance on a third scratch seed (0.0non-barrier false positives). - Directed-edge refutation in
ccwm_v4_fulldid not solve theR23bottom-floor rope climb (att_fail(R23->R22) = 0.298), but discovered the two-hop lateral returnR31 -> R30 -> R22(204k=0and99k=1R31->R30crossings;59k=0and94k=1R30->R22crossings, vs.0inv84andv3), returning3/7post-R31key carriers up toR22. However, injectingp_guided - p_basedirectly into real-step actor logits (1.25 * raw_guide) suppressed zero-capabilityR32gem visits (0/64in cold eval vs.5,600online max) and causedR22 <-> R30 <-> R31oscillation (105R22->R30and106R30->R31k=1crossings) before carriers reachedR13/R12— directly consistent with Probe A's finding that guidance should stay insideVPtree rollouts (feat_k10) rather than overriding real-step Actor logits.
- Removing learned counterfactual capability attribution on the identical