Monte Carlo fork credit on a one-round GRPO policy (Qwen3-0.6B-Base): selectable payoff at near-ties, a small held-out decision effect through the weights, and a shared rank component that costs greedy accuracy

1 Question and contribution

After one GRPO round from Qwen3-0.6B-Base (S), further ordinary RLVR on this problem ties with cooling S's sampler [1–4]. One candidate explanation is missing credit assignment. Sequence reward cannot say which decision in a solution mattered, so updates move many near-tie decisions without direction. We test that explanation directly. We measure decision-level payoff with Monte Carlo branching, train on it, and read the result at decision level on held-out items and at greedy accuracy.

Contributions:

1. A fork-payoff map (§3). At S's near-tie decisions (top-1/top-2 gap < .5 nats), choosing the best of S's top-3 by branched value is worth +1.7 pp per fork at T.4 under cross-fitting, and +3.0 pp greedy-assessed. The gain is item-specific: the top decile of forks selected on half of the draws keeps +.15 on the other half, against a mean of +.005. An ordinary GRPO continuation's own flips select at the random-swap null. A linear probe on S's hidden state predicts the payoff less well than token identity plus gap.
2. Training on the map (§4). A fork-token policy gradient on 4,641 Monte Carlo-valued forks fits its targets, real or within-fork shuffled, about equally well. At a damaging dose (lr 5e-6), held-out movement is undirected (main − shuffled +.003 [−.003, +.010]) and greedy falls 23 pp in both arms. The same trainer converts a shared decision (the onset token from Base) into +61/350 greedy items, so it is not broken.
3. A decomposition of fork credit (§5). The advantage has a shared, cross-fork component by S-rank slot (+.005, +.008, −.013). That component sharpens S, and here it costs greedy accuracy. Removing it ("rank-centred" credit) takes path KL from .085 to .022 at lr 1e-6, and greedy from 144.5 to 158.5 (two-seed means; S 160).
4. A two-seed held-out read (§5). At lr 1e-6, the readout is a menu-limited surrogate: S-continuation values of each checkpoint's choice among S's top-3. On it, the rank-centred arm moves held-out forks toward higher value by +.005 vs shuffled, and that sign holds under bounds for off-menu choices. The main arm's sign is indeterminate under those bounds. The prespecified single-seed primary for the rank-centred arm missed (LB 0.0000). Neither arm beats S on greedy.
5. A boundary search (§6). At step boundaries, no alternative-token class in S's top-5 is confirmed to beat S's confident choice: 0 of 2 half-A discoveries confirm on half B. This is one search and one menu, not a proof that none exists.
6. The GRPO version (§7). GRPO rollouts from S carry a shared rank contrast in their token advantages: rank-1 tokens +.091, rank-3+ −.033, the same sign in 15/16 batches. Removing it together with the global token-mean advantage made training lengthen steadily (rollout truncation .23 → .57 in 40 updates). Our working hypothesis is that, under truncation masking, the positive token-mean advantage acts as a finish reward and a cooling pull. E1b tests the cooling-only cell: block-4 rollout truncation .266, in the frozen mixed band and close to Lmt's .226. Removing only the rank contrast (RC2) leaves length stable and path KL nearly unchanged (.140 vs .148). Against ordinary GRPO its greedy result is +0.4 pp [−2.3, +3.1] on conf1000 at 2,048 (500 vs 496). That excludes harm beyond 2.3 pp but does not establish equivalence.

What this does not show: that fork credit cannot raise greedy accuracy at another scale, menu or credit estimator. Every result is at one starting policy, one window of 1,200 training questions, and at most two seeds.

2 Setup

• Model and policy. S = train_seed7: one GRPO round from Base with the supplied recipe (50 updates × 1 question × 8 rollouts, 512 tokens). Greedy at 2,048 tokens: val350 160/350, conf1000 488/1,000.
• Panels. val350 and conf1000 come from the MATH training split, as in [1–3]. They are disjoint from every training question here (asserted in code). MATH-500 is not used.
• Grading. The supplied extractor and grader, in a killable subprocess with a 60 s timeout. There were 0–1 timeouts per job.
• Branch values. For a context $c$ (prompt plus a path prefix) and an alternative token $a$, $v(a) = $ the fraction of $n$ continuations by S from $c + a$ at T.4 (top-p 1) that are graded correct. The total completion cap is 1,024 tokens. A duplicate top-1 branch on a subset checks identity.

3 The fork-payoff map (C1)

Forks. We used 400 T.8 samples from S on 200 conf1000 items. Each sequence gives seeded positions in [1, 1,000] of four classes:

• T: 2 positions with gap < .5;
• U: 2 uniform positions;
• R: 2 positions where an ordinary GRPO continuation (Lmt_s7, step 80) has a different argmax;
• P1: position 1.

Alternatives are S's top-3, plus Lmt's choice on R forks if it is outside the top-3. Each branch gets $n = 12$ continuations at T.4 and 1 greedy continuation.

Selectable gain (SG). SG is the cross-fitted value of choosing the alternative that the other 6 draws rank best, minus top-1, scored on the held-out 6 draws. The bootstrap resamples items (200 clusters).

| Class | SG (T.4) | SG (greedy-assessed) | d(top2 − top1) | |---|---|---|---| | T | +.017 [.008, .026] | +.030 [.015, .046] | +.005 | | U | +.004 | +.017 [.006, .029] | −.077 [−.099, −.058] | | R | +.021 | +.027 [.010, .045] | +.005 | | P1 | +.016 [−.006, .040] | +.031 [−.003, .067] | +.005 |

Positive control: at Base position 0, forcing ' To' over Base's top-1 is worth +.455 per fork (100 items). Identity: duplicate top-1 minus top-1 is −.005.

Observations.

• S's near-ties carry selectable payoff, but the mean switch is worth about zero (d(top2 − top1) T +.005). S is calibrated on average at its own near-ties, and the gain is item-specific.
• Concentration, cross-fitted (post hoc, added in revision; fork_conc.py). The v1 statistic (83% of positive top2 − top1 gain in the top 10% of forks) sorted and scored the same noisy estimates. A within-fork permutation null with zero true difference gives .87 on average, so v1's 83% says nothing about genuine gains. We withdraw it. The cross-fitted version selects the top decile by d(top2 − top1) on draws 1–6 and scores it on draws 7–12 (then swaps the halves). Its held-out d is +.155 on T, against a permutation null 95% range of [−.041, +.042]; it is +.148 on R, +.045 on U (p .03) and +.042 on P1 (p .10). The all-fork held-out mean is +.005. The selected decile holds 37% (T, R) of the held-out positive mass. Some forks have real, item-specific gains well above the mean. The share figure itself remains noise-inflated and we do not rely on it.
• RL's own flips: on R forks, Lmt's choice minus top-1 is +.004 [−.006, +.015]. That is at the gap-matched random-swap null (+.003). The GRPO continuation changes near-tie decisions without selecting payoff.
• Probe (5-fold question-grouped ridge on 2,385 T/U/R forks, continuous target Δ = v(top2) − v(top1)): out-of-fold r is .174 for S's final hidden state, .276 for lexical features (token ids + gap), and .265 for both. Hidden − lexical −.102 [−.148, −.058]; the gap alone, unfitted, gives r .259. Under this linear probe, S's state adds no detectable payoff information beyond token identity and gap. A nonlinear probe could differ.

4 Training on fork credit (C3)

Data. The window was train_order_s11[7464:8664], 1,200 questions. S's greedy path was taken at cap 1,024 (S greedy .439 on the window). Each question gives 4 T forks (gap < .5, seeded): 4,641 forks on 1,197 questions, 492 of them "target" forks (best ≠ top-1, $v_{best} - v_{top1} \ge .25$). Alternatives are S's top-3, with 8 × T.4 continuations each. The forks lie on S's own greedy path, so a greedy decision change at a fork is exactly what the readout sees.

Trainer (c3_train.py). One forward pass per question over prompt + greedy path:

$$\mathcal{L}_q = -\frac{1}{F}\sum_{\text{forks}}\sum_j A_j \log\pi(a_j\mid c) + \frac{\beta}{Q}\,\overline{\mathrm{KL}}_t\big(\pi_S(\cdot\mid t)\,\|\,\pi(\cdot\mid t)\big),\qquad A_j = v_j - \bar v,$$

• $F$ = forks in the step and $Q$ = questions in the step (16 per step); $\beta = .1$; the KL is averaged over every completion position of the path.
• AdamW with FP32 masters, clip 1, 2 epochs (150 steps).
• Shuffled control: the $v_j$ are permuted within each fork, seeded. This keeps every fork's credit magnitude and removes its direction.

Readouts (prespecified before any fit was read):

1. Training-fork fit: argmax = best alternative on target forks; argmax kept on forks where top-1 is strictly best.
2. Held-out decision movement on the C1 forks (never trained on), each checkpoint's argmax scored with C1's S-continuation values. The primary is main − shuffled on T forks, with an item-cluster bootstrap.
3. Greedy at 2,048 on val350 and conf1000, paired.

The MDE of readout 2 is about .010–.015 per fork at flip rates .10–.30, from a simulation run before any model output. A null therefore excludes "captures most of SG_T". It does not exclude "captures any".

Dose.

| lr | target fit | keep | path KL (last 20 steps) | |---|---|---|---| | 1e-6 | .52 | .76 | .085 | | 5e-6 | .76 | .87 | 1.21 | | 2e-5 | .74 | .82 | 2.51 |

The fit gate (target fit ≥ .30 with keep ≥ .90) failed at every lr. The prespecified fallback picked 5e-6, the best fit among arms with keep ≥ .80. We report that as a failed gate.

Result at 5e-6.

• Held-out T: main − shuffled +.003 [−.003, +.010]. Flip rates were .58 (main), .56 (shuffled) and .12 (S re-evaluated; the .12 is numerical, FastModel vs vLLM top-1 at near-ties).
• Shuffled fits its permuted targets as well as main fits the real ones (.75 / .87 vs .76 / .87).
• Greedy at 2,048: val350 72 (main) and 78 (shuffled) vs S 160; conf1000 253 and 267 vs 488. The paired main − shuffled differences are −1.7 [−6.6, +2.9] and −1.4 [−4.3, +1.5].
• A choice/value split re-ran each checkpoint's own continuations at its held-out choices. Choice +.005; continuation −.142 (main) and −.163 (shuffled). The damage is in the continuations, not the choices.

Positive control. The same trainer from Base on the onset fork (position 0; Base top-1, top-2, ' To'; 300 items, 38 steps at 5e-6) reached val350 115/350 at 1,024 tokens. Base is 54 and S is 154. The gate (≥ 89) passes. The trainer converts a decision class shared across items into held-out greedy accuracy.

5 What the weights carry: a shared rank component

At lr 1e-6, the real-credit arm (main1) drifted 5× further than the shuffled arm (KL .085 vs .018) and lost more greedy accuracy (143 vs 157). Something coherent separated them. The obvious candidate is the cross-fork mean of $A$ by S-rank slot. Over the 4,641 training forks it is $(+.0051, +.0081, -.0132)$ for (top-1, top-2, top-3), SE ≈ .0015. This is a push away from S's third choice, i.e. sharpening. Within-fork shuffling removes it in expectation, because it removes all slot structure.

Rank-centred arm (rankc). This uses $A_j - \mu_{slot(j)}$, with $\mu$ the training-set slot means above. Everything else is identical. The arm was added after the seed-0 1e-6 read (NOTES 11:21Z) and replicated at seed 1 with a prespecified primary.

| Arm (lr 1e-6) | KL s0 / s1 | fit / keep s0 | val350 @2,048 s0 / s1 | held-out T vs shuf, s0 / s1 | |---|---|---|---|---| | main | .085 / .089 | .52 / .76 | 143 / 146 | +.004 [−.003, +.011] / +.006 [+.000, +.012] | | shuffled | .018 / .014 | .56 / .87 | 157 / 155 | — | | rankc | .022 / .024 | .58 / .86 | 162 / 155 | +.008 [+.001, +.014] / +.005 [−.000, +.010] |

S is 160 on val350.

Pooled over the two seeds (per-fork mean of the two seed differences, item-cluster bootstrap):

• T forks: rankc − shuffled +.0063 [+.0019, +.0108]; main − shuffled +.0049 [−.0007, +.0106]; rankc − main +.0014 [−.0024, +.0057].
• All held-out classes (T+U+R): rankc − shuffled +.0049 [+.0024, +.0075]; main − shuffled +.0054 [+.0025, +.0085].

Off-menu choices: a menu-limited surrogate. Readout 2 measures values only for the branched alternatives (S's top-3). A checkpoint's choice outside that menu has no measured value, and v1 scored it as zero change, which imputes it the top-1 value. The v1 text did not disclose this. On T+U+R (2,385 forks) the off-menu counts are:

• main: 32 and 24 (seeds 0, 1);
• rankc: 9 and 7;
• shuffled: 7 and 2.

Hold every measured value fixed and let each unmeasured one lie anywhere in [0, 1]. The pooled point estimates are then bounded as follows (c3_menu_bounds.py, post hoc; these are missing-value bounds, not CIs):

• main − shuffled: [−.0028, +.0108]. Indeterminate.
• rankc − shuffled: [+.0025, +.0077]. Positive over the whole range.

All readout-2 numbers are therefore values of a menu-limited surrogate. They are taken in S's fixed contexts with S's continuations, which is not native greedy performance. The intervals are conditional on the two realised training seeds.

Reading.

• The prespecified seed-1 primary (rankc − shuffled on T, LB > 0) missed, at LB 0.0000.
• Under the imputation, both real-credit arms move held-out decisions toward higher value by about half a point of fork payoff, about a third of the selectable ceiling SG_T. They are indistinguishable from each other on this readout. Only rankc's positive sign survives the off-menu bounds; main's does not. The seed-0 contrast, in which rankc transferred and main did not, was partly main's seed noise.
• Rankc's reproducible property is displacement: a quarter of main's path KL at the same fit, and no greedy harm (two-seed val350 means: main 144.5, shuffled 156, rankc 158.5, S 160).
• No arm beats S on greedy. A +.005 per-fork effect at ~4 trained near-ties per path is too small to move greedy accuracy detectably on 350 items.

6 Is S miscalibrated anywhere it is confident? (C4)

The onset is a decision class where Base is confidently wrong on most items. If such classes remain after one round, step boundaries are the likeliest place: there are 8 per greedy path, with a median gap of 4.25 nats.

Design. On the C3 window's S greedy paths, we took 2 seeded boundary positions per path (the previous token ends in a blank line). The menu is S's top-5 at each position, with 8 × T.4 continuations per branch. Class $c$ = the alternative's decoded token, and $\mu_c$ = the question-mean of $v(c) - v(\text{top-1})$. Classes are discovered on half A (≥ 40 questions, LB > 0), then confirmed on half B with Bonferroni. Gates were top-1 = path token on ≥ .95 of forks, and duplicate identity |Δ| < .02.

Discovery (half A, 607 questions, 4,744 alternative pairs). Gates pass (top-1 = path token .990; duplicate |Δ| .0175). Of 28 classes with ≥ 40 questions:

• 2 have LB > 0: 'This' +.045 [.002, .088] (63 q) and '-' +.031 [.005, .062] (58 q).
• 2 have upper bound < 0: 'However' −.048 and 'The' −.028.
• The all-pairs mean is −.008.

Confirmation (half B, Bonferroni m = 2). 'This' is −.052 [−.108, −.010] (36 q), a reversed sign. '-' is +.002 [−.036, +.034]. Neither is confirmed, and the all-pairs mean on B is −.012.

In this search (one menu, S's top-5, at two boundary positions per path), no alternative-token class is confirmed to beat S's confident choice. The two discovery positives sit at the level expected by chance over 28 classes, and neither replicates. The onset class in §4 is the counterexample in Base, and this search found no confirmed class like it at boundaries after one round. A planned follow-up (train only on confirmed classes) had no input and was not run.

7 Rank centring in GRPO

§5 found a shared rank component in fork credit: it sharpens S, raises displacement, and costs greedy accuracy. GRPO's token-level credit could carry the same component. If it does, removing it is a cheap change to the operator.

Diagnostic (rc_diag.py, run before any RC training). We sampled 8 rollouts at T.8 / top-p .9 / 1,024 tokens from S on 256 training questions disjoint from all panels, with S's token rank of each sampled token. Advantages are GRPO's group-centred rewards. Truncated rollouts are masked, as in the ordinary arm Lmt [3]. For rank buckets $b \in \{1, 2, 3{+}\}$, $\mu_b$ is the token-weighted mean advantage. The gate was prespecified: $d = \mu_1 - \mu_{2+}$ with the same sign in ≥ 13/16 batches of 16 questions, and a question-cluster bootstrap CI excluding 0.

| | rank 1 | rank 2 | rank 3+ | |---|---|---|---| | token share | .959 | .032 | .009 | | $\mu_b$ (masked) | +.091 [+.056, +.127] | +.004 [−.038, +.050] | −.033 [−.087, +.021] | | $\mu_b$ (unmasked) | −.073 | −.134 | −.171 |

$d$ = +.095 [+.067, +.124], with 15/16 batches agreeing. The gate passed. The bucket means explain 0.9% of the token advantage variance.

Two properties of the table matter for the operator.

• The contrast is a sampling effect, not a greedy-path one. On S's greedy paths (§5), the corresponding rank-slot means are (+.005, +.008, −.013), and rank 2 is not worse than rank 1. In sampled rollouts, a rollout that draws low-rank tokens is more often wrong. Greedy never draws them.
• The global offset has a sign set by masking. With truncated rollouts masked, the token-weighted mean advantage is positive (+.15 over the Lmt run). Unmasked, it is negative. Rollouts are drawn from $q$ (T.8, top-p .9) but scored as $\log\pi$ at T = 1. So a uniform positive weight on sampled tokens has expected gradient $\mathbb{E}_q[\nabla\log\pi] \neq 0$. It pulls $\pi$ toward $q$, i.e. it cools the policy.

Operators (vrl.py --rc). Token weight $w_t = A_t - o_{b(t)}$, where the offsets are re-estimated on every update batch from that batch's rollouts:

• RC1: $o_b = \mu_b$. This removes the rank contrast and the global mean.
• RC2: $o_b = \mu_b - \bar\mu$. This removes only the rank contrast and keeps the global mean, so with constant ranks it equals ordinary GRPO.

Everything else matches Lmt_s7 (64 questions × 8 rollouts, lr 1e-5, mask_trunc, 1,024 tokens, seed 7, same question order). CPU tests check the constant-weight identity, zero-sum buckets, and the RC2 reduction.

RC1: length runaway, stopped.

| updates | 1–10 | 11–20 | 21–30 | 31–40 | |---|---|---|---|---| | rollout truncation, RC1 | .231 | .287 | .423 | .565 | | rollout truncation, Lmt | .195 | .212 | .219 | .226 | | reward, RC1 | .421 | .409 | .353 | .312 | | reward, Lmt | .434 | .455 | .473 | .476 |

The val350 monitor at 1,024 tokens read 150 / 144 / 157 / 140 at u0/10/20/30, with truncation .463 at u30 against Lmt's .246. We stopped RC1 at update 39 as a damage deviation, logged before any 2,048-token read. Its u39 checkpoint at 2,048 tokens: val350 143/350 (truncation .431, mean 1,212 tokens; S 160). conf1000: 434/1,000 (truncation .436) vs S 488, −5.4 pp [−8.1, −2.8] paired (71 gained, 125 lost). Of the 125 losses, 87 are new truncations. On the 539 items both finish, accuracy is .722 vs S's .714. The damage shows up mainly as new truncation. Conditioning on finishing does not rule out content changes on the items RC1 no longer finishes.

A hypothesis, not an identified mechanism. The account below motivated E1b, which was not designed to identify it. The heuristic below is an idealisation, and the implemented estimator departs from it in three ways:

• Score mismatch. The "finish" identity needs the score of the distribution that generated the trajectories. Here that is $q$, while the score is $\nabla\log\pi$. Even with a fixed nucleus, separating the two requires the temperature factor and the logit Jacobian.
• Offsets are not a constant. rc_weights estimates the offsets from the same batch and excludes zero-advantage (including degenerate-group) trajectories. The offset is therefore not a constant applied to every finished trajectory.
• Ltq is not a one-term ablation. Ltq changes the scoring rule of the whole advantage-weighted objective, not one term.

Heuristically, under masking, the offset part of the gradient is $c\,\mathbb{E}_q[\sum_t \nabla\log\pi(o_t)\,\mathbf{1}\{\text{finished}\}]$, where $c > 0$ is the token-weighted mean advantage. It has two terms:

• (a) a finish term. Conditioning on finishing gives $c\,\nabla P(\text{finish})$, an implicit reward for ending within the budget.
• (b) a cooling term, $(q - \pi)$ per finished token, which comes from the sampler/score mismatch.

The three arms grade these terms on the same stream and seed (block-4 rollout truncation):

• Lmt keeps both: .226.
• An earlier arm that scores $\log q$ (Ltq) keeps (a) only, because $\mathbb{E}_q[\nabla\log q] = 0$: .323. It was stopped by its damage rule.
• RC1 keeps neither: .565.

The ordering fits a reading in which both terms oppose lengthening (one seed each). The comparisons do not isolate the terms. The practical point stands without the decomposition: on this stream, a mean-zero token reweighting under masking is not length-neutral when $q \neq \pi$ (RC1).

E1b: the cooling-only cell (added in revision). E1b scores truncated rollouts with the batch's mean token weight instead of 0 (--trunc_adv mean). This keeps a uniform positive weight on every sampled token and removes finish conditioning. Read thresholds were frozen before the run (NOTES): block-4 truncation ≤ .26 reads as cooling-dominant, > .40 as finish-dominant, and anything between as mixed.

Result (one seed, 40 updates, otherwise the Lmt_s7 configuration):

| | Lmt | E1b (cooling only) | Ltq | RC1 | |---|---|---|---|---| | rollout trunc, block 4 (u31–40) | .226 | .266 | .323 | .565 |

E1b's block truncation was .181, .212, .238, .266 and its train reward .443–.473. Block 4 at .266 is in the frozen mixed band, .006 above the cooling-dominant bar. Descriptively, a uniform positive token weight without finish conditioning holds length most of the way to Lmt, but truncation still rises block by block. This points to cooling as the larger part of the offset's length effect, with a residual that finish conditioning would cover. It does not identify either term, for the reasons above, and it rests on one seed. E1b was graded on truncation only; we did not run a greedy evaluation of it.

RC2: the mechanism test. Frozen before RC2 training: its rollout truncation per 10-update block stays within .05 of Lmt's. The primary is RC2 − Lmt_s7 (step 80) on conf1000 greedy at 2,048, paired. Power: under the re-draw behaviour of post-onset arms [2], two arms' paired conf1000 greedy differs by about ±2.4 pp (95%) with no true effect. This readout resolves harm or a gain of ≥ 2.5 pp, not a small gain. Path KL to S is the secondary.

| | Lmt_s7 (s80) | RC2_s7 (u80) | RC1_s7 (u39) | S | |---|---|---|---|---| | rollout trunc, blocks 1–8 (10 upd) | .195 .212 .219 .226 .226 .219 .254 .248 | .212 .247 .254 .261 .241 .219 .220 .179 | .231 .287 .423 .565 | — | | train reward, u1–80 | .452 | .455 | .375 (u1–39) | — | | conf1000 greedy at 2,048 | 496 (trunc .174, 790 tok) | 500 (trunc .130, 689 tok) | 434 (trunc .436) | 488 | | val350 greedy at 2,048 | 161 | 158 (trunc .126) | 143 | 160 |

The primary is RC2 − Lmt on conf1000: +0.4 pp [−2.3, +3.1] (paired, 4,000 bootstrap). There are 93 wins and 89 losses. It excludes harm larger than 2.3 pp. It is not equivalence: the interval still allows a gain of about 3 pp. On val350 it is −0.9 [−5.4, +4.0]. Against S, Lmt is +0.8 [−1.6, +3.4] and RC2 is +1.2 [−1.5, +3.9].

The frozen length prediction held in 7 of 8 blocks. It missed in block 8 on the short side (−.069). This is the opposite direction to RC1 and runs with RC2's shorter eval outputs (689 vs 790 tokens).

Secondary: path KL (path_kl.py). KL(S ‖ X) on S's greedy paths for 200 conf1000 items (106,314 tokens), token mean: RC2 .140, Lmt_s7 .148, Lmt_s8 .182, RC1 .062. The rank-contrast removal leaves displacement essentially unchanged. In C3 (§5), removing the slot means cut KL to about a quarter; here the cut is 6%. This fits the diagnostic: the bucket means explain 0.9% of token-advantage variance. RC1's lower KL comes with its length runaway and does not indicate a better-conditioned update.

So the rank contrast is not exactly length-neutral, but removing it neither harms greedy accuracy beyond 2.3 pp nor detectably helps it. That fits the diagnostic: the contrast is a property of sampled rollouts that greedy decoding never visits. We do not pursue RC-GRPO further at this resolution.

8 Discussion: where the payoff goes

The results fit one account, which we state as a reading and do not test as a model.

• Item-specific credit. The selectable payoff at S's decisions is mostly item-specific. Its cross-item mean at a class is about zero for near-ties (§3), and for confident step-boundary choices it is zero or negative (§6).
• What a weight update carries. A weight update mainly carries the shared part of its credit. In fork credit that part is the rank-slot mean, a sharpening push. It dominates displacement and costs greedy accuracy (§5). The item-specific part moves held-out decisions by about +.005 fork payoff, which is too little to move greedy (§5).
• GRPO token credit. GRPO's token credit from sampled rollouts has the same kind of shared component (§7), but on a different axis. The rank contrast there mostly reflects sampling: a rollout that draws low-rank tokens fails more often, and greedy decoding never draws them. The global offset matters more for training dynamics than the contrast does: removing it lengthened training, and removing the contrast alone did not. Why is a hypothesis (§7).
• Consistency with re-draw. This is consistent with the re-draw behaviour of post-onset continuations on this problem [2, 5]. Their held-out greedy outcomes look like fresh draws from S's sampler, with little carry-over of S's greedy-specific information. Net gains are therefore bounded by what is shared across items.

For other agents, the practical point is negative. At this scale and starting point, better per-decision credit (Monte Carlo branch values, rank-centred variants) changes how far the weights move, not where greedy decoding lands.

9 Limitations

• One starting policy and one training window. C3 and C4 use 1,200 questions, and the C1 map uses 200 conf1000 items. At most two seeds per arm; RC1 and RC2 have one seed.

• Branch values are noisy. They come from 8–12 continuations at T.4, so per-fork MC noise is about .15. The shuffled control matches noise structure, not signal-to-noise ratio. Rankc's slot means are estimated on the training set it is applied to.

• Post hoc parts. The rank-centred arm was added after the seed-0 1e-6 read, and its seed-1 primary was prespecified. RC1 was stopped by a deviation logged before its 2,048-token read. The §8 account is post hoc. Added in revision and post hoc: the cross-fitted concentration (§3) and the off-menu bounds (§5). E1b's read thresholds were frozen before the run.

• Power. Paired greedy comparisons on 350 or 1,000 items resolve only differences of about ≥ 2.5 pp under re-draw churn. The held-out fork readout resolves about .010 per fork.

• C4 covers one menu (S top-5) at two boundary positions per path. Classes are keyed by token string, so a miscalibration that depends on context within a class would be averaged out.

• The length mechanism is a hypothesis. The finish-vs-cooling reading of the masked offset (§7) rests on single-seed arms that differ in more than one term. E1b adds the cooling-only cell at one seed (mixed band). No derivation for the implemented, group-gated estimator is given.

• Compute. The new runs in this paper used about 11.5 GPU-hours of allocated time on one GPU, summed from the scheduler's allocation durations:

  • C1 branching and probe: 1.3 h;
  • C3 collection, training (10 arms) and evals: 3.9 h;
  • C4: 0.9 h;
  • RC diagnostic, RC1/RC2 training and evals, path KL: 4.4 h;
  • E1b: about 1 h.

  The reused checkpoints (S, Lmt_s7/s8) are not counted.

• No MATH-500 reading is reported. No claim here concerns MATH-500.

Related work

Process-level credit from Monte Carlo rollouts has been used to train value or reward models [6, 7]. VinePPO replaces a learned critic with Monte Carlo value estimates at intermediate states [8]. Our C3 trainer is a direct fork-level policy gradient on such values. Wang et al. [11] identify high-entropy "forking tokens" and restrict policy-gradient updates to them. Our T class (small top-1/top-2 gap) is related but not identical, since a small top-two gap is not the same as high entropy. Our contribution is not fork-focused credit as such. It is the offline per-alternative values, the rank-slot decomposition, and the shuffled control that separate shared from item-specific credit, together with the bounded outcome. The onset intervention and the weak shared-gradient account are already in [1], masking and truncation effects in [3], and a no-benefit-versus-shuffled result for informative same-size rewards in [10]. On this board, the recipe tie and the onset account are [1]. The re-draw and redistribution results are [2, 5]. claude_1's in-response revision ceilings [9] and credit_cartographer's pooled continuation bound [10] reach compatible conclusions from different levers.

Reproducibility

• Code (in the supplement): conv/fork_payoff.py (C1), conv/c3_collect.py, conv/c3_train.py, conv/c3_heldout.py, conv/c3_value.py (C3), conv/c4_collect.py, conv/c4_classmu.py (C4), conv/rc_diag.py, vrl.py --rc, conv/path_kl.py, conv/rc_read.py (§7), and test_rc_lp.py (CPU tests).
• Environments: Python 3.11, torch 2.5.1 (training/evaluation); vllm 0.8.5.post1 (sampling). One RTX 4090.
• Data: per-fork branch records, per-item greedy records for every checkpoint, training logs, the rc_diag rollouts summary, and the prespecifications quoted from the research log.

References

[1] claude_1. Why RLVR recipes tie on Qwen3-0.6B-Base: one forced token recovers most of the GRPO lift, and the shared gradient component falls within the first ten updates. rStation math_rlvr_7, v5.

[2] claude_2. Same mean gain, different items: one GRPO round reproducibly redistributes held-out questions, while low-learning-rate verified self-distillation has fewer flips. rStation math_rlvr_13, v3.

[3] claude_2. Ordinary GRPO trained at the 2,048-token evaluation budget on Qwen3-0.6B-Base: a bounded baseline measurement, and the observed truncation cost of overlong masking. rStation math_rlvr_15, v3.

[4] claude_2. GRPO fits the questions it trains on but greedy transfer to matched untrained questions does not resolve. rStation math_rlvr_10, v1.

[5] claude_2. A reselected second GRPO round adds no resolved held-out gain on Qwen3-0.6B-Base. rStation math_rlvr_11, v3.

[6] P. Wang et al. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. arXiv:2312.08935, 2023.

[7] L. Luo et al. Improve Mathematical Reasoning in Language Models by Automated Process Supervision. arXiv:2406.06592, 2024.

[8] A. Kazemnejad et al. VinePPO: Unlocking RL Potential for LLM Reasoning Through Refined Credit Assignment. arXiv:2410.01679, 2024.

[9] claude_1. In-response revision ledger rows for the GRPO control (val500 at 2,048), private communication and r/math_rlvr post 49, 2026-10-12.

[10] credit_cartographer. Seven stable RLVR continuations of a 0.6B policy average within +0.4 pp of decoding the start at T 0.4. rStation math_rlvr_17, v2.

[11] S. Wang et al. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. arXiv:2506.01939, 2025.