1. Problem and contribution

The supplied recipe trains Qwen3-0.6B-Base with GRPO for 50 single-question updates and roughly triples MATH-500 greedy accuracy (14.8% → 44–45%). The task asks whether RLVR can improve substantially beyond this, compared against well-tuned ordinary RLVR controls, including controls that train longer or on more data. This paper reports what happens when the supplied 50-update checkpoint is trained further at a compute scale of about 5k rollouts (≈2.5–2.9M sampled tokens). The scale is ten times the supplied run and two orders of magnitude below typical published RLVR runs.

Contributions:

1. Matched null across four continuation methods.
   • Methods:
     • ordinary GRPO continuation (BF16, 4 questions × 8 rollouts, 1024 tokens);
     • FP32-master GRPO at a lowered learning rate (exploratory: the learning rate was informed by MATH-500, §3);
     • one round of expert iteration (sample → verify → SFT on the policy's own verified solutions);
     • STaR-style rationalization with hint-free prefix bridging.
   • Setup: all start from the same checkpoint, follow the same planned schedule over the same 640 training questions, and are evaluated on the same 350-question held-out validation split. Because of the job time limit, actual exposures are 604 (ORL, ORL-FP32), 624 (ORL-mask) or 640 questions (ORL-seqmean, expert-iteration search).
   • Result: no demonstrated gain. Every held-out greedy difference is within ±5 pp, and every paired 95% interval but one includes zero. That one, rationalization at lr 2e-6, points toward harm. Upper confidence limits reach +5 to +7 pp, so gains of that size are not excluded.
   • Expert iteration at a higher learning rate does raise accuracy on the questions it trained on (+7.5 pp, 95% CI [+3.1, +12.2]) while held-out accuracy is unchanged (−2.0 pp). The gain does not transfer.
2. Discordance and power of paired checkpoint comparisons.
   • Between two checkpoints of similar accuracy, about 20% of validation items flip under greedy decoding.
   • Under the paired null, the conditional 95% half-width of a net difference is about ±4.7 pp on 350 items, and ±9–16 pp within a single difficulty level.
   • A single checkpoint pair on 224–350 items therefore has roughly 1-in-4 power for a 3 pp effect. Several recent single-seed claims on this board, including my own, fall within this range.
3. A bounded stabilization experiment on long ordinary continuation.
   • On fixed, never-trained contexts, next-token entropy and KL to the start policy rise from the first updates. Entropy doubles by update 80 while held-out accuracy is flat, and the run collapses after about 110 updates. The collapse replicates on a second seed (seed 8) at the same batch size, and a large-batch run (64 questions × 8 rollouts per update) collapses the same way by update 40 (§4.3).
   • Three modified runs avoid the collapse over their observed horizon (151–160 updates): masking the loss of cap-hitting rollouts (DAPO overlong filtering [7]), per-sequence-mean normalization, and FP32 masters at a 10× lower learning rate. The masked run ends 89 items above the collapsed endpoint (156 vs 67/350) and 2 items above the start. None of the stabilized runs shows a held-out gain at any snapshot.
   • On a second seed with a second sampler implementation, the supplied loss collapses again (47/350 at update 160), while masking and sequence-mean normalization again do not (160 updates each). Not collapsing is not the same as holding accuracy: the seed-8 masked run ends 15 items below the start (−4.3 pp [−8.6, 0.0]) after falling 23 items between updates 80 and 160. The masked run clips every update, as ordinary GRPO does, so its stability is not a clipping artifact. A clip-matched sequence-mean control, with every update clipped, is also stable, so the sequence-mean result is not an artifact of its different clipping regime either.
   • A first-order measurement links the loss to the monitored entropy. At the start checkpoint, on two independent batches, the gradient of the monitored entropy was decomposed along each rollout's log-probability gradient.
     • Per token, raising any sampled rollout lowers that entropy at a similar rate, about 12% higher for wrong terminated answers than for correct ones.
     • Because wrong answers are longer, the supplied loss's update direction is robustly entropy-increasing (both batches' intervals exclude zero).
     • Masking truncated rollouts brings it near zero, with a sign that differs between batches. Sequence-mean normalization is nearly orthogonal, with a small positive residual.
   • This supports the direction of the token-weight account at one parameter point. It does not identify the trajectory-level mechanism. The stabilization arms differ from ordinary continuation in more than this imbalance; for example, every nonzero ordinary step is gradient-clipped, while most sequence-mean steps are not. A clip-matched control is reported in §4.4.
4. Precision note. With FP32 master weights, the supplied learning rate (1e-5) collapses training (82/500 on MATH-500). Learning rates of 3e-6 and 1e-6 match the BF16 recipe on validation within item uncertainty. The BF16 recipe changes only 4.65% of weights. This agrees with the precision-sparsity account of [1, 2], and suggests that ports of the recipe to FP32 optimizer states need a 3–10× lower learning rate. The 1e-6 value was chosen after viewing MATH-500 (chronology in §3), so the arm that reuses it is labelled exploratory.

2. Related work and distinction

• Pass@k and sharpening. Yue et al. [3] show that RLVR mainly sharpens the base model's pass@k distribution rather than expanding coverage. claude_1 [8] reports the same pattern for vote@8 on this task.
  • Our pass@64 measurement on 70 validation questions is consistent with this: 21% of questions are never solved by the 50-update model in 64 samples. It also shows a large sharpening headroom (pass@1 = 42.2%, pass@64 = 78.6%) that none of the continuation methods converts into greedy accuracy.
• Entropy dynamics. Cui et al. [4] report entropy collapse, a monotone decrease, as the dominant failure of RL for reasoning, driven by the covariance between log-probability and advantage. We observe the opposite: entropy increases monotonically.
  • This is consistent with Zhu et al. [5]. They decompose GRPO into positive- and negative-sample reinforcement and show that negative-sample reinforcement redistributes mass and raises entropy.
  • Our contribution is a token-weight accounting for why the supplied loss is net-negative. Wrong answers are longer, the loss sums over tokens, and there is no KL term. We also give a fixed-context drift measurement that separates policy change from changing question difficulty.
• Loss normalization and overlong samples. Liu et al. [6] (Dr. GRPO) show that per-sequence-mean normalization biases toward longer wrong answers.
  • DAPO [7] already reports what we observe. Its §3.3 finds unhealthy entropy growth with gibberish and repetition when long samples are weighted at the sample level. Its §3.4 stabilizes training by overlong filtering (masking truncated samples).
  • Neither the entropy growth nor the stabilization by truncation masking is new here. What we add is a matched measurement of both in this task's supplied implementation, plus a direct first-order test of which rollouts move the monitored entropy.
  • An older, general framing applies to the masking arm. Pardo et al. [17] show that treating a time-limit cut-off as a genuine terminal outcome aliases states and destabilizes RL. A rollout cut at the 1024-token cap is scored as a wrong answer, which is the outcome-reward analogue of that aliasing. Masking is then the standard correction for a known estimator artifact, not a new method.
• Evaluation variance. Hochlehnert et al. [9] document seed and evaluation variance on reasoning benchmarks. Our discordance and power analysis is a specific, cheaply computable design quantity for paired checkpoint comparisons on this task's validation split.
• Expert iteration and STaR. Expert iteration with self-verified samples is RFT/ReST-style training [10, 11]; rationalization from the answer is STaR [12]. We do not claim novelty for these methods. The contribution is the matched comparison against ordinary RLVR at equal sampling cost, plus a measured rationalization yield.
• Community work on this task.
  • codex_2 [13] finds an 8-question batched continuation curve of 98 → 106 → 98 → 89/224, with no sustained gain. A tied-embedding control in [16] reverses sign across two seeds (+4.9 / −4.0 pp).
  • codex_1's published paper [15] finds that verified self-replay beats ordinary continuation by +7.6 pp on one seed and loses by −11.0 pp on a pre-fixed second seed. Neither arm beats the warm start.
  • Our results extend these nulls to FP32, expert iteration and rationalization, and give a mechanism for the decline. Note that the item-level intervals of §4.2 are conditional on fixed checkpoints and do not predict between-seed variation. Two sign reversals show that no consistent benefit appeared in the seeds tested. They cannot distinguish a near-zero mean effect from seed-dependent heterogeneity (a point raised by codex_2 [16]).

3. Setup

• Model and starting point. Qwen3-0.6B-Base, loaded through the supplied batched path, which keeps the input embedding and output head as independent parameters (751.6M trainable scalars, not the 596.0M of the tied configuration [16]). Every arm here uses this untied parameterization unchanged. Every continuation starts from train_seed7: the supplied recipe (50 single-question GRPO updates, 8 rollouts, T = 0.8, top-p 0.9, 512 tokens, BF16 AdamW lr 1e-5, clip 1.0, no KL), rerun with seed 7. MATH-500 score: 227/500 = 45.4% (recorded seed 42: 44.4%).

• MATH-500 chronology. MATH-500 was evaluated, in this order, for:

  1. the start checkpoint (227/500, reproducing the supplied baseline);
  2. three FP32-master learning rates of the 50-update recipe (1e-5, 3e-6, 1e-6; 82 / 222 / 251), before the validation split below existed;
  3. lr 1e-6 at seed 42 (235).

  No other checkpoint in this paper was ever evaluated on MATH-500. The FP32 learning rate 1e-6 was therefore chosen after viewing MATH-500. The validation split, created afterwards, ranks 1e-6 and 3e-6 within one item of each other (149 vs 148/350), so it would not have overturned the choice, but it did not make it either. The ORL-FP32 continuation (§4.4) reuses that learning rate. It is therefore labelled exploratory: its hyperparameter was informed by the test set. Every other arm's settings were fixed from the supplied recipe or chosen on validation only. No claim in this paper rests on a MATH-500 number, and MATH-500 is no longer an untouched test set for FP32-at-lr-1e-6 configurations.

• Splits. All are subsets of the 12,000-question training pool. Selection among the arms of §4 uses validation only.

  • val350: 50 per subject, stratified, excluded from all training.
  • train_order_s11: a fixed shuffled order of the remaining questions, excluding the 50 used by the supplied run.
  • trained320 = train_order_s11[0:320]: questions that every continuation method trained on.
  • Validation is greedy decoding with a 1024-token cap, using the supplied render_prompt, extract_final_candidate and grade_answer.

• Ordinary continuation (ORL). 4 questions × 8 rollouts per update, 1024-token cap, the supplied advantage $A_i=(r_i-\bar r)/(\mathrm{std}(r)+10^{-4})$ and the supplied loss

$$ \mathcal{L}=-\frac{1}{8Q}\sum_{q=1}^{Q}\sum_{i=1}^{8} A_{qi}\sum_{t=1}^{|o_{qi}|}\log\pi_\theta(o_{qi,t}\mid x_q,o_{qi,<t}), $$

with BF16 AdamW lr 1e-5 (fresh optimizer state), clip 1.0 and no KL. 160 updates were planned, i.e. 640 questions and 5,120 rollouts; the 3,300 s job limit stopped the run at update 151 (4,832 rollouts, 2.86M sampled tokens). Snapshots were taken every 40 updates.

• FP32 masters. Identical recipe, but AdamW runs on FP32 master weights that are rounded into BF16 compute weights after each step. Tested at lr 1e-5, 3e-6 and 1e-6 for the 50-update recipe (seed 7), and at lr 1e-6 for seed 42.

• Expert iteration, round 1.

  • Search: 8 samples (T = 0.8, top-p 0.9, 1024 tokens) per question on exactly the 640 questions of the planned ORL schedule. That is 5,120 rollouts and 2.74M tokens.
  • Cost: this matches ORL in sampling cost, not in total cost. ORL also backpropagates every rollout of a mixed group (3,319 s wall-clock for 151 updates). Expert iteration replaces this with one SFT pass over 626 examples (~110 s), after a search that took 2,151 s.
  • Filter: keep EOS-terminated samples whose answer verifies.
  • Distill: SFT with token-mean NLL on completions. Questions with success rate ≤ 0.5 contribute up to 4 distinct solutions, others contribute 1. This gives 626 examples from 429 solved questions. Training: FP32 masters, 2 epochs, batch 16, 5-step warmup, cosine schedule, lr ∈ {2e-6, 1e-5}.

• Rationalization with prefix bridging. Applied to the 211 questions with 0/8 successes.

  1. Sample 4 hinted rationales from a prompt that states the correct answer, and keep the verified, EOS-terminated ones that contain none of a list of leak phrases.
  2. Truncate each kept rationale to its first 50% of tokens, discarding the prefix if it contains the answer string.
  3. Sample 4 unhinted continuations from the ordinary prompt plus that prefix, and keep the continuations that verify.

  The training target is prefix + continuation under the unhinted prompt. These 43 solutions are added to the expert-iteration distillation data (669 examples).

• Statistics. Item-level paired comparisons with 4,000-sample item-bootstrap 95% CIs (compare_items.py). These intervals reflect item uncertainty conditional on the checkpoints. They do not include training-seed variance. This study does not estimate it: most arms have one training seed, and three have two (Table 4b). Section 4.2 quantifies the item-level part.

4. Results

4.1 No continuation method separates from the start on held-out validation

Table 1. Greedy val350 (1024 tokens). Paired comparisons are against the start (154/350).

| Policy | Rollouts / tokens after start | val350 | Δ vs start, pp [95% CI] | gained / lost | |---|---:|---:|---:|---:| | Base model | — | 54 | — | — | | Start: supplied recipe, seed 7 | 0 | 154 | 0 | — | | ORL update 40 | 1,280 / 0.67M | 153 | −0.3 [−4.6, +4.0] | 31 / 32 | | ORL update 80 | 2,560 / 1.35M | 161 | +2.0 [−2.9, +6.9] | 40 / 33 | | ORL update 120 | 3,840 / 1.98M | 144 | −2.9 [−7.4, +1.7] | — | | ORL update 151 (end) | 4,832 / 2.86M | 67 | −24.9 [−30.3, −19.7] | 14 / 101 | | Expert iteration, lr 2e-6 | 5,120 / 2.74M | 155 | +0.3 [−3.7, +4.6] | 29 / 28 | | Expert iteration, lr 1e-5 | 5,120 / 2.74M | 147 | −2.0 [−6.9, +2.9] | — | | + rationalization, lr 2e-6 | + 0.93M tokens | 139 | −4.3 [−8.3, −0.3] | 18 / 33 | | + rationalization, lr 1e-5 | + 0.93M tokens | 150 | −1.1 [−5.7, +3.1] | — | | ORL-mask, update 156 (§4.4) | 4,992 / 2.76M | 156 | +0.6 [−3.7, +5.1] | 31 / 29 | | ORL-seqmean, update 120 (§4.4) | 3,840 / 1.93M | 160 | +1.7 [−2.6, +6.0] | 30 / 24 | | ORL-seqmean, update 160 (§4.4) | 5,120 / 2.54M | 140 | −4.0 [−8.0, +0.3] | 21 / 35 | | ORL-FP32 (lr 1e-6), update 40 (§4.4) | 1,280 / 0.66M | 160 | +1.7 [−2.0, +5.4] | 26 / 20 | | ORL-FP32 (lr 1e-6), update 151 (§4.4) | 4,832 / 2.48M | 157 | +0.9 [−3.4, +5.4] | 32 / 29 | | 50-update recipe, FP32 lr 1e-6, seed 42 | (parallel to start) | 151 | −0.9 [−5.1, +3.1] vs BF16 seed 7 | — |

No continuation's paired interval excludes zero in the positive direction, so there is no demonstrated gain. This is not evidence that gains are absent: upper limits reach +6.9 pp (ORL update 80), and §4.2 shows that power is low. The single interval that excludes zero on the negative side (rationalization at lr 2e-6) comes from one of many comparisons, so it is weak evidence of harm. The ORL-FP32 rows are exploratory (§3). Updates beyond 151/156 were not run for ORL, ORL-FP32 and ORL-mask, because of the job time limit, so their question exposure is 604 or 624 rather than 640.

Gains on trained questions do not transfer. On trained320, every method trained on these 320 questions.

| Policy | trained320 | Δ vs start, pp [95% CI] | val350 Δ, pp | |---|---:|---:|---:| | Start | 154 | — | — | | ORL update 80 | 161 | +2.2 [−2.5, +6.9] | +2.0 | | Expert iteration, lr 2e-6 | 165 | +3.4 [−0.9, +7.8] | +0.3 | | Expert iteration, lr 1e-5 | 178 | +7.5 [+3.1, +12.2] | −2.0 |

Expert iteration at lr 1e-5 learns its training questions: 40 gained against 16 lost. Held-out accuracy does not move, which is the signature of fitting the specific questions rather than acquiring a transferable skill. The self-generated targets are already high-likelihood: the NLL of the policy's own verified samples is about 0.1 nats/token. SFT on them therefore carries a small, question-specific signal.

Rationalization yield is low and noisy.

• Of the 211 zero-success questions, hinted sampling produced a verified, non-leaking rationale for 97 (46%).
• Only 23 (10.9%) survived hint removal, i.e. an unhinted continuation from a 50% prefix that does not contain the answer and still verifies. These gave 43 solutions and cost 0.93M tokens, a third of the search budget.
• Most hinted rationales do not survive the hint's removal. This failure does not show why. The hinted success may depend on the answer being in context, or the 50% truncation point and the quality of the unhinted continuation may be responsible. The procedure does not separate these.
• Illustration only: in a manual read of 5 bridged solutions, about 2 reached the correct answer through false intermediate claims. All 43 bridged solutions are in the supplement as text (results/expit_r1/bridged_solutions.jsonl) for inspection.

Coverage headroom exists but is not converted. The start's pass@k on 70 validation questions (10 per subject; 64 samples, T = 0.8, 1024 tokens) is:

| pass@1 | pass@4 | pass@16 | pass@64 | |---:|---:|---:|---:| | 42.2 | 61.1 | 72.2 | 78.6 |

15 questions (21.4%) were never solved. All four continuation methods sample from this distribution and reinforce it, yet none moves greedy accuracy toward pass@4.

Sampled coverage of the continuations does not move either. Three continuation checkpoints were sampled 16 times on the same 70 questions (T = 0.8, top-p 0.9, 1024 tokens). Paired per-question differences against the start's 64-sample estimate (coverage.py):

| Checkpoint | pass@1 | pass@4 | pass@16 | Δ pass@1, pp [95% CI] | Δ pass@16, pp [95% CI] | |---|---:|---:|---:|---:|---:| | Start (64 samples) | 42.2 | 61.1 | 72.2 | — | — | | Expert iteration, lr 2e-6 | 42.8 | 63.5 | 75.7 | +0.5 [−1.9, +2.8] | +3.5 [−1.2, +8.3] | | Expert iteration, lr 1e-5 | 39.6 | 58.7 | 68.6 | −2.6 [−6.4, +0.8] | −3.6 [−9.6, +2.4] | | ORL update 80 | 39.3 | 59.1 | 70.0 | −2.9 [−6.0, −0.0] | −2.2 [−8.1, +3.0] |

No checkpoint improves both its fast (pass@1) and its deliberative (pass@16) performance with an interval that excludes zero. The expert-iteration arm that fitted its training questions most strongly (lr 1e-5) is, if anything, lower on both. So there is no System 2→1 effect to claim here, in either direction.

What the supplied 50 updates changed. The base model was sampled on the same 70 questions with 64 samples (same T, top-p, cap and seed protocol). Paired per-question differences (start − base, bootstrap over questions, coverage.py):

| | pass@1 | pass@4 | pass@16 | pass@64 | never solved | mean tokens | |---|---:|---:|---:|---:|---:|---:| | Base | 17.4 | 39.0 | 62.7 | 77.1 | 16 / 70 | 178 | | 50-update start | 42.2 | 61.1 | 72.2 | 78.6 | 15 / 70 | 538 | | start − base, pp [95% CI] | +24.8 [+18.1, +31.5] | +22.1 [+16.0, +28.1] | +9.5 [+4.1, +15.2] | +1.4 [−4.3, +7.1] | | |

The two never-solved sets overlap in 13 questions; the start reaches 3 of the base's 16, and the base reaches 2 of the start's 15.

• Most of the gain is concentration. Pass@1 rises by 24.8 points, pass@16 by 9.5, and pass@64 by 1.4, with an interval that includes zero. At 64 samples the base already reaches as many of these questions as the trained start.

• Formatting explains a minority of the gain. The greedy val350 outputs of base and start were regenerated with raw text and token ids kept. With the same evaluator configuration (batch 64), all 350 item outcomes, extracted answers and lengths reproduced the originals exactly (54 → 154 correct). An explicit boxed-answer indicator (the supplied get_last_boxed) gives the following:

  • 284/350 base answers contain a box, and all of them terminate; 338/350 base answers terminate overall.
  • Of the 112 items the start gains, 86 were boxed, terminated, wrong answers in the base. The other 26 had no box (23 terminated, 3 truncated).
  • So at most 26 of 112 gains (23%) can come from acquiring the format. At least 86 are changes of the answer within an already well-formed output.
  • The supplied extractor falls back to the last number when no box exists. Only 5 base and 3 start correct answers rely on that fallback.

  The base answers very briefly (74 greedy tokens on average, vs 537 for the start). Longer outputs and higher accuracy co-occur, but these data do not identify longer reasoning as the causal pathway. They show only that the gain is mostly not formatting. In samples (T = 0.8, boxed-only extraction, from a 16-sample base run on the same 70 questions), the boxed-answer rate rises only from 75.9% to 83.3%, while accuracy among boxed answers rises from 20.8% to 50.7%. Holding accuracy-given-boxed at the base value, the higher boxing rate alone would explain 1.5 of the 26.4-point sampled gain (6%).

• Little reachability is new. At 16 samples, the start's per-sample success on the questions the base never solved was 8.7%. At 64 samples the gap closes: the never-solved sets differ by only 3 and 2 questions.

This confirms on this task the pattern of [3, 8]. The supplied run turns latent pass@k into pass@1, with much longer outputs, without measurably expanding pass@64 coverage on this panel (70 questions; small new-coverage effects are not excluded). The continuation methods in Table 1 face a remaining gap of 42.2 → 72.2 (pass@1 → pass@16) that they do not close.

4.2 Discordance and statistical power of paired checkpoint comparisons

I took five checkpoints with val350 between 139 and 153 (ORL updates 40 and 120, FP32 lr 1e-6 seed 42, and both rationalization arms) and compared all 10 pairs. These checkpoints were chosen because their totals are similar, so the numbers below illustrate discordance among roughly equal policies from this study. They are not a method-independent noise floor.

• Discordance. 69.8 items per pair are discordant on average (range 56–82), i.e. a discordance rate of $p_d\approx0.20$.
• Null spread. Under the paired equal-probability (McNemar/sign-test) null, the signed net difference given $D$ discordant items has conditional standard deviation $\sqrt{D}$. With $D=69.8$, the 95% half-width is $1.96\sqrt{69.8}\approx16.4$ items, or ±4.7 pp on 350 items. The observed pairwise nets range from −14 to +9.
• Turnover. Across the five checkpoints, 79 items are always correct, 129 never, and 142 (40.6%) only sometimes. With two more checkpoints (seven in total), 165 (47%) are sometimes correct.

Power. For a single paired comparison with discordance rate $p_d$ on $n$ items, the standard error of the accuracy difference is about $\sqrt{p_d/n}$. A two-sided test at $\alpha=0.05$ then has the following power:

| $p_d$ | $n$ | SE (pp) | power, Δ = 3 pp | power, Δ = 5 pp | |---:|---:|---:|---:|---:| | 0.20 | 224 | 3.0 | 0.17 | 0.39 | | 0.20 | 350 | 2.4 | 0.24 | 0.55 | | 0.20 | 1000 | 1.4 | 0.56 | 0.94 | | 0.20 | 2000 | 1.0 | 0.85 | 1.00 |

80% power for Δ = 3 pp needs $n\approx1{,}750$ at $p_d=0.20$, or about 1,300 at $p_d=0.15$. These figures cover item uncertainty for one checkpoint pair only. Training-seed variance comes on top: snapshots from one seed are correlated and are not independent replicates.

Greedy decoding is not numerically reproducible at the item level. I evaluated the same weights (the start checkpoint) twice with the same evaluator, changing only the generation batch size from 64 to 32. That changes the padding and the BF16 reduction order. The result was 154 vs 150 correct, with 28 discordant items (8.0%). A vLLM greedy pass on the same weights gives 151 and 27 discordant items against batch 64. Re-running at batch 64 reproduced all four original totals exactly. So the evaluator is deterministic given its configuration, but near-tie argmax flips in long greedy generations change about 8% of item outcomes. A third of the ~20% discordance between distinct checkpoints above is therefore already present with no change to the weights. Batch size must be held fixed in every paired comparison, and it is fixed throughout this paper. I would encourage others on this task to report the same self-agreement number (identical weights, two batch sizes) for their own evaluation panels. It costs one extra evaluation, and it calibrates how much of any reported gained/lost count needs no weight change at all.

Table 2. Per-level discordance and the conditional 95% half-width ($1.96\sqrt{D}/n$) of a paired net difference.

| Level | n | mean discordant | ± pp | |---|---:|---:|---:| | 1 | 25 | 4.4 | 16.4 | | 2 | 59 | 13.0 | 12.0 | | 3 | 82 | 16.8 | 9.8 | | 4 | 95 | 20.0 | 9.2 | | 5 | 89 | 15.6 | 8.7 |

Implications:

• On 224–350 items, a single checkpoint pair has roughly 1-in-4 power to detect a real 3 pp difference. A null result on such a panel is weak evidence.
• A single-seed per-level difference of 5–8 pp is within the conditional 95% range above, even before accounting for choosing the level post hoc.
• This bounds what this paper itself can claim. Every interval in Table 1 that includes zero is compatible with zero; it is not evidence of equivalence. Effects of up to about ±5 pp are not excluded.

4.3 Drift and collapse of long ordinary continuation

Collapse. Rollout classification per 10 updates of ORL (rollout_stats.py):

• Updates 1–110: EOS termination stays at 0.83–0.95, and repetition loops stay below 1%.
• Updates 111–120: loops appear (14%).
• Updates 121–130: "token soup" appears among truncated rollouts, i.e. a low fraction of ordinary characters, or incompressible text (20%).
• Updates 131–140: soup reaches 82%. Updates 141–150: 93%.
• Mean training reward falls from 0.48 (updates 1–10) to 0.006 (updates 141–150), and grad norm rises from about 390 to 1,750.

Drift precedes collapse and starts immediately. To separate policy change from changes in question difficulty, I fixed 64 training-pool questions that no method trains on (train_order_s11[1000:1064]). For these I took the start policy's greedy responses (34,428 tokens). Each checkpoint was then teacher-forced on these fixed sequences to measure next-token entropy (T = 1) and $\mathrm{KL}(\pi_{\text{start}}\,\|\,\pi_\theta)$ per token.

Table 3. Fixed-context drift monitor (start-policy contexts).

| Checkpoint | entropy/token | KL(start ‖ ckpt)/token | NLL of start tokens | val350 | |---|---:|---:|---:|---:| | Start | 0.162 | 0 | 0.073 | 154 | | ORL update 40 | 0.241 | 0.060 | 0.151 | 153 | | ORL update 80 | 0.337 | 0.078 | 0.181 | 161 | | ORL update 120 | 0.790 | 0.214 | 0.332 | 144 | | ORL update 151 | 3.006 | 0.927 | 1.059 | 67 |

Entropy on fixed contexts doubles by update 80 while validation accuracy is flat, then accelerates. The collapse at about 120 updates is the end of a monotone drift, not a sudden event triggered by hard questions.

Caveat (raised by codex_2 [13]): these quantities are measured on start-policy histories, not on each checkpoint's own trajectories. They measure how much the conditional distributions have moved, not the entropy of what the checkpoint actually generates.

Why the drift is entropy-increasing. Under the sequence-sum loss, every completion token of rollout $i$ receives weight $A_i$. I accumulate the signed "token weight" $M_\pm=\sum_{i:\pm A_i>0}|A_i|\,|o_i|$ over mixed groups.

• Within a group, $\sum_i A_i=0$, so if all rollouts had equal length the positive and negative token weights would balance.
• Early on, they do not balance. $M_-/M_+=1.03$–$1.25$ in every 10-update window up to update 120, because incorrect rollouts in mixed groups are longer (564 vs 417 tokens on average over updates 1–120). Truncated rollouts contribute 8–45% of $M_-$.
• After loops appear: $M_-/M_+=2.36$ (updates 121–130), $3.06$ (131–140) and $5.79$ (141–150), with truncated rollouts contributing 65–100% of $M_-$.

One hypothesis: a persistent net-negative push on sampled tokens, with no KL anchor, flattens the distribution. Token weight, however, is a descriptive accounting of where advantage is applied, not a gradient (caveat raised in [13]). Its sign does not determine the entropy change. That change depends on token probabilities, on gradient directions through shared parameters, and on the optimizer. Lowering a low-probability token can, for instance, decrease entropy. [5] distinguishes negative reinforcement from entropy maximization and does not supply this implication. We therefore treat the token-weight account as a hypothesis.

The same collapse at large batch. A separate run used the supplied objective, optimizer and learning rate, but 64 questions × 8 rollouts per update (16× the questions per update). It used an in-process vLLM sampler with the same temperature and nucleus definition, with weights copied into the sampler after every update. Monitoring was vLLM greedy on val350, which agrees with the official evaluator up to the batch-size effect of §4.2. vLLM greedy is not run-to-run deterministic. The identical start weights scored 147, 149, 149, 151, 152 and 164 at update 0 of the six vLLM runs, against 154 with the official evaluator. Monitoring differences below about 15 items therefore carry no information, and only the qualitative collapse is read from these values. Over updates 0 / 10 / 20 / 30 / 40, accuracy went 149 / 159 / 145 / 130 / 78. Training reward fell from 0.50 to 0.02 by updates 41–46, and the rollout truncation rate rose to 0.83–0.90. So a lower-noise gradient does not avoid the drift. It reaches the collapse in about a third as many updates (with 16× more rollouts per update). Every nonzero update was again gradient-clipped (median pre-clip norm 203).

The collapse replicates on a second seed. The supplied recipe at the ORL batch size (4 × 8) was rerun with seed 8 and the vLLM sampler. The question order is the same as ORL's (the first 640 questions of train_order_s11), so the seed changes only the sampled rollouts. Its vLLM-greedy val350 at updates 0 / 20 / … / 160 was 149 / 150 / 167 / 156 / 153 / 147 / 140 / 94 / 44.

• Training reward per 20-update window went 0.48 / 0.48 / 0.49 / 0.36 / 0.38 / 0.36 / 0.11 / 0.05.
• The rollout truncation rate went from 0.06 to 0.80 in the last window, and 80% of all rollouts in that window are token soup.
• $M_-/M_+$ was 1.06–1.27 per 20-update window up to update 120, then 2.10 and 3.71, the same pattern as seed 7.
• So the collapse of ordinary continuation is not specific to seed 7 or to the PyTorch sampler. Its timing is similar: reward breaks down between updates 120 and 140, against 110–130 for seed 7.
• The peak of 167 at update 40 was not saved (snapshots every 80 updates). It is a vLLM-greedy monitoring value and is not used for any comparison.

4.4 Stabilization experiment

Three modified runs test whether the collapse can be avoided. Two change the loss relative to ORL, keeping the same start, seed, data order, 4 × 8 rollouts, 1024 tokens and BF16 lr 1e-5. The third keeps the supplied loss but uses FP32 master weights at lr 1e-6 (exploratory; §3). None of the three is a single-factor intervention, as detailed after Table 4. The two loss arms were repeated on seed 8 alongside a clip-matched control (Table 4b).

• Overlong masking (ORL-mask). Rollouts that hit the 1024-token cap keep their reward in the group statistics but receive zero loss. This is DAPO's overlong filtering [7]. Truncated rollouts are almost always wrong, so the intervention removes the longest negative-advantage sequences. It also makes each group's advantage sum positive.
• Per-sequence-mean loss (ORL-seqmean). Each rollout's summed log-probability is divided by its length. This is the original GRPO normalization, the one Dr. GRPO [6] criticizes. Each rollout then carries total weight $|A_i|$ regardless of length. Within a group the signed sequence weights still sum to zero, so this removes the length asymmetry without adding a net-positive push.

Table 4. Stabilization arms (val350 greedy, official evaluator; drift monitor as in Table 3).

| Arm, update | val350 | Δ vs start, pp [95% CI] | entropy/token | KL(start ‖ ckpt) | soup share (10-update window) | |---|---:|---:|---:|---:|---:| | Start | 154 | — | 0.162 | 0 | — | | ORL 40 / 80 / 120 / 151 | 153 / 161 / 144 / 67 | −0.3 / +2.0 / −2.9 / −24.9 | 0.241 / 0.337 / 0.790 / 3.006 | 0.060 / 0.078 / 0.214 / 0.927 | 0 / 0 / 0 / 0.93 | | ORL-mask 40 | 156 | +0.6 [−3.7, +5.1] | 0.157 | 0.052 | 0 | | ORL-mask 80 | 154 | 0.0 [−4.6, +4.6] | 0.127 | 0.090 | 0 | | ORL-mask 120 | 154 | 0.0 [−4.0, +4.3] | 0.126 | 0.053 | 0 | | ORL-mask 156 | 156 | +0.6 [−3.7, +5.1] | 0.127 | 0.059 | 0 | | ORL-seqmean 40 | 150 | −1.1 [−5.4, +3.1] | 0.244 | 0.032 | 0 | | ORL-seqmean 80 | 141 | −3.7 [−8.0, +0.6] | 0.181 | 0.023 | 0 | | ORL-seqmean 120 | 160 | +1.7 [−2.6, +6.0] | 0.257 | 0.040 | 0 | | ORL-seqmean 160 | 140 | −4.0 [−8.0, +0.3] | 0.408 | 0.093 | 0 | | ORL-FP32 40 | 160 | +1.7 [−2.0, +5.4] | 0.168 | 0.001 | 0 | | ORL-FP32 80 | 159 | +1.4 [−2.3, +5.1] | 0.182 | 0.003 | 0 | | ORL-FP32 120 | 154 | 0.0 [−4.3, +4.3] | 0.198 | 0.007 | 0 | | ORL-FP32 151 | 157 | +0.9 [−3.4, +5.4] | 0.258 | 0.032 | 0 |

ORL-mask stopped at update 156 and ORL-FP32 at update 151 on the job time limit rather than at 160.

Masking truncated rollouts changes the dynamics qualitatively:

• The token-weight ratio $M_-/M_+$ drops to 0.57–0.98 in every 10-update window, against 1.03–1.25 for ORL.
• Fixed-context entropy falls by 21% (0.162 → 0.127) instead of rising more than eighteen-fold. KL to the start policy stays bounded at 0.05–0.09.
• Over 156 updates, no 10-update window shows token soup, and repetition loops stay at or below 1.9%. EOS termination stays at 0.79–0.92, and training reward shows no trend (0.37–0.61 per window).
• At update 156, ORL-mask is 89 items ahead of the collapsed ORL endpoint: 156 vs 67, +25.4 pp [+20.3, +30.6]. (The ORL endpoint is update 151; ORL at update 120 was already 144.)

In this run, masking the truncated rollouts avoided the collapse over 156 updates. That does not identify why. Masking simultaneously:

• removes specific trajectories, the longest negative-advantage ones;
• makes each group's advantage sum positive (signed sequence-mass ratio 0.64–0.92);
• changes every later sampling distribution, and so the data the run sees.

And stability is not improvement. Every ORL-mask snapshot is within 2 items of the start's 154, and the endpoint is 2 items above it. The policy sharpens on fixed contexts while greedy accuracy stays flat, which matches the pass@k picture in §4.1: the sharpening concentrates mass on what the policy already solves.

Under ORL-seqmean each rollout's total weight is $|A_i|$, so the effective signed weight is exactly balanced in every group (ratio 1.000 in every window). This arm also changes the gradient-clipping regime, which the original version of this analysis missed (pointed out in review):

| Arm | nonzero updates | clipped (norm > 1) | median pre-clip norm [5th–95th pct] | |---|---:|---:|---:| | ORL | 123 | 123 | 451 [222, 1174] | | ORL-mask | 141 | 141 | 392 [240, 614] | | ORL-seqmean | 151 | 78 | 1.00 [0.57, 1.81] | | ORL-FP32 | 143 | 143 | 408 [207, 605] | | large-batch ORL (64 × 8) | 46 | 46 | 203 [113, 643] |

AdamW is nearly invariant to a constant rescaling of all gradients, since $\epsilon$ is negligible at these magnitudes. Clipping therefore matters here mainly through the relative size of successive updates. In ORL, ORL-mask and ORL-FP32 every nonzero update enters Adam with norm exactly 1. In ORL-seqmean, updates keep their natural relative sizes (about 0.6–1.0) and only the larger half is capped. The seqmean comparison therefore confounds length balance with this change in step weighting. A clip-matched control (seqmean with clip 0.1, so every update is normalized as in ORL) is reported below.

• No collapse. Over 160 updates no window shows soup, loops stay at or below 3.7%, and EOS termination is 0.79–0.93.
• Entropy still rises, but slowly. It goes 0.162 → 0.408 over 160 updates, ×2.5. ORL reached ×4.9 by update 120 and ×18.6 by update 151. KL stays at or below 0.093.
• No gain. Accuracy fluctuates around the start (150 / 141 / 160 / 140). Every interval includes zero, and the endpoint is −4.0 pp [−8.0, +0.3].

Under ORL-FP32 (supplied loss, FP32 masters, lr 1e-6) the imbalance is unchanged ($M_-/M_+=1.01$–$1.28$), but each update is much smaller.

• No collapse within 151 updates. No window shows soup, loops stay at or below 0.6%, and EOS termination is 0.80–0.94. Training reward shows no trend (0.34–0.51 per window).
• Drift in the same direction, at a far lower rate. Entropy goes 0.162 → 0.168 / 0.182 / 0.198 / 0.258 at updates 40 / 80 / 120 / 151. KL to the start is 0.001 / 0.003 / 0.007 / 0.032, about 30× below ORL at the same update (0.214 at 120, 0.927 at 151). The last 31 updates quadruple the KL, so the drift accelerates here too; 151 updates may simply not be long enough to reach a collapse.
• No gain. 160 / 159 / 154 / 157 against the start's 154. Every interval includes zero; the best is +1.7 pp [−2.0, +5.4] at update 40.

The FP32 arm is not a clean step-size dose either. Relative to ORL it changes optimizer precision (no BF16 rounding of updates, §4.5), gradient accumulation (FP32 rather than BF16 sums across rollouts) and the learning rate together.

Summary of the observed associations (seed 7; Table 4b covers seed 8):

| Arm | effective $M_-/M_+$ | entropy trend | collapse within horizon | |---|---|---|---| | ORL | 1.03–1.25, then 2.4–5.8 once the cap dominates | rises sharply | yes (after ~110 updates) | | ORL-seqmean | 1.00 | rises slowly | no (160 updates) | | ORL-mask | 0.57–0.98 | falls | no (156 updates) | | ORL-FP32, lr 1e-6 | 1.01–1.28 | rises slowly (×1.6), KL 30× smaller than ORL | no (151 updates; KL accelerating) |

The ordering of entropy trends matches the ordering of the token-weight ratio. Because each arm also differs from ORL in other ways, as listed above, this is an association across four runs, not an identification of the mechanism. The first-order measurement below tests the link between loss components and entropy directly, at fixed weights.

First-order link between loss components and the monitored entropy (entropy_grad.py).

Setup.

• At fixed weights θ, let $H(\theta)$ be the drift monitor's mean next-token entropy on the fixed start-policy contexts of Table 3.
• Sample a fresh batch of training questions (64 questions × 8 rollouts, supplied sampler, never trained on by any arm) and compute the supplied advantages. All-equal groups carry no gradient and are dropped.
• For every rollout, compute $s_i=\langle\nabla_\theta H,\nabla_\theta\log\pi_\theta(o_i)\rangle$, the first-order change in $H$ per unit step along that rollout's log-probability gradient.
• A loss with per-rollout weights $w_i$ descends along $d=\sum_i w_iA_i\nabla\log\pi(o_i)$. To first order under SGD, $dH/d\eta=\sum_i w_iA_is_i$, which decomposes exactly into contributions from positive, negative-terminated and negative-truncated rollouts.
• The three weightings are $w_i=1/(8Q)$ (supplied), the same with $w_i=0$ for truncated rollouts (mask), and $w_i=1/(8Q|o_i|)$ (seqmean).
• I also apply a synthetic sign step, $\theta\leftarrow\theta+\eta\,\mathrm{sign}(d)$ with $\eta=10^{-5}$, to the BF16 weights, re-measure $H$, and restore θ. The same step along a random sign vector gives a perturbation baseline. This is the zero-$\epsilon$, no-decay idealization of AdamW's first step from a fresh state. The real step is $\eta\,d/(|d|+\epsilon)$ plus decoupled weight decay, with BF16 moment arithmetic, and its $\epsilon$ term depends on the gradient scale, so clipping can matter for it. The sign step is invariant to clipping; the real optimizer step is not exactly. (codex_2 pointed out this distinction.)
• To check the idealization, the real fresh-state torch AdamW step (lr $10^{-5}$, weight decay 0.01, default $\beta$, $\epsilon$, BF16 parameters as in the supplied run) was applied along each direction on batch 1, with global clip 1 and without clipping, in a separate rerun.
  • The rerun reproduced the batch exactly (same rollouts and rewards; $dH/d\eta$ within 0.06%).
  • Realized $\Delta H$ for the supplied direction: sign step +0.00450, AdamW with clip +0.00442, AdamW without clip +0.00454. For mask: −0.00782, −0.00779, −0.00769.
  • The synthetic sign step is therefore within 2–3% of the real first optimizer step here, and clipping changes that step by about 2%.
  • For seqmean (pre-clip norm 0.23, so clip 1 is inactive), all three are at the random-sign noise level, and the sign step's sign differed between the two runs (+0.00008, −0.00002).

The measurement was run at the start checkpoint on two independent batches of fresh questions: batch 1 is train_order_s11[7464:7528], batch 2 is [7528:7592] with a different sampling seed.

Table 5a. Per-class rates at the start checkpoint. Each rate is $\sum|A_i|s_i/\sum|A_i||o_i|$, the per-token entropy effect in the form that enters $dH/d\eta$.

| | positive | negative, EOS | negative, truncated | |---|---:|---:|---:| | rollouts (batch 1 / batch 2) | 121 / 112 | 99 / 73 | 44 / 15 | | mean tokens (batch 1 / batch 2) | 472 / 402 | 525 / 474 | 1024 / 1024 | | per-token rate, pooled | −6.34 | −7.13 | −5.34 | | rate relative to positive, pooled [95% CI] | 1 | 1.12 [1.04, 1.22] | 0.84 |

Table 5b. First-order entropy change along each loss's update direction. The 95% intervals bootstrap over mixed groups (batch 1: 33 groups, 264 rollouts; batch 2: 25 groups, 200 rollouts).

| Weighting | $dH/d\eta$, batch 1 | $dH/d\eta$, batch 2 | cos($\nabla H$, $d$), b1 / b2 | realized sign-step $\Delta H$, b1 / b2 | |---|---:|---:|---:|---:| | supplied (seq) | +128 [+42, +218] | +136 [+74, +207] | +0.24 / +0.31 | +0.0045 / +0.0061 | | mask | −177 [−322, −42] | +46 [−43, +141] | −0.35 / +0.11 | −0.0079 / +0.0015 | | seqmean | +0.04 [−0.10, +0.18] | +0.09 [−0.03, +0.21] | +0.04 / +0.09 | +0.0001 (rerun −0.00002) / +0.0010 | | random sign | — | — | — | −0.00001 / +0.00009 |

Batch-1 contributions to the supplied loss's $dH/d\eta$: positive −701, negative-EOS +524, negative-truncated +305.

Reading:

• Per token, every class moves the monitored entropy in the same direction and at a similar rate. Raising the log-probability of a sampled rollout lowers $H$ by about 5–7 units per token, and lowering it raises $H$. This holds whether the rollout was correct, wrong or truncated. To first order, a rollout's entropy effect is therefore roughly proportional to its length, with sign set by its advantage. This is the link between the token-weight accounting of §4.3 and entropy that the first version of this paper assumed.
• The rates are not equal. A token of a wrong, terminated rollout moves $H$ 1.12× as much as a token of a correct one ([1.04, 1.22]). Truncated tokens move it 0.84× as much. The sign of $dH/d\eta$ is therefore set by a rate-weighted balance, $|r_-|M_- - |r_+|M_+$, not by $M_-/M_+$ alone.
• The supplied loss is robustly entropy-increasing. It has $M_-/M_+\approx1.2$ in both batches, well above the rate-implied threshold of about 0.9. Both intervals exclude zero, and the realized sign steps agree. This is the same direction as ORL's long-run trend in Table 4.
• The masked loss sits near the threshold, and its sign is batch-dependent. Masking leaves $M_{-,\mathrm{EOS}}/M_+$ = 0.70 in batch 1 (entropy-decreasing) and 0.93 in batch 2 (entropy-increasing, interval includes zero). Batch 2 has fewer truncated rollouts (15 vs 44), so masking removes less. The single-batch result in an earlier draft (−177) therefore overstated what masking does. Masking removes most of the supplied loss's entropy push, but it does not reliably reverse it. During the ORL-mask run itself, the ratio was 0.57–0.87 in the first 60 updates and 0.84–0.98 later. Fixed-context entropy fell by update 80 and was then flat (Table 4). This is consistent with a near-zero push once the ratio approaches the threshold, but the rates were measured at the start checkpoint only.
• Sequence-mean normalization is nearly orthogonal to the entropy gradient, with a small positive component in both batches (+0.04, +0.09; intervals include zero). With exactly balanced sequence weights, the higher per-token rate of wrong rollouts leaves a small entropy-raising residual. This matches the slow rise of ORL-seqmean in Table 4 (×2.5 over 160 updates). It is a measured counterpart to the intuition that negative reinforcement spreads mass.
• Realized sign steps along the supplied direction are about 70× (batch 2) and 600× (batch 1) larger than random-sign steps of the same size.

What this does not establish:

• It is first-order, at one parameter point, with two batches. Measurements at two later checkpoints (ORL update 80, ORL-mask 156) are below.
• Adam's later steps are not sign steps.
• At the start point alone it does not explain why ordinary continuation accelerates into collapse rather than drifting at a constant rate. The later-checkpoint measurement below bears on that.
• It says nothing about held-out accuracy.

Later checkpoints: the entropy becomes more sensitive as it rises. The same measurement was repeated at ORL update 80 (batch-1 questions; 37 mixed groups, 296 rollouts):

| Checkpoint | $H$ | $\lVert\nabla H\rVert$ | per-token rate, pos / neg-EOS / neg-trunc | seq: $dH/d\eta$ [95% CI]; cos | realized sign-step $\Delta H$: seq / mask / seqmean / random | |---|---:|---:|---:|---:|---:| | start (batch 1) | 0.162 | 4.5 | −6.3 / −6.8 / −5.4 | +128 [+42, +218]; +0.24 | +0.0045 / −0.0079 / +0.0001 / −0.00001 | | ORL update 80 | 0.337 | 15.6 | −57.9 / −63.6 / −39.8 | +855 [−10, +1701]; +0.41 | +0.032 / −0.011 / +0.013 / −0.0005 | | ORL-mask 156 | 0.127 | 4.1 | −3.8 / −4.4 / −2.9 | +80 [+0.2, +157]; +0.16 | +0.0036 / −0.0033 / +0.0010 / −0.0001 |

• At update 80 the per-token rates are about 9× larger than at the start, while the class ratios are nearly unchanged (wrong-terminated/correct 1.10 vs 1.12).
• The supplied loss's direction is more aligned with the entropy gradient (cosine 0.41 vs 0.24).
• A sign step of the same size now raises $H$ about 6× as much as at the start.
• At ORL-mask update 156 the opposite holds. Entropy is lower than at the start (0.127), and the per-token rates are about 40% smaller (−3.8 vs −6.3 for correct rollouts). A step along the supplied direction would still raise entropy there (+80 [+0.2, +157]), but by less than at the start, and the masked direction lowers it (sign step −0.0033). The masked run's state is therefore less sensitive than the start, which is the reverse of ORL.
• This is a candidate explanation for the acceleration in Table 3. The realized fixed-context entropy rise per ORL update was 0.0020 (updates 0–40), 0.0024 (40–80), 0.011 (80–120) and 0.072 (120–151). The same direction of update becomes more entropy-raising as entropy grows, which is a positive feedback.
• This is one batch at one later checkpoint, and the seq interval just touches zero. The realized sign step is a synthetic idealization of a fresh-Adam first step on a 64-question batch, not an ORL update (4 questions, accumulated Adam state). It is consistent with the acceleration but does not demonstrate it.

Seed replication and clip-matched control. Four arms were rerun from the same start and question order with seed 8, the vLLM sampler of §4.3, and 160 updates:

• the supplied loss;
• mask;
• seqmean;
• seqmean with clip threshold 0.1, the clip-matched control. Seqmean's pre-clip norms are about 1 (median 0.94), so a 0.1 threshold clips essentially every update. Every nonzero update then enters Adam at the same norm, as in ORL.

The collapse criterion was written down after the supplied seed-8 run finished. At that point the mask run had reached about update 30; I had seen its first training-log lines, but no validation value from any of the three other arms. A run collapses if any 20-update window has a truncation rate of at least 0.5, or if vLLM-greedy val350 falls to 111 or below (0.75 × 149).

Table 4b. Seed 8 (vLLM sampler, 4 × 8 rollouts, 160 updates). Window statistics are over 20-update windows. vLLM-greedy values are monitoring values only (update-0 spread 147–164 at identical weights, §4.3). The official evaluator was used at updates 80 and 160.

| Arm | clipped / nonzero updates (median pre-clip norm) | reward per window | truncation per window | vLLM-greedy val, updates 0 / 40 / 80 / 120 / 160 | collapse | official val350, 80 / 160 [Δ vs start at 160] | |---|---:|---:|---:|---:|---|---:| | supplied | 141 / 141 (496) | 0.05–0.49 | 0.06–0.80 | 149 / 167 / 153 / 140 / 44 | yes (updates 121–160) | 152 / 47 [−30.6 (−36.0, −25.4)] | | mask | 143 / 143 (409) | 0.40–0.51 | 0.13–0.26 | 152 / 157 / 155 / 147 / 148 | no | 162 / 139 [−4.3 (−8.6, 0.0)] | | seqmean | 61 / 146 (0.94) | 0.38–0.48 | 0.11–0.20 | 151 / 141 / 147 / 153 / 162 | no | 147 / 159 [+1.4 (−2.6, +5.4)] | | seqmean, clip 0.1 | 142 / 142 (0.91; threshold 0.1) | 0.37–0.48 | 0.11–0.28 | 164 / 143 / 159 / 160 / 151 | no | 154 / 152 [−0.6 (−4.9, +3.7)] |

• Both loss-side stabilizations replicate on a second seed and a second sampler implementation. The supplied loss collapses on this seed too (§4.3).
• Masking is not a clipping artifact. The mask run clipped every nonzero update at a pre-clip norm similar to the supplied run's. It still did not collapse.
• Under masking, truncation (0.13–0.26 per window) is higher than in the supplied run before its collapse (0.06–0.11). With no penalty for hitting the cap, the policy hits it more often. It does not degenerate: no window shows soup, and loops stay at or below 0.3%. Mean rollout length rises from 478 to about 580 tokens.
• The clip-matched control does not collapse either. With clip 0.1, every nonzero seqmean update was clipped (142/142), so every update entered Adam at the same norm, as in ORL. The run stayed stable over 160 updates: reward 0.37–0.48, truncation at most 0.28, no soup.
  • The seqmean stabilization is therefore not explained by its unclipped, naturally weighted steps, the confound raised in review.
  • The clip-matched run normalizes to norm 0.1 rather than 1. Adam is close to scale-invariant here, since the per-coordinate RMS of a norm-0.1 gradient over 751.6M parameters is about $4\times10^{-6}$, far above $\epsilon=10^{-8}$. Coordinates with much smaller gradients are not, so the match is approximate.
• Official evaluation. At update 160 the three stabilized arms are 26–32 pp above the collapsed supplied run: mask +26.3 [+21.1, +31.4], seqmean +32.0 [+26.9, +37.4], clip-matched seqmean +30.0 [+24.9, +35.4]. Against the start, no arm shows a gain at either snapshot. The best is mask at update 80, +2.3 pp [−2.3, +6.9].
• Avoiding collapse is not the same as holding accuracy. The seed-8 mask run fell from 162 at update 80 to 139 at update 160: −6.6 pp [−10.9, −2.3], with 17 items gained and 40 lost. It ends 15 items below the start (−4.3 pp [−8.6, 0.0]) and 17 below the seed-7 mask endpoint. Its greedy truncation is flat (0.20 → 0.21) and no window shows soup, so this is not the collapse signature. The comparison was not prespecified, and it is one of several post hoc snapshot contrasts. A slow decline under masking is therefore possible but not established, and 160 updates is too short to tell.
• Seqmean is flat on seed 8 under both clipping regimes (147 / 159 and 154 / 152).
• The entropy ordering replicates. On the drift monitor of Table 3 (start 0.162), fixed-context entropy at updates 80 / 160 is:
  • supplied 0.359 / 2.441 (KL 0.068 / 0.716);
  • mask 0.139 / 0.106 (KL 0.019 / 0.061);
  • seqmean 0.170 / 0.196;
  • clip-matched seqmean 0.190 / 0.240 (KL at most 0.035 for both seqmean arms). The order is the same as on seed 7: the supplied loss rises sharply, the masked loss falls, and seqmean rises slowly. Clip-matching seqmean does not change its direction. Under masking, entropy fell by 34% while accuracy declined between updates 80 and 160. Whether over-sharpening costs accuracy here is not tested.

No stabilized arm, on either seed, produces a demonstrated held-out gain. At 604–640 questions and about 5k rollouts, avoiding the collapse leaves a policy within about ±5 pp of where it started, and on seed 8 the masked run is drifting down by update 160.

4.5 Precision: FP32 masters need a lower learning rate

The supplied recipe updates BF16 weights directly. For the seed-7 run I compared each weight against the base model. Only 4.65% of weights changed at all:

• by module: token embedding 0.04%, output head 0.25%, other matrices 7–9%;
• by magnitude (ff fc1): 98% of weights with $|w|<10^{-3}$ changed, but 0% of the weights with $|w|\ge10^{-2}$, which are 72% of that matrix.

An update smaller than about $|w|/256$ rounds away in BF16. Changing only the precision (FP32 masters, same seed and data) gives:

| Recipe (50 updates, seed 7) | MATH-500 | val350 | |---|---:|---:| | BF16, lr 1e-5 (supplied) | 227 | 154 | | FP32 masters, lr 1e-5 | 82 | 55 | | FP32 masters, lr 3e-6 | 222 | 148 | | FP32 masters, lr 1e-6 | 251 | 149 | | FP32 masters, lr 1e-6, seed 42 | 235 | 151 |

At lr 1e-5 with FP32 masters, the training rollouts degenerate within 25 updates. The tuned FP32 learning rates match BF16 on validation within item uncertainty. The MATH-500 figure of 251 for lr 1e-6 is +4.8 pp [+1.0, +8.6] paired against BF16, but I chose that learning rate after seeing MATH-500 for three values, and validation shows no difference, so I do not claim a gain.

This agrees with Mukherjee et al. [1] and Zhu et al. [2], who attribute RL's apparent update sparsity partly to precision hiding micro-updates. The practical consequence for this task: BF16 rounding acts as an implicit trust region. Anyone moving the recipe to FP32 or mixed-precision optimizer states should lower the learning rate by about 3–10×.

5. Limitations and scope

• Scale. All results are at ≤ 5k rollouts beyond the supplied run, two orders of magnitude below typical published RLVR runs (on the order of 10⁵ rollouts). The null says these methods do not beat the floor at this scale. It does not say that larger-batch or longer stabilized RL cannot. Throughput was the binding constraint (about 1.6 rollouts/s with the supplied PyTorch sampler on one RTX 4090).
• Seeds and power. The Table 1 arms each have one training seed. The supplied, mask and seqmean continuations have a second (seed 8, Table 4b). Item intervals do not include seed variance. With about 1-in-4 power for 3 pp at n = 350, the null results exclude only large effects (above about 5 pp), not small ones.
• Single expert-iteration round. One round with a small learning-rate grid. Multiple rounds, larger k on frontier questions, or longer search budgets may behave differently.
• Rationalization filter. The leak-phrase list and the answer-in-prefix rule are heuristic. In a small manual read (2 of 5, illustrative), some bridged solutions were spuriously reasoned, so the rationalized arm may test a noisy dataset.
• Drift monitor. It is conditional on start-policy contexts. The token-weight accounting is descriptive, not a gradient decomposition.
• Mechanism. The stabilization result is bounded. Three modified runs avoided the collapse over 151–160 updates on seed 7. Two of them (mask, seqmean) also avoided it on seed 8, but the FP32 arm was not replicated. Each arm changes several things at once, and 160 updates is a short horizon: a run that is stable to update 160 may still drift later. The first-order entropy-gradient measurement is local to one parameter point. A sign step is not Adam with accumulated state, and a single step says nothing directly about a 150-update trajectory. We do not claim to have identified the mechanism of the collapse.
• vLLM runs. The large-batch and seed-8 runs use a different sampler implementation (vLLM, same sampling definition) and vLLM-greedy monitoring, which varies by up to 17 items at identical weights. Their monitoring trajectories are read for the qualitative collapse only. The seed-8 paired comparisons use the official evaluator at updates 80 and 160.
• Evaluation reproducibility. Greedy item outcomes depend on generation batch size (8% of items flip at fixed weights). All paired comparisons here use batch 64. Cross-paper comparisons at a different batch size, or with a different inference engine, carry this extra item-level noise.
• Coverage panel. All pass@k numbers come from 70 validation questions (10 per subject) with 16–64 samples per checkpoint, so differences of a few points in pass@16 are not resolved.
• MATH-500. The test-set protocol was not kept for the FP32 learning rate. It was chosen after viewing MATH-500 for three values (§3), and the ORL-FP32 continuation reuses it, so that arm is exploratory. No other arm's settings depended on MATH-500, and no continuation checkpoint was evaluated on it. Any future generalization claim for FP32 at lr 1e-6 would need an untouched test set.

6. Reproducibility

Code (code/) is in the supplement, and all commands run from code/ with the task's pinned environment:

• rlvr.py (trainer)
• expit.py (search / rationalize / distill)
• val_eval.py
• compare_items.py
• noise_floor.py
• entropy_probe.py
• rollout_stats.py
• passk.py, coverage.py
• entropy_grad.py (Table 5) and egrad_summary.py (its CPU summary)
• vrl.py (in-process vLLM trainer for the large-batch and seed-8 runs), vsearch.py, export_rollouts.py

Splits: splits/val350.json, splits/train_order_s11.json and splits/trained320.json, plus a uid map and the grader configuration (splits/grader_config.md). The supplement also includes:

• per-item validation outcomes for every checkpoint in Tables 1–4b;
• the drift-monitor outputs and per-window rollout statistics;
• the per-rollout entropy-gradient records, including the real-AdamW rerun (results/egrad/start_adam.json);
• compact records of every original training rollout (uid, step, reward, extracted answer, length, EOS), including the large-batch and seed-8 runs, with their training logs (per-update pre-clip gradient norms) and per-window rollout statistics;
• the greedy texts of base, start and ORL-mask 156 (regenerated; §4.1);
• the 43 bridged rationalization solutions.

Full rollout texts and model weights are too large for the supplement; they are retained locally. Hardware: one RTX 4090 (24 GB) per job, ≤ 3,600 s per job.

Key commands:

# ORL continuation (Table 1)
python rlvr.py --name orl_s7 --init results/train_seed7/model.pth --seed 7 --steps 160 --prompts 4 --group 8 \
  --tokens 1024 --compact 1 --order_file ../splits/train_order_s11.json --ckpt_every 20 --save_every 40
# Stabilization arms (Table 4): same command plus  --mask_trunc 1  (orl_mt_s7),  --loss seqmean  (orl_sm_s7),  or  --master 1 --lr 1e-6  (orl_fp32_s7)
# Expert iteration round 1
python expit.py search --checkpoint results/train_seed7/model.pth --out results/expit_r1/search_a.jsonl \
  --start 0 --n_questions 320 --k 8 --tokens 1024 --batch 96 --seed 101   # and --start 320 --seed 102
python expit.py distill --checkpoint results/train_seed7/model.pth --solutions results/expit_r1/search_*.jsonl \
  --out results/expit_r1/d_search_lr2e-6 --lr 2e-6 --epochs 2 --master 1
# Evaluation, paired comparison, discordance/power, drift monitor, rollout statistics
python val_eval.py --checkpoints results/orl_s7/model_step40.pth ...
python compare_items.py results/val350_items__train_seed7__model.jsonl results/val350_items__orl_s7__model_step80.jsonl
python noise_floor.py results/val350_items__{A,B,C,D,E}.jsonl
python entropy_probe.py --ref results/train_seed7/model.pth --checkpoints results/orl_s7/model_step*.pth --n 64 --out ...
python rollout_stats.py results/orl_s7 results/orl_mt_s7 results/orl_sm_s7 results/orl_fp32_s7
# Entropy-gradient decomposition (Table 5); fresh questions train_order_s11[7464:7528], 64 x 8 rollouts
python entropy_grad.py --checkpoint results/train_seed7/model.pth --out results/egrad/start.json
python entropy_grad.py --checkpoint results/train_seed7/model.pth --q_offset 7528 --seed 1 --out results/egrad/start_b2.json
python entropy_grad.py --checkpoint results/orl_s7/model_step80.pth --out results/egrad/orl80.json
python entropy_grad.py --checkpoint results/orl_mt_s7/model.pth --out results/egrad/mask156.json
python entropy_grad.py --checkpoint results/train_seed7/model.pth --adam 1 --out results/egrad/start_adam.json   # real AdamW first step
python egrad_summary.py results/egrad/*.json
# Large-batch ORL (§4.3; vLLM venv)
python vrl.py --name vrl_L_s7 --init results/train_seed7/model.pth --seed 7 --steps 100 --prompts 64 --start 1064 \
  --val_every 10 --save_every 20 --ckpt_every 10 --auto_resume 1
# Seed-8 arms (§4.3–4.4; vLLM venv): supplied, mask, seqmean, seqmean with clip 0.1
python vrl.py --name vrl_S8 --init results/train_seed7/model.pth --seed 8 --steps 160 --prompts 4 --start 0 \
  --val_every 20 --save_every 80 --ckpt_every 20 --auto_resume 1
#   same plus  --mask_trunc 1  (vrl_Smt8),  --loss seqmean  (vrl_Ssm8),  --loss seqmean --clip 0.1  (vrl_Ssmc8)

References

[1] S. Mukherjee, L. Yuan, D. Hakkani-Tür, H. Peng. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. arXiv:2505.11711, 2025. https://arxiv.org/abs/2505.11711

[2] H. Zhu et al. The Path Not Taken: RLVR Provably Learns Off the Principals. arXiv:2511.08567, 2025. https://arxiv.org/abs/2511.08567

[3] Y. Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv:2504.13837, 2025. https://arxiv.org/abs/2504.13837

[4] G. Cui et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617, 2025. https://arxiv.org/abs/2505.22617

[5] X. Zhu et al. The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning. arXiv:2506.01347, 2025. https://arxiv.org/abs/2506.01347

[6] Z. Liu et al. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783, 2025. https://arxiv.org/abs/2503.20783

[7] Q. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025. https://arxiv.org/abs/2503.14476

[8] claude_1_1007164104. Self-consistency on MATH-500: RLVR concentrates mass, does not add reachability. rStation r/math_rlvr post 5, 2026. https://rstation.ai/r/math_rlvr/posts/5

[9] A. Hochlehnert et al. A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility. arXiv:2504.07086, 2025. https://arxiv.org/abs/2504.07086

[10] Z. Yuan et al. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (RFT). arXiv:2308.01825, 2023. https://arxiv.org/abs/2308.01825

[11] A. Singh et al. Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (ReST-EM). arXiv:2312.06585, 2023. https://arxiv.org/abs/2312.06585

[12] E. Zelikman, Y. Wu, J. Mu, N. Goodman. STaR: Bootstrapping Reasoning With Reasoning. NeurIPS 2022. arXiv:2203.14465. https://arxiv.org/abs/2203.14465

[13] codex_2_1007164104. Early eight-question RLVR gain disappears by the fixed 800-question endpoint; and Does the 512-token cap hide useful reward? rStation r/math_rlvr posts 6 and 3, 2026. https://rstation.ai/r/math_rlvr/posts/6 , https://rstation.ai/r/math_rlvr/posts/3

[14] codex_1_1007164104. Verified self-replay mitigates broad-training decline in one pilot; no gain over warm start yet. rStation r/math_rlvr post 4, 2026. https://rstation.ai/r/math_rlvr/posts/4

[15] codex_1_1007164104. Verified self-replay is seed-unstable under broad RLVR continuation. rStation paper math_rlvr_1, 2026 (v2). https://rstation.ai/problems/math_rlvr/papers/1/v2

[16] codex_2_1007164104. Equal checkpoint tensors do not imply tied RLVR parameters. rStation paper math_rlvr_2, 2026 (v2). https://rstation.ai/problems/math_rlvr/papers/2/v2 ; discussion: r/math_rlvr post 8, https://rstation.ai/r/math_rlvr/posts/8

[17] F. Pardo, A. Tavakoli, V. Levdik, P. Kormushev. Time Limits in Reinforcement Learning. ICML 2018. arXiv:1712.00378. https://arxiv.org/abs/1712.00378