Verified self-replay is seed-unstable under broad RLVR continuation

Problem and contribution

Small-model RLVR often loses held-out capability while training on a broader prompt stream. We tested whether adding a likelihood term on stored, verifier-approved responses from the model itself stabilizes that continuation. The intervention is online verified self-replay: after each new question, a successful own-model response enters a buffer; each update adds a weight-0.1 summed negative log-likelihood term for one buffered response before the same global gradient clip and AdamW step as ordinary training.

The central result is a prospective two-seed replication failure. In a fixed 210-question, 512-token continuation, replay improved the held-out endpoint over ordinary training by 7.62 percentage points on seed 31415, but reduced it by 10.95 points on seed 27182. The sign reversal means the one-seed replay mitigation is not a stable capability method. The second seed also shortened greedy answers from 633 to 395 tokens at the endpoint, with 27 warm-start successes regressed and 14 acquired. A fixed ordinary eight-question accumulation control also reached 90/210, matching replay on seed 31415 while remaining below the 96/210 warm start. We provide a paired audit protocol that separates acquisition, retention, regression, output-format failure, and auxiliary-gradient dominance.

Related work and distinction

Replay and correct-set turnover are established themes in ReMind [1], POPO [2], CPR [3], Prompt Replay [4], and RePO [5]. Our intervention is narrower: an explicit supervised likelihood term on stored verified responses generated by the same model, inserted alongside the on-policy GRPO-style loss. We do not claim replay, homogeneous-group intervention, or acquisition/retention accounting as new algorithms. The contribution is the controlled sign-reversal result and a reusable audit of when an auxiliary loss becomes the effective update.

Methods

We use the supplied Qwen3-0.6B-Base implementation, BF16 parameters, eight sampled responses per question, temperature 0.8, top-p 0.9, summed token log-probabilities, global norm clip 1, and AdamW learning rate $10^{-5}$. MATH-500 is never used for method selection. The validation set has 210 fixed questions (30 per subject) disjoint from all training schedules. Both seeds begin from an independently trained 50-update warm checkpoint and then follow the same fixed 210-question schedule (30 per subject).

The ordinary arm uses only the on-policy group-relative objective. The replay arm initializes a map from successful warm-start rollouts, keeps the shortest verified response per question, samples questions uniformly from that map, and adds $0.1$ times the sampled response's summed negative log likelihood before clipping. A new successful response replaces the stored response only if it is shorter, and is inserted after its question's update. Both arms backpropagate every sampled response, including zero-advantage responses. The seed-31415 continuation restores optimizer and RNG state after update 100; the seed-27182 runs are uninterrupted 210-update runs. Greedy validation uses the same 2,048-token decoding and verifier for all checkpoints.

For every checkpoint we compare item-level correctness with the same-seed warm start. The audit reports warm-start correct items retained, wrong-to-correct acquisitions, correct-to-wrong regressions, generated length, cap hits, and extracted-answer validity. Training logs also record fresh gradient norm, replay loss, combined norm, and the buffer size. Paired item intervals are bootstrap intervals over the 210 validation items; they are conditional on checkpoints and do not represent training-seed uncertainty.

Results

| seed | warm | ordinary +100 | replay +100 | ordinary +210 | replay +210 | batch-8 +210 | |---|---:|---:|---:|---:|---:|---:| | 31415 | 96 | 80 | 93 | 74 | 90 | 90 | | 27182 | 74 | 76 | 80 | 84 | 61 | 64 |

At +210, replay minus ordinary is +16 items (+7.62 pp; paired item-bootstrap 95% CI [+1.90,+13.33]) for seed 31415, and -23 items (-10.95 pp; CI [-17.14,-4.76]) for seed 27182. Replay versus warm is -2.86 pp (CI [-9.06,+3.33]) and -6.19 pp (CI [-11.90,-0.48]) respectively. At +100, replay-minus-ordinary is +6.19 pp (seed 31415; CI [+0.95,+11.90]) and +1.90 pp (seed 27182; CI [-4.76,+8.57]). The early first-seed contrast is therefore positive on its item-conditional interval, but it does not predict the second-seed sign reversal.

The second seed makes the failure mode visible. At +210, ordinary training reaches 84/210 from a warm 74/210, whereas replay reaches 61/210. Replay retains 47 warm-start successes and acquires 14, while ordinary retains 58 and acquires 26. Mean greedy length is 633 tokens for ordinary and 395 for replay. The retained evaluation records show that none of these 35 seed-31415 ordinary regressions reaches the 2,048-token cap. They do not by themselves establish a valid boxed-format answer for every regression, so this length observation is not presented as a formatting audit. The seed-31415 replay benefit therefore cannot be generalized as a retention improvement.

The original seed-31415 continuation also crossed rollout cap with replay. At +100, 512-token replay was 93/210 versus ordinary 80/210, but the 1,024-token arms were both 85/210 at +100; the paired cap-by-replay interaction was -6.19 points (bootstrap CI [-13.81,+0.95]). Both long arms were below the 96/210 warm start. Thus extra sampled reasoning did not establish a durable gain.

The completed batch control accumulated eight question objectives before each global clip and AdamW step, producing 27 updates (the final batch had two questions). It scored 90/210, versus 74/210 for single-question ordinary training and 96/210 for warm. The paired ordinary-to-batch contrast was 26 acquired/10 regressed, +7.62 points (item-bootstrap 95% CI approximately [+2.38,+13.33]); warm-to-batch was 16 acquired/22 regressed, -2.86 points (CI [-8.57,+2.86]). It generated 608,082 tokens, backpropagated 817,674 sequence tokens and took 1,923.4 seconds. Thus aggregation can resist the single-question decline on this schedule, but neither it nor replay establishes a sustained gain above warm start.

The fixed post hoc MATH-500 evaluation uses the common greedy 2,048-token rule. The seed-31415 warm, ordinary, replay and batch checkpoints score 229, 207, 224 and 236/500 respectively. Batch versus warm gains 55 items and loses 48 (+1.4 points; paired item-bootstrap CI [-2.6,+5.2]); replay gains 46 and loses 51 (-1.0 point; CI [-4.8,+3.0]). On seed 27182, ordinary and replay score 219 and 176/500; replay versus ordinary gains 39 and loses 82 (-8.6 points; CI [-12.8,-4.4]). These are fixed endpoint measurements, not method-selection scores. The batch result is suggestive but its interval includes zero.

The predeclared batch replication on seed 27182 scores 64/210 validation and 200/500 MATH-500. It is below both its 74/210 warm and 84/210 ordinary endpoints; on MATH-500 it loses to ordinary by 3.8 points (47 gains/66 regressions; CI [-8.0,+0.4]). Thus the ordinary accumulation endpoint is also seed-sensitive, and the two-seed evidence does not support a stable batching improvement.

Interpretation and limitations

A coefficient of 0.1 does not imply a 10% update contribution. In the seed-31415 replay continuation, many groups have zero fresh policy gradient and the replay likelihood gradient alone exceeds the global clipping threshold. The auxiliary term can therefore determine the clipped direction. This provides a mechanism for seed-sensitive behavior but does not prove causality for the endpoint reversal. The study has two seeds, one model scale, one replay weight, one schedule family, and a validation split of 210 items. Item-bootstrap intervals do not quantify seed variance. We do not claim replay is harmful in all RLVR settings or that the audit proves impossibility.

Reproducibility

The supplement contains fixed schedules, configurations, compact JSONL validation records, update summaries, source hashes, analyze_replication.py, and the paired audit protocol. The pinned baseline environment and reconstruction instructions are specified relative to the supplied task bundle; no private workspace path or credential is required. A compact MATH-500 endpoint table and hashes of the evaluation records are included; full generated transcripts are retained locally because they exceed the supplement budget. All training commands used ./gpu-run with at most one GPU.

References

[1] Qin et al. ReMind: Correct-Set Turnover for Continual RLVR. arXiv:2606.03087, 2026. https://arxiv.org/abs/2606.03087

[2] Mao et al. POPO: Prompt-level replay and importance weighting for RLVR. arXiv:2606.01281, 2026. https://arxiv.org/abs/2606.01281

[3] Luo et al. CPR: Continual Prompt Replay for Verifiable Reward Learning. arXiv:2608.18574, 2026. https://arxiv.org/abs/2608.18574

[4] Prompt Replay for Reinforcement Learning with Verifiable Rewards. arXiv:2603.21177, 2026. https://arxiv.org/abs/2603.21177

[5] RePO: Replay-enhanced Policy Optimization for RLVR. arXiv:2506.09340, 2025. https://arxiv.org/abs/2506.09340