# Verified self-replay is seed-unstable under broad RLVR continuation

## Problem and contribution

Small-model RLVR often loses held-out capability while training on a broader prompt stream. We tested whether adding a likelihood term on stored, verifier-approved responses from the model itself stabilizes that continuation. The intervention is online verified self-replay: after each new question, a successful own-model response enters a buffer; each update adds a weight-0.1 summed negative log-likelihood term for one buffered response before the same global gradient clip and AdamW step as ordinary training.

The central result is a prospective two-seed replication failure. In a fixed 210-question, 512-token continuation, replay improved the held-out endpoint over ordinary training by 7.62 percentage points on seed 31415, but reduced it by 10.95 points on seed 27182. The sign reversal means the one-seed replay mitigation is not a stable capability method. The second seed also shortened greedy answers from 633 to 395 tokens at the endpoint, with 27 warm-start successes regressed and 14 acquired. A fixed ordinary eight-question accumulation control also reached 90/210, matching replay on seed 31415 while remaining below the 96/210 warm start. We provide a paired audit protocol that separates acquisition, retention, regression, output-format failure, and auxiliary-gradient dominance.

## Related work and distinction

Replay and correct-set turnover are established themes in ReMind [1], POPO [2], CPR [3], Prompt Replay [4], and RePO [5]. Our intervention is narrower: an explicit supervised likelihood term on stored verified responses generated by the same model, inserted alongside the on-policy GRPO-style loss. We do not claim replay, homogeneous-group intervention, or acquisition/retention accounting as new algorithms. The contribution is the controlled sign-reversal result and a reusable audit of when an auxiliary loss becomes the effective update.

## Methods

We use the supplied Qwen3-0.6B-Base implementation, BF16 parameters, eight sampled responses per question, temperature 0.8, top-p 0.9, summed token log-probabilities, global norm clip 1, and AdamW learning rate $10^{-5}$. MATH-500 is never used for method selection. The validation set has 210 fixed questions (30 per subject) disjoint from all training schedules. Both seeds begin from an independently trained 50-update warm checkpoint and then follow the same fixed 210-question schedule (30 per subject), with optimizer and RNG state restored after update 100.

The ordinary arm uses only the on-policy group-relative objective. The replay arm samples one previously verified own-model response uniformly from the buffer and adds $0.1$ times its summed negative log likelihood before clipping. New successful responses are inserted after each update. Both arms backpropagate every sampled response, including zero-advantage responses, and use a fresh optimizer restart for the continuation. Greedy validation uses the same 2,048-token decoding and verifier for all checkpoints.

For every checkpoint we compare item-level correctness with the same-seed warm start. The audit reports warm-start correct items retained, wrong-to-correct acquisitions, correct-to-wrong regressions, generated length, cap hits, and boxed-answer validity. Training logs also record fresh gradient norm, replay loss, combined norm, and the buffer size. Paired item intervals are bootstrap intervals over the 210 validation items; they are conditional on checkpoints and do not represent training-seed uncertainty.

## Results

| seed | warm | ordinary +100 | replay +100 | ordinary +210 | replay +210 |
|---|---:|---:|---:|---:|---:|
| 31415 | 96 | 80 | 93 | 74 | 90 |
| 27182 | 74 | 76 | 80 | 84 | 61 |

At +210, replay minus ordinary is +16 items (+7.62 pp; paired item-bootstrap 95% CI [+1.90,+13.33]) for seed 31415, and -23 items (-10.95 pp; CI [-17.14,-4.76]) for seed 27182. Replay versus warm is -2.86 pp (CI [-9.06,+3.33]) and -6.19 pp (CI [-11.90,-0.48]) respectively. The +100 replay-minus-ordinary differences are +6.19 pp (seed 31415) and +1.90 pp (seed 27182), both compatible with zero.

The second seed makes the failure mode visible. At +210, ordinary training reaches 84/210 from a warm 74/210, whereas replay reaches 61/210. Replay retains 47 warm-start successes and acquires 14, while ordinary retains 58 and acquires 26. Mean greedy length is 633 tokens for ordinary and 395 for replay. This is not a formatting-only explanation: the paired audit of the seed-31415 ordinary arm found that all 35 held-out regressions had a nonempty boxed extraction and none hit the 2,048-token evaluation cap. The seed-31415 replay benefit therefore cannot be generalized as a retention improvement.

The original seed-31415 continuation also crossed rollout cap with replay. At +100, 512-token replay was 93/210 versus ordinary 80/210, but the 1,024-token arms were both 85/210 at +100; the paired cap-by-replay interaction was -6.19 points (bootstrap CI [-13.81,+0.95]). Both long arms were below the 96/210 warm start. Thus extra sampled reasoning did not establish a durable gain.

The completed batch control accumulated eight question objectives before each global clip and AdamW step, producing 27 updates (the final batch had two questions). It scored 90/210, versus 74/210 for single-question ordinary training and 96/210 for warm. The paired ordinary-to-batch contrast was 26 acquired/10 regressed, +7.62 points (item-bootstrap 95% CI [+2.37,+12.87]); warm-to-batch was 16 acquired/22 regressed, -2.86 points (CI [-8.57,+2.86]). It generated 608,082 tokens, backpropagated 817,674 sequence tokens and took 1,923.4 seconds. Thus aggregation can resist the single-question decline on this schedule, but neither it nor replay establishes a sustained gain above warm start.

## Interpretation and limitations

A coefficient of 0.1 does not imply a 10% update contribution. In the seed-31415 replay continuation, many groups have zero fresh policy gradient and the replay likelihood gradient alone exceeds the global clipping threshold. The auxiliary term can therefore determine the clipped direction. This provides a mechanism for seed-sensitive behavior but does not prove causality for the endpoint reversal. The study has two seeds, one model scale, one replay weight, one schedule family, and a validation split of 210 items. Item-bootstrap intervals do not quantify seed variance. We do not claim replay is harmful in all RLVR settings or that the audit proves impossibility.

## Reproducibility

The local workspace contains the exact schedules, configurations, JSONL rollouts and validation records under `code_local/results/`, the paired analysis in `notes/replication_analysis.json`, and the audit protocol in `notes/paired_rlvr_audit_protocol.md`. The pinned environment is `/mnt/stephen/research_data/math_rlvr/.venv/bin/python`; all training commands were run through `./gpu-run` with at most one GPU. The public evidence bundle will include source hashes, update summaries and paired validation records without credentials.

## References

[1] Qin et al. ReMind: Correct-Set Turnover for Continual RLVR. arXiv:2606.03087, 2026. https://arxiv.org/abs/2606.03087

[2] Mao et al. POPO: Prompt-level replay and importance weighting for RLVR. arXiv:2606.01281, 2026. https://arxiv.org/abs/2606.01281

[3] Luo et al. CPR: Continual Prompt Replay for Verifiable Reward Learning. arXiv:2608.18574, 2026. https://arxiv.org/abs/2608.18574

[4] Prompt Replay for Reinforcement Learning with Verifiable Rewards. arXiv:2603.21177, 2026. https://arxiv.org/abs/2603.21177

[5] RePO: Replay-enhanced Policy Optimization for RLVR. arXiv:2506.09340, 2025. https://arxiv.org/abs/2506.09340