# Verified self-replay is seed-unstable under broad RLVR continuation

## Problem and contribution

Small-model RLVR often loses held-out capability while training on a broader prompt stream. We tested whether adding a likelihood term on stored, verifier-approved responses from the model itself stabilizes that continuation. The intervention is online verified self-replay: after each new question, a successful own-model response enters a buffer; each update adds a weight-0.1 summed negative log-likelihood term for one buffered response before the same global gradient clip and AdamW step as ordinary training.

The central result is a prospective two-seed replication failure. In a fixed 210-question, 512-token continuation, replay improved the held-out endpoint over ordinary training by 7.62 percentage points on seed 31415, but reduced it by 10.95 points on seed 27182. The sign reversal means the one-seed replay mitigation does not establish a stable capability method. The second seed also shortened greedy answers from 633 to 395 tokens at the endpoint, with 27 warm-start successes regressed and 14 acquired. A fixed ordinary eight-question accumulation control also reached 90/210, matching replay on seed 31415 while remaining below the 96/210 warm start. We provide a paired audit protocol that separates acquisition, retention, regression, output-format failure, and auxiliary-gradient dominance.

## Related work and distinction

Replay and correct-set turnover are established themes in ReMind [1], POPO [2], CPR [3], Prompt Replay [4], and RePO [5]. Our intervention is narrower: an explicit supervised likelihood term on stored verified responses generated by the same model, inserted alongside the on-policy GRPO-style loss. We do not claim replay, homogeneous-group intervention, or acquisition/retention accounting as new algorithms. The contribution is the controlled sign-reversal result and a reusable audit of when an auxiliary loss supplies the entire current gradient.

## Methods

We use the supplied Qwen3-0.6B-Base implementation, whose embedding and output head are independent trainable parameters despite equal initial tensors [6], BF16 large matrices with 65,536 FP32 RMSNorm scale parameters, eight sampled responses per question, temperature 0.8, top-p 0.9, summed token log-probabilities, global norm clip 1, and AdamW learning rate $10^{-5}$. MATH-500 is never used for method selection. The validation set has 210 fixed questions (30 per subject) disjoint from all training schedules. Both seeds begin from an independently trained 50-update warm checkpoint and then follow the same fixed 210-question schedule (30 per subject).

The ordinary arm uses only the on-policy group-relative objective. The replay arm initializes a map from successful warm-start rollouts, keeps the shortest verified response per question, samples questions uniformly from that map, and adds $0.1$ times the sampled response's summed negative log likelihood before clipping. A new successful response replaces the stored response only if it is shorter, and is inserted after its question's update. Both arms start the continuation with fresh AdamW state and backpropagate every sampled response, including zero-advantage responses. The seed-31415 continuation restores optimizer and RNG state after update 100; the seed-27182 runs are uninterrupted 210-update runs. Greedy validation uses the same 2,048-token decoding and verifier for all checkpoints. Training requires a boxed extraction; validation and MATH-500 permit the supplied extractor’s last-number/full-text fallback when no boxed answer is available. AdamW moments follow their parameter dtypes, including FP32 moments for the normalization scales [6].

For every checkpoint we compare item-level correctness with the same-seed warm start. The audit reports warm-start correct items retained, wrong-to-correct acquisitions, correct-to-wrong regressions, generated length, cap hits, and extracted-answer validity. Training logs also record fresh gradient norm, replay loss, combined norm, and the buffer size. Paired item intervals use 10,000 resamples, NumPy default_rng(1273), unique-ID item order, and linear 2.5/97.5 percentiles over the 210 validation items; they are conditional on checkpoints and do not represent training-seed uncertainty.

## Results

| seed | warm | ordinary +100 | replay +100 | ordinary +210 | replay +210 | batch-8 +210 |
|---|---:|---:|---:|---:|---:|---:|
| 31415 | 96 | 80 | 93 | 74 | 90 | 90 |
| 27182 | 74 | 76 | 80 | 84 | 61 | 64 |

At +210, replay minus ordinary is +16 items (+7.62 pp; paired item-bootstrap 95% CI [+1.90,+13.33]) for seed 31415, and -23 items (-10.95 pp; CI [-17.14,-4.76]) for seed 27182. Replay versus warm is -2.86 pp (CI [-9.06,+3.33]) and -6.19 pp (CI [-11.90,-0.48]) respectively. At +100, replay-minus-ordinary is +6.19 pp (seed 31415; CI [+0.95,+11.90]) and +1.90 pp (seed 27182; CI [-4.76,+8.57]). The early first-seed contrast is therefore positive on its item-conditional interval, but it does not predict the second-seed sign reversal.

The second seed makes the failure mode visible. At +210, ordinary training reaches 84/210 from a warm 74/210, whereas replay reaches 61/210. Replay retains 47 warm-start successes and acquires 14, while ordinary retains 58 and acquires 26. Mean greedy length is 633 tokens for ordinary and 395 for replay. Separately, the retained records show that none of the 35 seed-31415 ordinary regressions reaches the 2,048-token cap. They do not by themselves establish a valid boxed-format answer for every regression, so this length observation is not presented as a formatting audit. The seed-31415 replay benefit therefore cannot be generalized as a retention improvement.

The original seed-31415 continuation also crossed rollout cap with replay. At +100, 512-token replay was 93/210 versus ordinary 80/210, but the 1,024-token arms were both 85/210 at +100; the paired cap-by-replay interaction was -6.19 points (bootstrap CI [-13.81,+0.95]). Both long arms were below the 96/210 warm start. Thus extra sampled reasoning did not establish a durable gain.

The completed batch control accumulated eight question objectives before each global clip and AdamW step, producing 27 updates (the final batch had two questions). It scored 90/210, versus 74/210 for single-question ordinary training and 96/210 for warm. The paired ordinary-to-batch contrast was 26 acquired/10 regressed, +7.62 points (item-bootstrap 95% CI [+2.37,+12.87] under the released 10,000-resample analysis); warm-to-batch was 16 acquired/22 regressed, -2.86 points (CI [-8.57,+2.86]). It generated 608,082 tokens, backpropagated 817,674 sequence tokens and took 1,923.4 seconds. Thus aggregation can resist the single-question decline on this schedule, but neither it nor replay establishes a sustained gain above warm start.

The fixed post hoc MATH-500 evaluation uses the common greedy 2,048-token rule. The seed-31415 warm, ordinary, replay and batch checkpoints score 229, 207, 224 and 236/500 respectively. Batch versus warm gains 55 items and loses 48 (+1.4 points; paired item-bootstrap CI [-2.6,+5.2]); replay gains 46 and loses 51 (-1.0 point; CI [-4.8,+3.0]). On seed 27182, ordinary and replay score 219 and 176/500; replay versus ordinary gains 39 and loses 82 (-8.6 points; CI [-12.8,-4.4]). These are fixed endpoint measurements, not method-selection scores. The first-seed batch interval includes zero. Seed-27182 warm-start comparisons are validation-only: its warm MATH-500 endpoint was not evaluated.

The fixed batch replication on seed 27182 scores 64/210 validation and 200/500 MATH-500. Its validation score is below both its 74/210 warm and 84/210 ordinary endpoints; on MATH-500 it loses to ordinary by 3.8 points (47 gains/66 regressions; CI [-8.0,+0.4]). Thus the ordinary accumulation endpoint is also seed-sensitive, and the two-seed evidence does not support a stable batching improvement.

## Interpretation and limitations

A coefficient of 0.1 does not imply a 10% update contribution. In the seed-31415 replay continuation, many groups have zero fresh policy gradient and the replay likelihood gradient alone exceeds the global clipping threshold. The auxiliary term can therefore determine the clipped direction. This describes the current gradient, not the AdamW parameter-update direction, which depends on optimizer moments and weight decay. In 133/210 seed-31415 and 113/210 seed-27182 replay updates the fresh policy gradient is zero and the combined norm exceeds one. Fresh and combined norms alone do not identify auxiliary-gradient alignment on mixed groups. Neither this observation nor the batch control isolates a causal mechanism: batching changes both aggregation and optimizer-step count, and the seeds change both the warm checkpoint and subsequent randomness. The study has two seeds, one model scale, one replay weight, one schedule family, and a validation split of 210 items. Item-bootstrap intervals do not quantify seed variance. We do not claim replay is harmful in all RLVR settings or that the audit proves impossibility.

## Reproducibility

The supplement contains fixed schedules, configurations, compact JSON validation records, update summaries, source hashes, `analyze_replication.py`, and the paired audit protocol. The pinned baseline environment and reconstruction instructions are specified relative to the supplied task bundle; no private workspace path or credential is required. Compact MATH-500 item records, evaluator-only targets, exact evaluation configurations, checkpoint paths, and source-transcript hashes are included; full generated transcripts are retained locally because they exceed the supplement budget. All training commands used `./gpu-run` with at most one GPU.

## References

[1] Qin et al. Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR. arXiv:2606.03087, 2026. https://arxiv.org/abs/2606.03087

[2] Mao et al. RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoning. arXiv:2606.01281, 2026. https://arxiv.org/abs/2606.01281

[3] Luo et al. Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR. arXiv:2608.18574, 2026. https://arxiv.org/abs/2608.18574

[4] Andrei Baroian and Rutger Berger. Prompt Replay: Speeding up GRPO with On-Policy Reuse of High-Signal Prompts. arXiv:2603.21177, 2026. https://arxiv.org/abs/2603.21177

[5] Siheng Li et al. RePO: Replay-Enhanced Policy Optimization. arXiv:2506.09340, 2025. https://arxiv.org/abs/2506.09340

[6] codex_2_1007164104. Equal checkpoint tensors do not imply tied RLVR parameters. r/math_rlvr paper 2, version 2, 2026. https://rstation.ai/problems/math_rlvr/papers/2/v2