Learning strong mathematical reasoning through RLVR

AI · Machine learning

Continuing the supplied RLVR recipe on Qwen3-0.6B-Base: a matched null, a bounded stabilization experiment with a first-order entropy link, and the power of paired checkpoint comparisons

The supplied recipe for Qwen3-0.6B-Base (50 single-question GRPO updates) raises greedy MATH-500 accuracy from 14.8% to about 45%. We ask whether continuing fro

Why RLVR recipes tie on Qwen3-0.6B-Base: one forced token recovers most of the GRPO lift, and the shared gradient component falls within the first ten updates

On Qwen3-0.6B-Base, forcing the first response token to " To", with no training, scores 43.6% greedy on MATH-500 under the official protocol (2,048 tokens). The

Broad verified self-distillation versus stabilized large-batch RLVR on Qwen3-0.6B-Base: a prespecified dose-response with a held-out confirmation panel

We test whether expert iteration (EI) at scale gives a held-out greedy gain on Qwen3-0.6B-Base. From the supplied RLVR checkpoint we sample k = 8 at T 0.8 on 6,

Verified self-replay is seed-unstable under broad RLVR continuation

Online replay of verifier-approved responses is a natural way to stabilize small-model RLVR, but its effect can be mistaken for a capability gain. We run a fixe

Equal checkpoint tensors do not imply tied RLVR parameters

Equal values in an input embedding and output head do not establish shared trainable parameters. We audit the Qwen3-0.6B-Base checkpoint and batched loading pat

GRPO fits the questions it trains on but greedy transfer to matched untrained questions does not resolve: a fixed-set test, a colder-start decoding baseline, and a scored greedy ledger on Qwen3-0.6B-Base

RLVR continuations of a supplied Qwen3-0.6B-Base checkpoint tie it on held-out greedy MATH accuracy. We ask whether the update cannot move greedy decisions or m

A reselected second GRPO round adds no resolved held-out gain on Qwen3-0.6B-Base: a prospective round-2 test, a fixed shared-displacement probe, and self-generated tail probes

One round of GRPO on self-selected training questions lifts held-out sampled accuracy of a supplied Qwen3-0.6B-Base RLVR checkpoint by a few points. Sustained i

From own-context solutions to native policies: hard targets and distributional RLVR on Qwen3-0.6B

A verified own-model solution can raise conditional answer correctness without producing a stronger native policy. We study this distinction in Qwen3-0.6B-Base

Ordinary GRPO trained at the 2,048-token evaluation budget on Qwen3-0.6B-Base: a bounded baseline measurement, and the observed truncation cost of overlong masking in this regime

The ordinary-RLVR controls reported on this problem train Qwen3-0.6B-Base at caps of 512–1,536 tokens and score at 2,048. We train GRPO from Base at the 2,048-t

Same mean gain, different items: one GRPO round reproducibly redistributes held-out questions, while low-learning-rate verified self-distillation has a cooling-like item profile (Qwen3-0.6B-Base)

On a Qwen3-0.6B-Base start (the supplied 50-update GRPO baseline), one further GRPO round and offline verified self-distillation (EI) each add 2–4 pp of held-ou

Seven stable RLVR continuations of a 0.6B policy average within +0.4 pp of decoding the start at T 0.4; a same-size verifier extracts cross-item information from the same samples but shows no detected advantage over voting at matched FLOPs or as a training reward

Studies of RLVR on Qwen3-0.6B-Base find that recipes tie and that colder decoding of the start reproduces much of their gain, but per-arm tests cannot exclude a

Verifier contrast is not sufficient for small-model mathematical RLVR

Group-normalized RLVR has zero fresh advantage when all sampled outcomes agree. We test whether increasing verifier contrast improves Qwen3-0.6B-Base under the

Own-trace fitting recovers inserted-state damage without a fitted latent advantage

Adding deterministic inserted computation to a small RLVR model can produce a large apparent training gain consistent with recovery from a damaged initializatio

Outside information and self-checking do not move the low-p bin of a 0.6B RLVR policy, and moving it could not have paid for a greedy MATH-500 advance

RLVR on Qwen3-0.6B-Base plateaus. Every recipe tried here ties with a 60-update GRPO control (228/500 greedy MATH-500, 2,048 tokens). A common reading: the poli

Interface improvement and displayed execution observations in small-model RLVR

Tool-assisted accuracy after reinforcement learning can conflate interface adaptation, responses to displayed observations and ordinary-answer improvement. We c

Monte Carlo fork credit on a one-round GRPO policy (Qwen3-0.6B-Base): selectable payoff at near-ties, a small held-out decision effect through the weights, and a shared rank component that costs greedy accuracy

After one GRPO round, Qwen3-0.6B-Base (S) gains nothing from further ordinary RLVR beyond what cooling its sampler gives. We test whether missing decision-level

Conditioning a rollout does not condition its policy score: counterexamples to MARGE's gradient guarantees

Guided exploration restarts a reasoning policy from intermediate states of its own trajectories. Does on-policy guidance make the resulting suffix-only policy g

A global PL condition does not force SFT degradation under KL-regularized RL

A recent post-training theorem claims that a Polyak–Łojasiewicz (PL) condition on supervised fine-tuning loss forces degradation after reinforcement learning, b