Learning strong mathematical reasoning through RLVR

AI · Machine learning

Learning strong mathematical reasoning through RLVR

Research goal

Develop methods that train a small language model through reinforcement learning with verifiable rewards (RLVR) to achieve substantially stronger mathematical reasoning, measured primarily on MATH-500, without supervision or assistance from larger models. Start from Qwen3-0.6B-Base.

The long-term ambition is an AlphaGo moment for LLMs: a learning process in which the model's own reasoning and verifiable feedback produce large, sustained capability gains. Seek a substantial advance over strong RLVR baselines and an explanation of why it works.

Starting point

The supplied GRPO-style implementation is adapted from Raschka's RLVR baseline.

| MATH-500 evaluation | Accuracy | | --- | ---: | | Qwen3-0.6B-Base before training | 14.8% (74/500) | | After 50 RLVR updates | 44.4% (222/500) |

These are recorded results from one seed. Training used eight responses per question and the first 50 questions, all algebra, from a separate 12,000-question MATH training pool. Evaluation used greedy decoding with a 2,048-token generation limit. This short run is a starting point, not a strong ceiling.

The task supplement ZIP contains baseline code, datasets, tokenizer, recorded outputs, provenance and reproduction instructions. Begin with its README and code/README.md. Pretrained weights are downloaded through the supplied helper; trained weights are regenerated by the training command.

Directions to explore

• System 2–System 1 feedback: turn verified deliberation into a better fast policy, then use that policy to improve subsequent deliberation. See Thinker, Chung's thesis, Section 5.4, and STaR. Aim for continuing gains across learning rounds; shorter answers alone do not establish a successful feedback loop.
• Looped transformers and latent reasoning: investigate recurrent computation at comparable parameter scale, inspired by recurrent-depth models.
• Better RL objectives and exploration: investigate credit assignment, normalization, curricula, replay or learning from failed attempts. Relevant references include DeepSeek-R1 and R1-Zero and Dr. GRPO.

These are starting ideas. Pursue whichever mechanism produces a convincing advance, and distinguish it from existing work.

Experimental constraints

Use Qwen3-0.6B-Base as the common starting checkpoint. Architectural variants should remain at a comparable parameter scale; report additional trainable components. Training questions, ground-truth answers, programmatic verification, the model's own outputs, earlier checkpoints and critics or verifiers trained within this small-model setting are allowed.

Larger models must not supply demonstrations, distilled reasoning, logits, judgments, reward or value estimates, answer selection, or other assistance during training or evaluation. This excludes teacher-generated datasets and externally distilled reasoning checkpoints. Self-distillation within the experimental model family is allowed. Research agents may design and implement methods, but their answers and judgments must not become supervision for the experimental model.

Keep MATH-500 held out. Use separate training and validation problems for curricula, replay, method selection and hyperparameter tuning. Test answers may be used by the evaluator for grading, not by the reasoning system to select or repair its answers.

Evidence of progress

Compare against both the supplied baseline and a well-tuned ordinary RLVR baseline under comparable resources, including controls that train longer or use more training questions.

Keep a common single-answer evaluation for the main comparison, with the same decoding rule and generation budget across methods. Report MATH-500 accuracy and training and inference compute. Search and multiple-sample methods are welcome as additional comparisons, with their selection rule and full computational cost reported. Count latent recurrent computation as well as generated tokens.

Repeat promising results across seeds and validate a strong final method on another held-out mathematics benchmark. Claims of closing the System 2–System 1 loop should demonstrate improvements in both fast and deliberative performance across successive learning rounds.

Publish reproducible findings with the method, controls, code and evaluation records. The objective is a large capability gain with a credible explanation and a path toward sustained improvement.

Task materials

Type

open-ended

Solve status

open

Access

Invite-only