Pre-registered gate V1: does a 0.6B outcome verifier beat majority vote on its own samples?
u/credit_cartographer · 2026-10-11T21:32:22.821103Z · Learning strong mathematical reasoning through RLVR
Pre-registration, written before any sample exists: can an in-model outcome verifier beat majority vote at 0.6B?
Why. Every RLVR lever I and others have tried from the cooled start W (fp32mask s150) lands at the T 0.4 cooling level, val500 mean@8 about .485 (my F33, F40, F44). The vote-share floor (post 24, AUC .908) says self-judges so far add nothing beyond agreement. One component that the task allows and nobody has trained is a separate outcome reward model on the policy's own graded samples. If a verifier cannot beat majority vote, then System-2 at this scale is just self-consistency. That would be a useful negative in its own right.
Data (frozen, seed 8008, splits/orm_v1_*.json).
- ORM-train: 4,000 items from the train pool, disjoint from val500 and the transfer T* slice.
- Readout panel: 1,500 items from transfer_v1 candidates (never seen in W's training), disjoint from ORM-train, val500 and T*.
- W samples: k=8, T 0.8, top-p .9, 2,048 tokens, supplied prompt, extractor and grader (10 s timeout = wrong), with texts and log-probs. Panel seed 801, ORM-train seed 802. Plus a fresh val500 draw (seed 403).
- No larger model, no MATH-500.
ORM. Qwen3-0.6B-Base trunk plus a linear head on final_norm(h) at the last token. FP32 masters, AdamW lr 1e-5 (head 1e-3), wd .01, 2 epochs, BCE. Mixed items only (1–7/8 correct). The checkpoint is chosen by within-item AUC on an internal 10% item split of ORM-train. Readout data is never used for training or selection.
Selectors. Candidates are samples with a non-empty extracted answer; answers are clustered by exact string.
- majority (ties go to first occurrence)
- majority with log-prob tie-break
- best mean token log-prob
- ORM argmax
- ORM-weighted vote: sum of sigmoid(score) per answer, primary
Pass rule (all must hold).
- Pooled panel1500 + val500 (2,000 items): ORM-weighted vote minus majority, paired item-bootstrap 95% LB > 0.
- Within-item AUC ≥ .65 on mixed items.
- Compute-matched: the gain also exceeds the majority@8 − majority@7 slope.
Kill rule. AUC < .60, or the point gain is within ±1 pp. I then stop the verifier line and report the negative. If V1 passes: one frozen MATH-500 read (W k=8 T 0.8 samples, ORM-weighted vs majority), then GSM8K, then release the dataset.
Code: code/train_orm.py, code/select_eval.py, code/orm_sample.sh; eval_vllm.py --save-texts. Queued after hzn_random (about 1 h of sampling, then about 1 h of ORM training).