General improvements to language-model pretraining

AI · Machine learning

General improvements to language-model pretraining

Research goal

Develop general methods for language-model pretraining that improve held-out prediction for comparable training resources, or preserve quality while substantially reducing resource requirements. Aim for a contribution substantial enough for the main track of NeurIPS, ICML, or ICLR.

The central question is what new method or principle would matter to researchers who pretrain language models. Seek an advance that transfers beyond one small model and one training recipe, with evidence explaining why it works. Lower validation loss is evidence of progress; novelty, significance and generality determine the scientific value.

Starting point

The supplied baseline adapts autoresearch, derived from nanochat. Its decoder has eight layers, width 512, 50,332,176 parameters, a 2,048-token context and an 8,192-token vocabulary. It includes rotary embeddings, QK normalization, value embeddings and a combined Muon/AdamW optimizer.

The first comparison uses 499,646,464 training tokens: 953 optimizer updates of 524,288 tokens each. Training uses a microbatch of 64 sequences with four accumulation steps. The supplied implementation preserves the reference model and optimizer equations, corpus, tokenizer procedure, document packing and validation calculation, while using a fixed-token schedule and correcting a pinned-memory transfer race in the upstream loader.

| Measurement | Recorded RTX 4090 baseline | | --- | ---: | | Validation cross-entropy, nats/token | 2.785243 | | Validation perplexity | 16.2038 | | Validation bits per byte | 0.994008 | | Training loop, including initial compilation | 18.29 minutes | | Complete run, including final validation | 18.91 minutes | | Peak allocated GPU memory | 22.27 GiB |

These are measured results from one seed. Validation uses 20,971,520 target tokens; cross-entropy and perplexity exclude special tokens, consistently with the bits-per-byte calculation. Lower is better for all three measures, and perplexity is the exponential of cross-entropy.

The task supplement ZIP contains the baseline source, preparation and verification scripts, pinned dependencies, recorded results and provenance. Start with its README and code/README.md. The preparation script downloads the ClimbMix data and builds the reference tokenizer; the attention backend also has a pinned download. Training starts from random initialization. Routine experiments should be practical on one 24 GB GPU, generally within two hours. The supplied baseline uses a fixed token budget, not a five-minute training limit.

Directions to explore

• Optimization and update geometry: develop update rules or explanations that improve learning across depths and training stages. Muon and SOAP are relevant existing methods. Replacing an optimizer with a published alternative is a baseline comparison; a contribution needs a new mechanism, extension or insight.
• Low-bit learning: investigate quantization-aware optimization, gradient estimation and the relationship between precision and learning. Compare with Quartet and Quartet II. On RTX 4090, simulated quantization can test learning behavior; it does not establish native FP4 speed or memory savings.
• Architecture and parameterization: explore attention, residual pathways, normalization, nonlinearities, parameter sharing or recurrent computation. Starting references include Hyper-Connections, manifold-constrained Hyper-Connections, Dynamic Tanh, Derf and recurrent-depth models. Account for every recurrent pass when comparing computation.
• Objectives and other learning mechanisms: investigate auxiliary prediction, representation learning, regularization, credit assignment or principled use of the fixed training corpus. Improvements should ultimately benefit or explain ordinary causal next-token prediction.

These are starting ideas, not prescribed methods. A clear finding that motivates a simple new method can be more valuable than combining many known techniques.

Experimental comparisons

Use the supplied fixed-token baseline for the first comparison. Retain its tokenizer, held-out data, context length and scoring rule. Report validation cross-entropy, perplexity, bits per byte and learning curves. Keep the training microbatch at 64 sequences. The effective batch may change when it is part of the method; preserve the training-token budget and report the resulting number of optimizer updates.

Matching tokens alone does not match computation. Report parameter count, training FLOPs or a justified estimate, wall time, peak memory and additional passes, auxiliary networks or preprocessing. If a method uses more capacity or computation, compare it with a conventional baseline given a comparable budget. Longer runs and larger models are useful for testing generality when both methods receive comparable treatment.

Compare promising methods against relevant, well-tuned alternatives. Optimization claims should include the supplied Muon/AdamW optimizer and an appropriate AdamW baseline, with a fair tuning budget. Repeat promising results across seeds, isolate the mechanism with ablations, and test transfer to another model scale, backbone, data distribution or training regime appropriate to the claim.

The supplied validation stream is available for development. Confirm final results on an untouched held-out set; evaluation examples must not enter training or method-specific supervision. If studying corpus sampling, retain the same eligible training corpus and token budget, and account for selection costs. Broader data changes belong in separately controlled experiments.

Evidence of progress

A strong result should establish what is new, why it matters, which mechanism produces the advantage and how far the evidence supports generalization. A general empirical or theoretical finding can also qualify if it changes how researchers understand or design pretraining methods.

A small isolated loss gain, a favorable seed, extra compute or an unexplained combination of known tricks is insufficient on its own. Publish reproducible findings with the method, relevant comparisons, code and evaluation records. The objective is a broadly useful pretraining discovery, supported by affordable initial experiments and convincing follow-up evidence.

Task materials

Type

open-ended

Solve status

open

Access

Invite-only