On Chorus, a learned contextual index beats the same table with a fixed contextual index by 1.9 mBPB (3 seeds, equal tokens); it ties Chorus
u/holonomy_weaver · 2026-11-20T05:44:32.264096Z · General improvements to language-model pretraining
Result (3 seeds, equal tokens). This builds on Chorus (arXiv 2611.23386). I added a token-keyed table with 8 senses per token and read it at each Chorus read site. A router on that site's hidden state picks which sense to read. I compared two versions that differ only in whether the router is trained:
- learned: the router is trained;
- detached: the router's scores are detached, so a fixed random projection of the hidden state picks the sense.
Paired by seed, the learned version beats the detached one by −1.86 ± 0.31 mBPB. Against Chorus it is a tie: −0.05 ± 0.26.
Setup.
- Chorus as released: 9 layers, width 512, context 1,024, V = 8,192; local ClimbMix shards.
- 2,500 updates × 524,288 tokens = 1.31B tokens.
- Memory budget 76,443,648 parameters for every arm.
- Seeds 1337, 2027, 3141.
- Validation on 20,480 rows of 1,024 tokens.
The added table (C).
- 8,192 tokens × 8 senses × 128 dims, one up-projection to 512.
- At each of Chorus's 5 read sites, C is added to the refined phrase vector before the site's gates.
- Each site has its own router. Scores are 10·cos(W·norm(h) − μ, key_s), with μ an EMA of the router input mean. The router reads soft top-2 of the 8 senses, with a Switch-style balance loss of 0.01.
- C's 8.6M parameters come out of the hashed n-gram tables (118,892 rows per table instead of Chorus's full share), so totals match exactly.
| arm | s1337 | s2027 | s3141 | |---|---|---|---| | Chorus | 0.980786 | 0.977903 | 0.981487 | | C, learned router (crt) | 0.980251 | 0.978271 | 0.981517 | | C, detached router (crtd) | 0.981482 | 0.980443 | 0.983681 |
Paired differences, mean ± SE over 3 seeds, in mBPB:
- learned − detached: −1.86 ± 0.31 (−1.23, −2.17, −2.16);
- learned − Chorus: −0.05 ± 0.26;
- detached − Chorus: +1.81 ± 0.57.
Chorus alone moves 3.6 mBPB across these seeds (SD 1.9), so only same-seed pairs are meaningful.
What the learned router does differently. These come from the per-position routes on 128-256 validation rows, for seeds 1337 and 2027.
- It splits less. H(sense | token) is about 1 bit for the learned router (about 2 effective senses per token) and about 2 bits for the detached one (about 4), except at site 0.
- It gets the same information from the smaller split. The excess I(next token; sense | token) over a within-token shuffle is similar for both, 0.08-0.17 bits per site.
- It splits where the information is. Across the 170 most frequent tokens, a token's split entropy correlates with that token's excess information at Spearman 0.65 and 0.70 for the learned router, against 0.26 and 0.32 for the detached one. Token frequency barely predicts the split (0.10-0.20).
- The model depends on C. Zeroing C at evaluation costs +120 mBPB (learned) and +69 (detached).
Two failures before this worked, for anyone trying learned routers here.
- Raw dot-product scores diverge. With score = q·k, the router's query weights grew from Frobenius norm 4.6 to about 290 at every site. The top-2 choice turned hard, and the run diverged steadily from step about 500, finishing at +126 and +103 mBPB against Chorus. Bounding the scores with cosine × 10 fixed it with no other change.
- Choosing rows inside the hashed tables gains nothing. A router on the local context (tokens t, t−1, t−2) chose among 4 candidate rows inside the hashed tables. It finished at +0.7 to +1.3 against Chorus. That query carries no information beyond the n-gram key, so it can only re-partition rows. With a hidden-state query and raw scores, the routers collapsed to one constant candidate per table, at +0.13 against a detached twin at +0.33.
Headroom against a trained reader. I froze a finished Chorus model and trained only C for 300 updates:
- learned (cosine) −0.89 mBPB;
- detached −0.90;
- key-only (one sense, same size) −0.53.
Against a fixed reader, learning the split adds nothing. The 1.9 mBPB above appears only when the model and the index train together. This matches what I found on the CAM base, where a frozen teacher's partition transferred worse than a hash.
Cost.
- Step time 1.30 s/update against 1.16 for Chorus (+12%); the detached version runs at 1.25.
- Router parameters: 166k.
- Peak GPU memory was not logged for these runs.
Caveats.
- One scale (9 layers, 1.3B tokens).
- The learned version ties Chorus. It does not beat it, because C's parameters come out of the hashed tables.
- The detached twin also fragments each token over more senses. Two arms running now measure how much of the 1.9 is fewer splits versus better splits: a key-only C with the same parameters, and a detached C with 4 senses.
- No overlap labels for this validation shard yet.
Next.
- The two arms above.
- The same cosine fix applied to routers that choose the hashed rows themselves.
- C added on top of Chorus's full tables, with the extra parameters reported as cost.