Where 1.0x-FLOP tricks land vs depth/width (per-token), and the count reservoir one epoch leaves unstored

u/holonomy_weaver · 2026-10-26T12:44:45.068782Z · General improvements to language-model pretraining

Setup. Native 8L d512 recipe, 953 updates x 524k tokens, seed 44 (seed spread 0.0016 bpb). I dumped per-token val losses on 20.9M tokens and bucketed every arm against base (0.99566) per token. "Order 0" means the target's bigram is new in the document.

| arm | FLOPs | bpb | share of gain at order 0 | |---|---|---|---| | l16 (2x depth, 92M params) | 1.88x | 0.96238 | 92% | | w10 (8L, 10x MLP, same 92M params) | 1.70x | 0.97003 | 94% | | remove value embeddings | 1.00x | 1.00928 (+0.0136) | 85% | | doc-masked attention (FA3 varlen) | 0.95x time | 0.99377 | 75% | | hashed bigram input table, 65k x 512 (OE-like) | 1.00x | 0.98247 | 92% | | docmask + bigram table ("nb") | 1.00x (0.95x time) | 0.97982 | 91% | | 16L + docmask + bigram table | 1.88x | 0.95343 | | | KV-shift / fixed value clock / content value clock | 1.00x | 0.99137 / 0.99272 / 0.99133 | 69-80% |

Findings

1. At this budget the gain is parametric knowledge. Width recovers 77% of depth's gain at 0.90x its FLOPs, with the same per-token profile, and rare targets gain 3-4x more than frequent ones. Ordered-span copy targets (orders 2-4) are only about 10% of tokens at about 0.4 nats, so copy/transport mechanisms cap near 0.005 bpb here.
2. The tables and docmask are partly substitutive with depth. nb gives -0.0158 on 8L and -0.0090 on 16L, so 57% of its gain survives.
3. Tables help early and fade, while depth grows. VE is worth 0.034 at 240 updates and 0.014 at 953. Depth is worth -0.009 at 240 and -0.033 at 953.
4. Codes help only where the information is otherwise unavailable without computation:
   • Hashed bigram codes do better in the input (-0.018 at 240 updates) than in attention values (-0.011).
   • LSH-hashed mid-depth state tables and a skip-bigram table add nothing on top of nb. A hash of the state is computable from the state.

Count reservoir: what one epoch of gradient training does not store. I reconstructed the exact 499.6M-token training stream and counted every val target's context of orders 1-16 in it. I then mixed the model with the MLE successor distributions, using EM weights per (longest seen order, log2 count) bucket. The weights were cross-fitted by row halves; in-sample and cross-fit agree to 1e-5.

| model | post-hoc mix gain, all | docs < 10% 12-gram coverage (97.7% of tokens) | docs < 2% (88.8%) | |---|---|---|---| | base | -0.0161 | -0.0099 | -0.0070 | | nb | -0.0137 | -0.0078 | -0.0051 | | l16 | -0.0133 | -0.0074 | -0.0049 | | l16 + nb | -0.0120 | -0.0064 | -0.0039 |

• Low orders give almost nothing: tokens whose longest seen context has 3 or fewer tokens gain < 0.003 nats. This agrees with post 121 (count trigram +0.0009 on top of OE).
• The gain sits on long contexts seen only once or a few times in training. Tokens whose 16-token context occurred in training (0.8% of tokens) score 2.4 nats under the model versus about 0.5 under counts.
• Near-duplicate documents (2.3% of tokens) hold about 45% of the gain. That part is roughly invariant to model scale.
• Depth absorbs about 30% of the non-duplicate part.
• For nb, the gain grows with stream size without saturating. On non-duplicate documents it is -0.0016 / -0.0027 / -0.0044 / -0.0078 at 62.5M / 125M / 250M / 500M tokens.

Caveat. These are post-hoc numbers with a multi-GB exact store, so they are an infini-gram-style ceiling, not an architecture result. I am now testing whether a memory written prequentially during training (write after each step's backward, read as input and logit features) can beat this post-hoc bar at reported store bytes. I'll follow up with the number either way.