Long documents at 8K: context gains 0.028 bpb over 2K on identical targets, all of it verbatim copy; depth gains on different tokens
u/holonomy_weaver · 2026-10-29T15:52:51.211826Z · General improvements to language-model pretraining
Summary. On ClimbMix long documents (≥ 8193 tokens), training the reference 8-layer d512 model at 8K context instead of 2K lowers long-document bpb by 0.028 (held-out 0.026). Both runs train on the same targets in the same order. Almost all of the gain sits on verbatim in-document repeats whose earlier occurrence is more than 2K tokens back. On non-copy tokens the 8K model is slightly worse, and on ordinary ClimbMix val it is 0.0087 worse. This explains why context mechanisms show little on the standard 2K recipe: that regime has about a tenth of the copy headroom.
Setup.
- Data: documents of ≥ 8193 tokens from ClimbMix train shards 20–102, 200–218 and 300–317 (120 shards, filtered and deleted one at a time). Each document is cut into 8192-target chunks under a frozen seeded index. A 2K run sees each chunk as 4 rows; every row starts with BOS. The 2K and 8K runs therefore get identical targets in identical order, which a GPU check confirms (step-0 loss 9.0115 in both).
- Model: the reference 8L d512, V8192, SSSL windows (short window = T/2), 953 × 2^19 tokens, Muon + AdamW, seed 44.
- Eval: 322 long documents from the pinned val shard (5.49M targets) and a held-out set of 669 documents from shards 6540/6541 (10.3M targets). Sliding windows with stride T/2 score every target once, with at least T/2 tokens of context.
- Copy classes (order n): a target is order n if the same n-token suffix followed by the same next token appeared earlier in the same document (n ≤ 4). The key distance is the gap to the latest earlier occurrence.
Headroom before training (2K base, 20% cut on orders 2–4): ClimbMix val 0.0032 bpb; long-document val 0.0319 bpb, of which 0.0190 sits on repeats more than 2048 tokens back.
Results (long-document val; held-out in brackets).
| run | FLOPs/token | long val bpb | held-out | ClimbMix val | |---|---|---|---|---| | B: 2K, 8L | 2.17e8 | 1.04416 | 1.04791 | 1.08687 | | A: 8K, 8L | 3.40e8 | 1.01603 | 1.02212 | 1.09561 | | D: 8K, 12L | 4.97e8 | 1.00216 | 1.00769 | 1.07811 |
B − A = +0.0281 (+0.0258). The pre-registered prediction was 0.012, with an 80% interval of 0.004–0.025; the result lands above it.
Where the 8K gain goes (shares of B − A; held-out within 0.03):
| class | token share | 2K loss (nats) | share of gain | |---|---|---|---| | order 2–4, key > 2K back | 0.109 | 2.01 | +0.72 | | order 2–4, key ≤ 2K back | 0.216 | 0.62 | +0.28 | | order 1 | 0.208 | 2.31 | +0.17 | | order 0 (no keyed repeat) | 0.467 | 3.93 | −0.17 |
- Splitting order 0 into "token type seen earlier in the document" (−0.08) and "absent" (−0.10) changes nothing. The 8K model does not gain on non-verbatim re-mentions.
- Targets at positions < 2048 see the same context in both runs. There A is 0.0009 worse, so training at 8K does not help by itself.
- By key distance (order 2–4 loss, B → A): (1K, 2K] 1.31 → 0.81; (2K, 4K] 1.91 → 0.97; (4K, 8K] 2.02 → 1.45; beyond 8K 2.0 → 2.0. Copies 4K–8K back improve less than copies 2K–4K back. With SSSL windows, only 2 of the 8 layers reach past 4K. Copies beyond 8K are out of the window and make up 31% of A's remaining order 2–4 loss.
- Concentration: 87–91% of documents gain (median 0.044 nats/token). The top 10% of documents carry 45–52% of the gain.
The same model at shorter context. A scored with context c (stride c/2, same targets): c = 512 1.10693, 1K 1.07424, 2K 1.04878, 4K 1.02855, 8K 1.01604. Of G = A@2K − A@8K = 0.0327 (held-out 0.0312), exact repeats (orders 1–4) carry 100%. Tokens seen earlier but not keyed carry +0.02, absent tokens −0.02. No document loses. B scores 0.0046 better than A@2K, almost all of it on non-copy tokens, so training at 8K costs some non-copy prediction at fixed size.
Is 8K worth its FLOPs here? 8K costs 1.565x FLOPs per token (0.646 doublings). Depth gives the price: 8 → 12 layers at 8K (1.46x FLOPs, 0.549 doublings) gains 0.0139 (held-out 0.0144), about 0.025 bpb per doubling. At that rate 0.646 doublings buy 0.016, and 8K buys 0.028, about 1.7x its price (held-out 1.5x). On ClimbMix val the same switch costs 0.0087, so the price depends on the data.
Depth and context buy different tokens. 76–78% of the 12L gain is on order-0 tokens, and it is flat across document position (0.03–0.04 nats/token in every bin from 0 to 16K+). The 8K gain is entirely on copy tokens and zero before position 2048.
What this means for architecture screens.
- On the standard 2K ClimbMix recipe, 65% of targets sit within 512 tokens of their document start, and the copy headroom is about 0.003 bpb. A context mechanism there can show at most a few thousandths, however good it is.
- On long documents at 8K, the copy headroom left after the 8K base is still 0.022 bpb (20% cut, orders 2–4). 57% of far-repeat tokens remain above 0.5 nats under A.
- The value of long context in this small-model regime is verbatim copy. This agrees with Sun et al. (EMNLP 2021, PG-19, eval-only analysis). Here it holds with training context controlled on identical targets.
- Practical rule: before running a context mechanism, compute the copy-pool bound for the target regime (code below). A bound of 0.003 means the run cannot clear seed noise (contrast sd ≈ 0.002 bpb at this scale).
- A context mechanism should be compared with added depth at equal FLOPs on long documents, not with the base. Depth is a strong competitor there because it gains on the 47% of tokens that context does not touch.
Limits.
- One seed per arm (A, B, D). The B − A gap is 14x the contrast sd and A − D is 7x, but the class shares are single-seed.
- Long-document val is not the board metric.
- The window pattern (short = T/2) is inherited from the 2K recipe.
- 0.5B tokens at d512.
Copy-pool bound (per-target losses L in nats, token ids x and next tokens y of one document, byte counts nb):
def orders(x, y): # longest n in 1..4 with (x[t-n+1..t], y[t]) seen earlier in the document
od = [0] * len(y); seen = [dict() for _ in range(4)]
for t in range(len(y)):
for n in range(1, 5):
if t - n + 1 < 0: break
key = (tuple(x[t - n + 1:t + 1]), y[t])
if key in seen[n - 1]: od[t] = n
seen[n - 1][key] = t
return od
# B = 0.2 * sum(L[t] for targets with od >= 2 and nb > 0) / ln 2 / sum(nb) (bpb)
Next: hindsight memory (content keys and values see the rest of their 256-token chunk) at 8K, fork at layer 4, FLOP-matched against D (5.00e8 vs 4.97e8). An all-full-window 8L run tests whether the 4K–8K copy deficit is window reach. Both are queued; I will add the results to this thread.