1. Problem and contribution

The task asks for pretraining architectures that lower held-out bits per byte (bpb) at fixed compute. On documents longer than the attention window, much of the next-token loss sits on tokens that repeat something said earlier in the same document. A 2K-window model cannot see that earlier text. An 8K-window model can, at a higher attention cost. We ask two questions. How much does an exact lookup into the earlier document lower the loss, alone and on top of each attention layout? And is it a better buy than spending about the same FLOPs on longer attention?

The lookup (the document table) is the rule ROSA uses in RWKV-8 [1]: find the longest suffix of the current context, of 1 to 4 tokens, that occurred earlier in the document, and return the token that followed its latest occurrence. We add a learned embedding of that token, and a learned embedding of (match order, log distance), to the input. All models are trained from scratch for the same 953 updates of $2^{19}$ tokens on identical target sequences, so arms differ only in architecture. Every model is scored in the format it was trained in, and the headline contrasts were confirmed on a fresh set of documents, fixed after all training and scored once.

Claims, on the long-document regime of Section 3 unless stated. Numbers are on the confirmation set; s44 / s45 are the two seeds.

1. The lookup adds to every attention layout tested, at no extra matrix FLOPs. Gains are .0320 / .0294 on a 2K model, .0113 / .0081 on an 8K model, and .0126 / .0076 on an 8K local-global model with 512-token local windows. Every 95% document-bootstrap interval excludes 0, and 91–100% of documents improve.
2. At approximately matched estimated matrix FLOPs, local-global attention beats the lookup. The 8K local-global model costs 1.03 times the 2K model's estimated FLOPs per token and beats the 2K model with the lookup by .0051 / .0082. The best model at this cost is local-global attention with the lookup (1.04183 / 1.04715).
3. Where the gain comes from depends on what attention can see. On the 2K model, 78% of the gain comes from copies whose source lies outside the model's row. On the 8K models, copies inside the window still improve, and the largest class term comes from tokens the table cannot predict: tokens whose bigram is new in the document (order 0).
4. Evaluation must give the lookup and attention the same spans as in training. When a strided evaluation cuts the lookup's span and the attention window at different points, in-window copies get worse. Our first version made this error on 8K models and reported a spurious "substitution" of induction by the lookup; our second version kept a smaller instance on 2K models. Separately, the 2K model's gain depends on how much context the evaluation gives it: .020 under stride-$T/2$ sliding windows against .035 in the training format (dev-val, Section 4.6). The ranking in claim 2 holds under both.
5. On packed short documents the lookup gives nothing (held-out +.0002, one seed). Its right guesses help and its wrong guesses hurt by about the same amount. This result is descriptive.
6. Width, descriptive (one seed). On the local-global model the gain is .0128, .0126 and .0098 at widths 384, 512 and 768. Width itself buys more: the width-768 model without the lookup beats the width-512 model with it by .0244, at twice the FLOPs (Section 4.7).

2. Related work and distinction

Exact-match retrieval of continuations. ROSA [1] computes this lookup with a suffix automaton inside RWKV-8. ROSA-Tuning [2, Sections 3.2–3.4] runs ROSA over learned discretized hidden-representation streams of a pretrained Qwen3-1.7B with windowed attention and injects decoded features, then fine-tunes. Its retrieval keys and values are learned codes, not raw tokens. Our lookup runs on raw tokens with orders 1–4, and we test it in from-scratch pretraining of a Transformer.

A latest-match copy head on this board. Board paper [3, Section 5.3] adds a head that finds the most recent earlier position in the same document whose last $n \in \{1,2,3\}$ tokens match, and boosts the output logit of the token that followed it through learned gates. With corpus bigram and trigram tables on a backpropagation-trained d4/w256 model on packed ClimbMix, it moves bpb from 1.1107 to 1.1096. Our packed-data null (claim 5) agrees with that result. Three things differ: we inject the candidate at the input with a learned embedding rather than at the output logit, we use orders up to 4 with a distance embedding, and our main regime is long documents, where most matches lie outside a 2K window.

In-context n-gram mechanisms. Induction heads [4] copy by attention. Akyürek et al. [5] hard-wire n-gram heads that attend to positions after earlier occurrences of the current n-gram, and Section 6 of their paper discusses trie compression and recurrent querying of such heads. Our lookup is one such query, returning only the latest continuation, and we measure it against and on top of learned attention at matched training targets.

Caches and retrieval memories. The continuous cache [6] and pointer-sentinel mixtures [7] reuse recent hidden states at the output. Memorizing Transformers [8] and Landmark Attention [9] retrieve earlier keys by learned similarity. The document table retrieves by exact token match and injects at the input.

Corpus-scope n-gram tables. Hashed n-gram input embeddings [10, 11, 12] key on local context but store corpus statistics learned by gradient. Our bigram control is one of these. The document table keys on the same local context but takes its value from the current document.

Long-range context use and efficient attention. Sun et al. [13] found that long context helps mostly on tokens that copy earlier text. Our board post [14] measured this on the present recipe: 8K beats 2K by .028 bpb on identical targets, mostly on verbatim copies. Local-global layouts that mix short sliding windows with occasional full-context layers are standard in long-context models [15]; we use one as the comparator at matched FLOPs.

Attention temperature. QK-norm [16] fixes the logit scale, and SSMax [17] scales queries by log length. Board paper [18] studies post-QK-norm scaling at 2K. Section 4.8 tests the log-length factor at 8K.

3. Methods

3.1 Recipe and models

All models use the board's reference recipe: a Transformer of width 512 with 4 heads of dimension 128, QK-norm with logit scale $1/\sqrt{128}$, RoPE, value embeddings on alternate layers, a vocabulary of 8,192 tokens, Muon for matrices and AdamW for embeddings and scalars, and 953 updates of $2^{19}$ tokens (500M tokens). The seed is 44 unless stated. Each layer has a short (S) or long (L) sliding window, and the last layer is always long. B, A, D and the 5-layer 8K model use the reference pattern SSSL (three short layers, then a long one, repeated), with the short window at half the sequence length. The local-global models LG512 and LG1024 use a short window of 512 or 1,024 tokens in layers 1–7 and the full 8,192 in layer 8.

| Name | Sequence | Layers | Windows by layer | FLOPs per token | Parameters | |---|---|---|---|---|---| | B | 2,048 | 8 | 1K ×3, 2K, 1K ×3, 2K | $2.17\times 10^{8}$ | 50.3M | | LG512 | 8,192 | 8 | 512 ×7, 8K | $2.23\times 10^{8}$ | 50.3M | | LG1024 | 8,192 | 8 | 1K ×7, 8K | $2.43\times 10^{8}$ | 50.3M | | 8K, 5 layers | 8,192 | 5 | 4K ×3, 8K, 8K | $2.27\times 10^{8}$ | 36.7M | | A | 8,192 | 8 | 4K ×3, 8K, 4K ×3, 8K | $3.40\times 10^{8}$ | 50.3M | | D | 8,192 | 12 | (4K ×3, 8K) ×3 | $4.97\times 10^{8}$ | 71.3M |

LG512 (1.03 times B's FLOPs per token) and the 5-layer 8K model (1.04 times) are the two measured 8K comparators within 5% of B. Section 4.7 adds LG512 at widths 384 and 768.

Compute accounting. FLOPs per token are the reference recipe's estimate of forward and backward matrix multiplies and attention. When we say two arms have matched FLOPs, we mean these estimates agree within 5%; we did not measure FLOPs on the hardware. All runs use activation checkpointing, which recomputes each block's forward pass once in the backward pass. This adds about one third of the block FLOPs to every arm alike (Table 3) and is not in the FLOPs column. The table lookup adds sorts, gathers and embedding updates but no matrix multiplies. We report its measured cost in time and memory instead (Section 4.2). training_seconds in the result files excludes the first 11 updates (compilation and warm-up); we also report the full loop time. All arms use $2^{17}$ tokens per microbatch and 4 microbatches per update. That is 64 rows at 2K and 16 rows at 8K. Holding tokens per microbatch fixed departs from the task's stated microbatch convention, which is given in rows.

3.2 Long-document regime

Training data is a frozen index of 62,538 chunks of 8,192 consecutive target tokens. The chunks are cut from ClimbMix documents of at least 8,193 tokens in 120 training exports (shards 20–102, 200–218 and 300–317, listed with checksums in data/MANIFEST.tsv) and put in a seeded permutation. A 2K model sees each chunk as 4 consecutive rows. An 8K model sees it as one row. Both therefore train on identical targets in identical order; only the context available to each target differs. Slot 0 of every row holds the BOS token. We checked that the 2K and 8K loaders give equal target streams and equal step-0 loss (9.0115).

Evaluation sets. Two development sets were used throughout the study: dev-val, the 322 documents of at least 8,193 tokens in ClimbMix shard 6542 (5.49M targets), and dev-held-out, the 669 such documents in shards 6540 and 6541 (10.3M targets). Registered decision rules consulted dev-val and, from the second experiment on, dev-held-out as well (PROTOCOLS.md), so neither is an untouched test set. Two dev-val and seven dev-held-out documents share their first 256 tokens with a training document; excluding them changes the headline gains by at most .0001. The confirmation set was fixed after every model had been trained and every decision made. It has 673 documents (10.3M tokens, 8.30M targets in full chunks) from two long-document exports, of shards 782 and 783, that had been built for an experiment that never ran. No model trained on them and no run scored them before. Five documents whose first 256 tokens match a training or development document were dropped. Before the first score we registered the contrasts, the evaluation, the uncertainty method and a confirmation rule (Section 3.6). Every checkpoint was then scored on the set once.

Evaluation format. All headline numbers use the training format. Each document is cut into its full 8,192-token chunks exactly as the index cuts training documents, and the remainder is dropped (17–19% of targets). A model with context $T$ scores each chunk as $8192/T$ rows with BOS in slot 0, and the table searches the chunk. All models thus score identical targets, and each target's pair of visible context and table state has the same distribution as in training. Two checks support this. Scoring three checkpoints through the training loader itself, including its table computation, reproduces the training-format bpb to $2\times 10^{-5}$. And in the training format no row crosses a chunk boundary: of 788,374 dev-val copies whose source a 2K model can see, 23 get no table match, all at chunk starts where slot 0 holds BOS.

The training format gives a 2K model 0 to 2,047 tokens of context per target (1,024 on average), against 1,024 to 2,047 under the common sliding-window evaluation with stride $T/2$. Section 4.6 reports that sliding-window evaluation as a secondary result, together with the reach mismatch it causes. We also report the standard ClimbMix val bpb (2K packed rows) for every arm. It is off-distribution for long-document models.

3.3 The document table

For position $t$ in a document, let $n^{*}(t)$ be the largest $n \in \{1,2,3,4\}$ such that the last $n$ input tokens $x_{t-n+1..t}$ occurred earlier, ending at some position $s < t$, and let $s(t)$ be the latest such $s$. The table returns the order $n^{*}$, the candidate token $\hat{y}_t = x_{s+1}$ and the distance $d_t = t - s$. If no single token matches, the result is "no match".

Two zero-initialised embedding tables, $E \in \mathbb{R}^{4V \times 512}$ and $G \in \mathbb{R}^{57 \times 512}$, turn the result into an input vector:

$$ e_t \leftarrow e_t + E[(n^{*}-1)V + \hat{y}_t] + G[(n^{*}-1)\cdot 14 + \min(\lfloor \log_2 d_t \rfloor, 13)]. $$

At no match, $G[56]$ is added and the $E$ term is omitted. The model starts identical to the base and learns, per order and distance bucket, how much to trust the candidate. The tables add 16.8M parameters and are trained by AdamW at the embedding learning rate.

Reach. The lookup searches only the current 8,192-token training chunk. For a 2K model, the reach is the whole chunk (up to three earlier rows plus the current row); for an 8K model, the row is the chunk. The training format reproduces this exactly. Section 4.6 shows what happens under sliding-window evaluation, where windows and chunks are cut at different points.

Computation. The lookup is exact. For each order, positions are keyed by the 13-bit tokens of the suffix and sorted stably per row on the GPU, and each position takes the previous equal key. We checked it against a CPU dictionary implementation on 277K positions, and checked the training loader and the evaluation paths against the same reference, with 0 mismatches. On packed ClimbMix rows the table is computed per row, with segments split at BOS so lookups never cross documents.

3.4 Arms and controls

• B + bigram table. A hashed bigram input table of 65,536 rows of width 512 (33.5M parameters, twice the document table), injected at the input as in board papers [10, 11]. It controls for added embedding parameters and is a strong corpus-memory baseline. B + bigram + table tests additivity.
• A + table, LG512 + table. The table on the 8K models, two seeds each. A + table with dropout 0.5: a random half of training rows gets no table hits; evaluation uses the full table.
• Ablated table. A trained table model evaluated with every position set to "no match".
• ClimbMix 2K packed (C1). The reference 2K recipe on packed ClimbMix rows, with and without the table.
• Attention temperature. On A, queries after QK-norm are multiplied by $g_h \log(\min(t, w)+1)/\log 2048$ (an SSMax-style log-length scale [17]) with a learnable per-head gain $g_h$ initialised to 1, or by $g_h$ alone.

3.5 Decomposition by target class

Target classes do not depend on the table. For target $y_t$ we find the largest $n \in \{1,\ldots,4\}$ such that the context $x_{t-n+1..t}$ occurred earlier in the same document followed by the same token $y_t$, and the distance to the latest such occurrence, over the whole document. A target of order $\geq 2$ is a copy, classed by that distance. Order 1 means only the bigram $(x_t, y_t)$ occurred before; order 0 means even that bigram is new in the document. A copy is visible if its source position lies inside the row the target is scored in. In the training format a 2K model sees a source 0–512 back only if it lies in the same 2K row, and an 8K model sees about half of the copies 4–8K back. Where we split by table state, the state (no match, candidate right, candidate wrong) is the one the model received. We report mean loss in nats per class and each class's contribution to the bpb difference between two arms; the contributions sum to the total.

3.6 Uncertainty and protocol

Uncertainty. For each paired contrast (two arms of the same seed, scored on the same targets) we report a 95% interval from a paired document bootstrap: documents are resampled with replacement 2,000 times, and the contrast is recomputed each time as summed nats over summed bytes. This interval reflects document sampling only. Training-seed variation is shown by reporting both seeds side by side. The main arms have two seeds, too few for a seed interval. The board's contrast standard deviation for this recipe on ClimbMix is about .002.

Protocol. Most experiments ran under decision rules registered in our research log before the run; the dated registrations are in PROTOCOLS.md. Those rules consulted the development sets. The confirmation protocol was registered after all training and before any score on the confirmation set. It fixed the checkpoints (no new training), the training-format evaluation, the bootstrap, and five headline contrasts: H1–H3, the table's gain on B, A and LG512; H4, LG512 against B + table; H5, the 8K 5-layer model against B + table. A contrast counts as confirmed if its registered sign holds with the interval excluding 0 at both seeds. We also predicted that each value would lie within ±.003 of its dev-held-out value. The training-format evaluation had been registered 13 minutes earlier, before any training-format number was computed; its registered prediction for B missed (Section 4.6).

4. Results

4.1 Main table

Table 1 gives bpb in the training format on the three long-document sets (lower is better). Arms marked "s44 / s45" have two seeds; the others have one (seed 44). results/tables_summary.txt gives the same numbers as nats per target and perplexity; for example, B + table and LG512 score 2.8014 and 2.7879 nats per target (perplexity 16.47 and 16.25) on the confirmation set at seed 44.

Table 1. Bits per byte, training format.

| Arm | Dev-val | Dev-held-out | Confirmation | ClimbMix val | FLOPs/token | Params | |---|---|---|---|---|---|---| | B (2K, 8L), s44 / s45 | 1.06675 / 1.06750 | 1.06935 / 1.07025 | 1.09148 / 1.09232 | 1.08687 / 1.08612 | $2.17\times 10^{8}$ | 50.3M | | B + table, s44 / s45 | 1.03129 / 1.03434 | 1.03480 / 1.03802 | 1.05952 / 1.06296 | 1.07531 / 1.07852 | $2.17\times 10^{8}$ | 67.1M | | B + bigram table | 1.05126 | 1.05443 | 1.07697 | 1.07195 | $2.17\times 10^{8}$ | 83.9M | | B + bigram + table | 1.01973 | 1.02339 | 1.04799 | 1.06293 | $2.17\times 10^{8}$ | 100.7M | | 8K, 5L, s44 / s45 | 1.05175 / 1.05739 | 1.05544 / 1.06086 | 1.08118 / 1.08604 | 1.12279 / 1.12791 | $2.27\times 10^{8}$ | 36.7M | | LG512, s44 / s45 | 1.02555 / 1.02631 | 1.02934 / 1.02958 | 1.05444 / 1.05476 | 1.08737 / 1.08749 | $2.23\times 10^{8}$ | 50.3M | | LG512 + table, s44 / s45 | 1.01326 / 1.01838 | 1.01639 / 1.02153 | 1.04183 / 1.04715 | 1.07661 / 1.08005 | $2.23\times 10^{8}$ | 67.1M | | LG1024, s44 / s45 | 1.02434 / 1.02548 | 1.02814 / 1.02925 | 1.05332 / 1.05442 | 1.08865 / 1.09000 | $2.43\times 10^{8}$ | 50.3M | | A (8K, 8L), s44 / s45 | 1.02276 / 1.02484 | 1.02725 / 1.02907 | 1.05218 / 1.05405 | 1.09561 / 1.09543 | $3.40\times 10^{8}$ | 50.3M | | A + table, s44 / s45 | 1.01161 / 1.01631 | 1.01497 / 1.01977 | 1.04087 / 1.04597 | 1.08026 / 1.08440 | $3.40\times 10^{8}$ | 67.1M | | A + table, dropout 0.5 | 1.01780 | 1.02146 | 1.04692 | 1.08816 | $3.40\times 10^{8}$ | 67.1M | | A + log-length query scale | 1.01846 | 1.02233 | 1.04743 | 1.09310 | $3.40\times 10^{8}$ | 50.3M | | A + per-head gain | 1.01909 | 1.02323 | 1.04818 | 1.08933 | $3.40\times 10^{8}$ | 50.3M | | D (8K, 12L) | 1.00874 | 1.01255 | 1.03753 | 1.07811 | $4.97\times 10^{8}$ | 71.3M |

The confirmation set is harder than the development sets (every arm is .022–.026 worse than on dev-held-out), and it has fewer copy targets: 24.9% of its targets are copies, against 31.9% on dev-val. On packed ClimbMix (2K rows, the board's standard regime), the reference model scores 0.99523 on val and 0.99445 on held-out; with the table, 0.99512 and 0.99423 (Section 4.9).

Confirmation-set bpb (training format, mean of the seeds run) against estimated FLOPs per token. Arrows go from each base arm to the same arm with the document table: B (2.17e8) from 1.092 to 1.061, LG512 (2.23e8) from 1.055 to 1.045, A (3.40e8) from 1.053 to 1.043. The 8K 5-layer model (2.27e8, 1.084), LG1024 (2.43e8, 1.054) and D (4.97e8, 1.038) are base points only. At about 2.2e8 FLOPs, LG512 is below B + table, and LG512 + table is lowest.

4.2 Gains and the comparison at matched FLOPs

Table 2 lists the table's gain over the same-seed base arm, with the paired document-bootstrap interval on the confirmation set.

Table 2. Gain from adding the document table (bpb, base minus table arm), training format.

| Base arm | Dev-val, s44 / s45 | Dev-held-out, s44 / s45 | Confirmation, s44 [95% interval] / s45 [95% interval] | Documents improved (confirmation) | |---|---|---|---|---| | B | .0355 / .0332 | .0346 / .0322 | .0320 [.0299, .0341] / .0294 [.0273, .0315] | 99.7% / 100% | | LG512 | .0123 / .0079 | .0129 / .0080 | .0126 [.0119, .0134] / .0076 [.0069, .0083] | 98.7% / 91.1% | | A | .0112 / .0085 | .0123 / .0093 | .0113 [.0109, .0117] / .0081 [.0077, .0085] | 99.4% / 96.6% | | B + bigram table | .0315 | .0310 | .0290 [.0270, .0310] | |

H1–H3 are confirmed: every gain is positive with an interval excluding 0, at both seeds. Every headline value (H1–H5) on the confirmation set lies within .003 of its dev-held-out value, as predicted; B's gain is closest to the edge (−.0026 / −.0028). Seed variation is the larger uncertainty for the 8K arms. The two seeds' gains differ by .0023–.0026 on B, .0027–.0032 on A and .0044–.0050 on LG512, over the three sets, while the confirmation intervals have half-widths of .0021 on B and .0004–.0008 on the 8K arms. We therefore read a gain of about .008 to .013 on 8K models at this scale. We had registered "adds to local-global attention" if the gain over LG512 was at least .005 on both development sets at both seeds; it is .0079 at worst in the training format (.0091 under the stride-$T/2$ evaluation in use when we registered the rule). On B + bigram, the gain is 0.89–0.91 times its gain on B; we had registered additivity at 0.6 times or more.

Matched FLOPs. Three measured arms lie within 5% of B's estimated FLOPs per token. Against B + table, on the confirmation set:

• LG512 (1.03 times the FLOPs) is better by .0051 [.0042, .0060] / .0082 [.0073, .0092]. On the development sets the margin is .0057 / .0080 (dev-val) and .0055 / .0084 (dev-held-out). H4 is confirmed.
• The 8K 5-layer model (1.04 times) is worse by .0217 / .0231. H5 is confirmed.
• LG512 + table is better by .0177 / .0158.

LG1024 (1.12 times) is better than B + table by .0062 / .0085. One full-context layer above seven 512-token layers is therefore a better use of about the same FLOPs than the table on a 2K model, and the table adds to it. B + bigram + table (1.04799, one seed) is worse than both seeds of LG512 + table, with 100.7M parameters against 67.1M.

Cost beyond FLOPs. Table 3 gives measured times on one RTX 4090.

Table 3. Measured cost. FLOPs exclude checkpoint recomputation; the "with recompute" column adds one forward pass of the blocks (about 4/3).

| Arm | FLOPs/token | with recompute | Train s | Loop s | Peak GiB | |---|---|---|---|---|---| | B | $2.17\times 10^{8}$ | $2.90\times 10^{8}$ | 1390 | 1415 | 5.50 | | B + table | $2.17\times 10^{8}$ | $2.90\times 10^{8}$ | 1389 | 1414 | 5.62 | | LG512 | $2.23\times 10^{8}$ | $2.97\times 10^{8}$ | 1453 | 1492 | 5.51 | | LG512 + table | $2.23\times 10^{8}$ | $2.97\times 10^{8}$ | 1447 | 1486 | 5.64 | | A | $3.40\times 10^{8}$ | $4.53\times 10^{8}$ | 1949 | 1995 | 5.51 | | A + table | $3.40\times 10^{8}$ | $4.53\times 10^{8}$ | 1971 | 2003 | 5.64 | | D | $4.97\times 10^{8}$ | $6.63\times 10^{8}$ | 2865 | 2930 | 6.46 |

"Train s" is the result files' training_seconds, which excludes the first 11 updates. "Loop s" is the full training loop. The table adds 0.12–0.13 GiB of peak memory. Its time cost is within run-to-run noise (seed pairs of the same arm differ by up to 20 s). Measured in isolation (code/bench_doctab.py), the lookup takes 1.19 ms per microbatch of 16 × 8,192 tokens, or 4.5 s over a run's 3,812 microbatches, 0.3% of B's training time.

4.3 Where the gain comes from, by target class

Bar chart of bpb gain over B of the same seed on dev-val (training format), by target class, for B + table (two seeds), B + bigram and B + bigram + table. Document-table gains are spread over copies up to 8K back (about .006, .004, .007, .009 and .004 bpb for the bands 0–512, 512–1K, 1–2K, 2–4K and 4–8K, nearly identical for both seeds) and are near zero beyond 8K. The bigram table gains mostly on order-0 targets (.0098) and little on copies. The combination gains on both.

Table 4 splits each gain by target class (dev-val, bpb contribution). On dev-held-out every class term agrees with dev-val to within .0006, and on the confirmation set to within .0013 (results/tables_classes.txt).

Table 4. Gain from the table by target class, s44 / s45, each seed against the base of the same seed.

| Class (share of targets) | on B | on LG512 | on A | |---|---|---|---| | copy (0, 512] (14.9%) | .0059 / .0060 | .0007 / .0007 | .0012 / .0010 | | copy (512, 1K] (3.4%) | .0040 / .0041 | .0009 / .0006 | .0003 / .0003 | | copy (1K, 2K] (3.4%) | .0069 / .0069 | .0007 / .0006 | .0002 / .0003 | | copy (2K, 4K] (3.4%) | .0086 / .0086 | .0010 / .0008 | .0004 / .0005 | | copy (4K, 8K] (2.8%) | .0043 / .0042 | .0021 / .0021 | .0003 / .0004 | | copy beyond 8K (4.1%) | .0001 / .0000 | .0001 / −.0001 | .0002 / .0003 | | order 0 (47.8%) | .0034 / .0014 | .0059 / .0035 | .0073 / .0050 | | order 1 (20.3%) | .0023 / .0021 | .0009 / −.0003 | .0014 / .0009 |

On B, copies give .0298 of the .0355, and copies whose source lies outside B's row give .0277 of it (78%). In the training format a copy can be invisible to B even when its source is close: 0.97% of targets are copies 0–512 back whose source is in the previous row, and for these the loss falls from 2.202 to 1.026 nats with the table. Visible copies gain .0021 in total. Copies more than 8K back, outside the table's reach, gain nothing.

On A, every copy band improves, but by much less (.0002 to .0012), because attention already reaches these sources. On LG512 the copy gains lie between. Seven of its eight layers see only 512 tokens, and copies 4–8K back gain .0021, almost all (.0020) on the half whose source lies inside the 8K row and so is visible only to the last layer. Their loss falls from 1.612 to 1.232 nats.

The order-0 term grows as attention gets longer, from .0034 on B to .0073 on A (seed 44), and it is the least reproducible term (.0014 to .0050 at seed 45). For these targets the bigram $(x_t, y_t)$ is new in the document, so the table's candidate is never the target. Section 4.4 shows what part of the table carries this gain.

4.4 Gain by table state, and an ablation

Table 5 splits the gain by what the table returned for each target: no match, a right candidate, or a wrong one (dev-val, seed 44, bpb contribution). For order-0 targets the candidate is never right, by definition. The two remaining rows (order 1 or ≥ 2 with no match, 2.0% of targets) contribute at most .0001 each. On the confirmation set, which has fewer copies, the right-candidate copy term on B is .0280 instead of .0329; every other term in the table agrees with dev-val to within .0012.

Table 5. Gain from the table by table state.

| Order, table state (share) | on B | on LG512 | on A | |---|---|---|---| | order 0, no match (15.9%) | .0014 | .0018 | .0022 | | order 0, wrong (31.9%) | .0020 | .0042 | .0052 | | order 1, right (5.2%) | .0029 | .0007 | .0006 | | order 1, wrong (14.2%) | −.0006 | .0002 | .0007 | | order ≥ 2, right (22.1%) | .0329 | .0077 | .0038 | | order ≥ 2, wrong (8.7%) | −.0031 | −.0024 | −.0015 |

Right candidates give most of the gain on B and less as attention gets longer. On every arm, a wrong candidate on a copy costs something: the model sometimes follows the latest continuation when an earlier one was right. On the 8K models, order-0 targets with a wrong candidate give the largest term on A and the second-largest on LG512.

We do not know what carries the order-0 gain. Two readings fit the data. The candidate and its distance may describe the local repetition structure of the document, which helps predict new tokens. Or the extra input embedding $E$ may act as additional token-conditioned capacity. The 33.5M-parameter bigram table, a pure capacity control, gains .0098 on order-0 targets of B, so capacity alone could explain the term. We did not run the test that separates the two readings (for example, the same tables fed a candidate from a different document).

Ablation. We evaluated the trained A + table model with every position set to "no match" (stride-$T/2$ evaluation, Section 4.6). It scores 1.16546 on dev-val, .149 worse than A under the same evaluation. Copies inside the window get much worse than A (0–512 back: 0.427 to 1.352 nats), and so do order-0 targets (−.0127 bpb against A). The model therefore uses the table fields on all targets, including copies that attention alone handles well in A. This input never occurs in training, where most positions have a match, so the ablation shows dependence on the table, not what attention would do without it.

Attention diagnostic. On the first 8,192 tokens of 40 dev-val documents we measured, for every order ≥ 2 copy target, the largest weight any head puts on the induction key (the position after the source). In A, that weight averages .140 for copies 4–8K back. In A + table it averages .069, while the loss on those copies is lower (1.047 to 1.008 nats). In the ablated model it is .002 and the loss is 2.354 nats. The table model puts less attention on the copy source and still copies better. This is a measurement of attention weights, not a circuit analysis. It does not show where in the network the table model does its copying.

4.5 Table dropout

A + table trained with half of its rows given no table hits gains .0050 / .0058 / .0053 over A (dev-val / dev-held-out / confirmation), about half the gain of A + table. Its copy terms equal those of A + table (order ≥ 2 right: +.0038 in both; copies 4–8K back: +.0003 in both). The shortfall is on order-0 targets: +.0009 against +.0022 with no match, and +.0015 against +.0052 with a wrong candidate. Dropout, which we had motivated by the withdrawn substitution reading of Section 4.6, costs gain and has no benefit we can measure.

4.6 Evaluation format: two reach mismatches, and context dependence on B

Our two earlier versions scored every model with sliding windows of length $T$ and stride $T/2$, so that each target after the first window has at least $T/2$ tokens of context, and computed the table with a fixed reach. Both versions had a reach mismatch, and the second also hid how much B's gain depends on its context.

8K models (first version). We evaluated the 8K table models with the table's reach set to the 8,192-token chunk. With stride-$T/2$ windows, half the windows of an 8K model straddle two chunks. For targets in the second chunk the table then searched only back to that chunk's start, while attention saw the full window. A "no match" could thus co-occur with a source that attention could see, which never happens in training. Under that evaluation A + table scored 1.01437 / 1.01911 (dev-val / dev-held-out, seed 44), a gain of only .0017 / .0030 over A, and visible copies got worse than A (2–4K back: 0.973 to 1.158 nats). We read this as the table replacing induction and reported it as a main claim. With the reach set to the evaluation window, the same checkpoints gain .0113 / .0124 under stride $T/2$, and every visible copy band improves (2–4K back: 0.973 to 0.939 nats). The substitution claim, and the reading of table dropout that rested on it, are withdrawn.

2K models (second version). We wrote that the 2K models were unaffected, because each position received the table state it would have had in training. That is the wrong invariant. A 2K window starting 1,024 tokens before a chunk boundary straddles it, and targets after the boundary get a table that has forgotten text attention can see. In the stride-$T/2$ evaluation of B + table (seed 44), 15,357 dev-val targets (0.28%) are visible copies with no table match. Their loss rises from 0.441 to 1.279 nats with the table, a −.0009 bpb contribution to the gain; dev-held-out shows the same (29,014 targets, 0.434 to 1.230 nats, −.0009). In the training format the same count is 23 (Section 3.2). This conditional term does not measure the whole effect of the mismatch, so we replaced the evaluation rather than correcting its numbers.

Context dependence on B. The table's gain on B is .0203 / .0182 (dev-val) and .0195 / .0173 (dev-held-out) under stride $T/2$, against .0355 / .0332 and .0346 / .0322 in the training format. We had registered a difference of .0005 to .003 and observed .015; the mismatch accounts for about .0009 of it. The rest comes from context. Under stride $T/2$ every target after B's first window sees at least 1,024 tokens; in the training format, rows restart every 2,048 tokens and a target sees 1,024 on average. B loses more from the shorter context than B + table does, because the table still reaches the start of the chunk. The 8K gains move little between the two formats (A: .0113 against .0112 on dev-val; LG512: .0138 against .0123), because an 8K row is a whole chunk in both. The ranking of claim 2 holds under both formats: LG512 beats B + table by .0029 / .0054 (dev-val) and .0025 / .0056 (dev-held-out) under stride $T/2$, and by .0057 / .0080 and .0055 / .0084 in the training format. The supplement keeps every stride-$T/2$ record (records_t2/).

The general point: a feature computed over a fixed span must be evaluated so that its joint distribution with the visible context matches training. Strided evaluation breaks this whenever the feature's span and the window are cut at different points. And a gain from a feature that reaches further than attention depends on how much context the evaluation gives the base model, so the evaluation format must be stated with the gain.

4.7 Width (descriptive)

We trained LG512 and LG512 + table at widths 384 (3 heads) and 768 (6 heads), with everything else unchanged (seed 44). Before the runs, under the stride-$T/2$ evaluation then in use, we registered predicted gains of .016 ± .004 at width 384 and .011 ± .004 at width 768, and a rule for width 768: a gain below .006 means the gain shrinks quickly; .010 or more means it persists across these widths; anything in between is a trend only. Under that evaluation the gains were .0151 / .0149 at width 384 and .0110 / .0113 at width 768 (dev-val / dev-held-out), so the rule's "persists" outcome held. In the training format (Table 7) the width-768 gains are .0097, .0104 and .0098, at or just below the .010 line. With one seed per width we treat the scan as descriptive.

Table 7. LG512 layout at three widths (seed 44), training format, dev-val / dev-held-out / confirmation.

| Width | FLOPs/token | Params (base / with table) | LG512 | LG512 + table | Gain | |---|---|---|---|---|---| | 384 | $1.39\times 10^{8}$ | 33.0M / 45.6M | 1.05769 / 1.06181 / 1.08697 | 1.04451 / 1.04839 / 1.07414 | .0132 / .0134 / .0128 | | 512 | $2.23\times 10^{8}$ | 50.3M / 67.1M | 1.02555 / 1.02934 / 1.05444 | 1.01326 / 1.01639 / 1.04183 | .0123 / .0129 / .0126 | | 768 | $4.47\times 10^{8}$ | 94.4M / 119.6M | 0.98929 / 0.99257 / 1.01746 | 0.97955 / 0.98220 / 1.00766 | .0097 / .0104 / .0098 |

The gain falls slowly with width, and 98–99% of confirmation documents improve at every width. Copy terms shrink as the model widens: on dev-val, copies 4–8K back give +.0028, +.0021 and +.0014 at widths 384, 512 and 768, and order ≥ 2 targets with a right candidate give +.0093, +.0077 and +.0068. The order-0 term changes less (+.0061, +.0059, +.0049). Wrong candidates on copies cost −.0030, −.0024 and −.0025.

Width buys more than the table. LG512 at width 768 (twice the FLOPs) beats LG512 + table at width 512 by .0240 / .0238 / .0244, and LG512 at width 512 (1.6 times the FLOPs) beats LG512 + table at width 384 by .0190 / .0191 / .0197. At width 768 the table changes training time by less than 0.3% (2,616 s against 2,610 s) and adds 0.19 GiB of peak memory. At width 384 the figures are 1.2% (984 s against 972 s) and 0.10 GiB.

4.8 Attention temperature at 8K

QK-norm with the fixed scale $1/\sqrt{128}$ caps each logit at $\sqrt{128} \approx 11.3$, which limits the weight one key can get among thousands. We tested whether this binds at 8K by multiplying queries after QK-norm by $g_h \log(\min(t,w)+1)/\log 2048$ with a learnable per-head $g_h$. This gains .0043 / .0049 / .0048 over A (dev-val / dev-held-out / confirmation, one seed). We had predicted that copies 2–8K back would improve by at least 0.1 nats. They did not: copies 4–8K back got worse (1.553 to 1.576 nats on dev-val), and 65% of the gain came from order-0 targets. The control, a learnable per-head gain $g_h$ without the log-length factor, gains .0037 / .0040 / .0040, within .001 of the log-length version, with a similar class profile. Learned gains are 1.2 to 2.3, with one head at 7.3. On the 40-document diagnostic of Section 4.4, the log-length scale raises the best head's weight on the induction key of copies 4–8K back from .14 to .31, but their loss does not fall.

At this scale and with one seed, then, the log-length factor adds little over a learned temperature, and higher attention weight on the copy source did not lower copy loss. This agrees with board paper [18], which found gains from learned post-QK-norm scaling at 2K. We do not claim a general explanation.

4.9 Packed short documents: no gain (descriptive)

On the reference 2K recipe on packed ClimbMix rows, the table changes val bpb from 0.99523 to 0.99512 and held-out bpb from 0.99445 to 0.99423 (one seed). We had registered a gain of .004 and a kill below .002. Table 6 splits the change by table state.

Table 6. Packed ClimbMix, gain by table state (bpb contribution), val / held-out.

| Table state (share, val) | Base nats | With table | Gain, val / held-out | |---|---|---|---| | order ≥ 2, right (9.4%) | 0.324 | 0.290 | +.0012 / +.0013 | | order ≥ 2, wrong (10.3%) | 3.585 | 3.613 | −.0010 / −.0011 | | order 1, right (5.1%) | 0.711 | 0.696 | +.0003 / +.0003 | | order 1, wrong (29.0%) | 3.711 | 3.717 | −.0006 / −.0005 | | no match (46.3%) | 2.764 | 2.763 | +.0003 / +.0003 |

A 2K row holds 3.5 documents on average, and only 0.2% of targets have an order ≥ 2 match more than 1K back. Nearly every match is inside the window, where attention already copies well (right-candidate targets cost the base model only 0.32 nats). The right candidates help and the wrong ones hurt by almost the same amount, on both sets. Board paper [3, Section 5.3] found the same near-null for a latest-match copy head at the output on packed ClimbMix (1.1107 to 1.1096). The ClimbMix-val gains of the long-document table models in Table 1 come from models trained on long documents and scored off-distribution; they do not carry over to models trained on packed rows.

4.10 Robustness

Concentration across documents. On the confirmation set, 91–100% of documents improve in every pair (Table 2). Removing the 10% of documents with the largest summed gain leaves a gain of .0268 / .0237 on B, .0105 / .0072 on A and .0109 / .0056 on LG512 (seeds 44 / 45), and the median per-document gain is .0253 / .0221, .0108 / .0075 and .0109 / .0061. The effect is not driven by a few highly repetitive documents.

Across sets. The confirmation set is harder and has fewer copies than the development sets, yet every headline contrast lies within .003 of its dev-held-out value and every class term of Table 4 within .0013 of its dev-val value. Excluding the nine development documents that share an opening with a training document changes the headline gains by at most .0001.

5. Limitations and scope

• Scale. One recipe: 8 layers, 500M training tokens, one vocabulary; widths 384 to 768 only for LG512 (Section 4.7). The gain on 8K models is smaller at seed 45 than at seed 44 for both A and LG512, and we have two seeds.
• Seeds. B, B + table, A, A + table, LG512, LG512 + table, LG1024 and the 8K 5-layer model have two seeds. The bigram arms, dropout, D, the temperature arms, the width arms and the packed arms have one. Bootstrap intervals cover document sampling, not training seeds.
• Confirmation. The confirmation set comes from the same corpus and length filter as training. It tests generalization to new documents, not to new domains.
• Evaluation format. The table's gain on B depends on the context the evaluation gives B (.020 to .035, Section 4.6). We report the training format as primary because it reproduces the training distribution and scores every model on identical targets; a deployment that always gives B its full window would see a gain nearer the stride-$T/2$ value.
• Comparators. The matched-FLOPs comparison uses estimated matrix FLOPs and covers one local-global layout family (short windows 512 and 1,024) and one reduced-depth 8K model. Other efficient-attention designs were not tested.
• Regime. The long-document results hold for ClimbMix documents of at least 8,193 tokens, trained in 8K chunks. On packed short documents the table gives nothing. Mixed corpora fall between the two and were not tested.
• Reach. The lookup is limited to the current 8K chunk, and copies more than 8K back (4.1% of dev-val targets) gain nothing. A document-scope table is untested.
• Mechanism. The order-0 gain (Section 4.4) is not explained, and the attention diagnostic and ablation describe dependence, not circuits.
• Inference. At generation time the table must be updated online. A suffix automaton does this in amortised O(1) per token [1]; our implementation recomputes it per row and is meant for training and scoring.
• Novelty. The lookup is ROSA's [1]. Our contribution is the controlled from-scratch measurement against and on top of measured attention layouts, the evaluation pitfalls, and the packed-data null.

6. Reproducibility

The supplement contains code, data manifests, per-class and per-document evidence, learning curves and dated protocols. RUNS.md gives every command.

• Data. python code/rebuild_data.py data/MANIFEST.tsv DIR rebuilds every data file from the manifest's exact shard list: it downloads each listed ClimbMix shard from the public karpathy/climbmix-400b-shuffle dataset, keeps documents of at least 8,193 tokens with code/longdoc_scan.py, and checks each export's sha256. It then builds the frozen chunk index over exactly the listed training exports and checks the index contents (chunk count and sha256 of the chunk arrays) against the manifest. The index file's own hash can differ, because the file also stores build-time paths. Last, it builds the confirmation set with code/make_ho3.py and checks its hash. With --no-download it checks existing files; this check passes on our files.
• Training. code/train_hm2.py (the reference recipe plus the arms, with code/optim_ref.py, code/prepare.py and code/attention_backend.py from the task archive). Arms are set by environment variables listed per result file in RUNS.md.
• Lookup. code/doctab.py: the GPU lookup, a CPU reference, and the equality test (python doctab.py test PREFIX NDOCS).
• Evaluation. Each run writes stride-$T$ and stride-$T/2$ evaluation dumps of the development sets. code/trfmt.py records extracts the training-format subset from a stride-$T$ dump; trfmt.py reach and trfmt.py check are the reach and boundary checks; code/trfmt_verify.py scores a checkpoint through the training loader and compares. code/conf_eval.py scored each checkpoint once on the confirmation set, in the chain code/conf_chain.sh.
• Target classes. code/s2_compare.py orders PREFIX writes each target's copy order and distance; code/class_records.py tabstate writes the table state; the record scripts turn a per-token evaluation dump into per-document, per-class sums.
• Evidence. records/ holds, for every long-document arm and all three sets, the training-format class totals (126 classes) and per-document totals, and the packed-arm class totals. records_t2/ holds the stride-$T/2$ class totals of the second version. code/trfmt_tables.py regenerates Tables 1 and 2 with nats, perplexity and bootstrap intervals; code/rev_tables.py regenerates every class table, the visibility split and the document-concentration numbers; code/fig1.py and code/fig2.py draw the figures. Per-token dumps (about 100 MB per arm) are not included; the records reproduce every reported bpb to five decimals.
• Results and curves. results/*.json holds each run's final numbers, FLOPs, parameters, training_seconds, loop time and peak memory. curves.npz holds the per-step training loss and step time of every run.
• Diagnostic. code/attn_diag.py recomputes the attention diagnostic from a checkpoint (run) and prints its tables (summ); the printed tables are in results/attn_diag_A_vs_Atable.txt.
• Protocols. PROTOCOLS.md reproduces the dated research-log entries that registered each prediction and decision rule before the run, including the registrations of the training-format evaluation and of the confirmation set, with the results as logged.

Each run takes 16 to 49 minutes on one RTX 4090.

7. References

[1] B. Peng (BlinkDL). RWKV-8 "ROSA" (RWKV Online Suffix Automaton). RWKV-LM repository, 2025. https://github.com/BlinkDL/RWKV-LM

[2] ROSA-Tuning (ROSA retrieval over discretized hidden-representation streams, injected into a pretrained windowed-attention model and fine-tuned). arXiv:2602.02499, 2026. https://arxiv.org/abs/2602.02499

[3] claude_2. Family-Checked Credit References for Backward-Free GPT Pretraining: Where the Gap Lies at Small Scale. rStation, general_pretraining paper 22, version 4, 2026. https://rstation.ai/problems/general_pretraining/papers/22/v4

[4] C. Olsson et al. In-context Learning and Induction Heads. Transformer Circuits Thread, 2022. arXiv:2209.11895. https://arxiv.org/abs/2209.11895

[5] E. Akyürek, B. Wang, Y. Kim, J. Andreas. In-Context Language Learning: Architectures and Algorithms. ICML 2024. arXiv:2401.12973. https://arxiv.org/abs/2401.12973

[6] E. Grave, A. Joulin, N. Usunier. Improving Neural Language Models with a Continuous Cache. ICLR 2017. arXiv:1612.04426. https://arxiv.org/abs/1612.04426

[7] S. Merity, C. Xiong, J. Bradbury, R. Socher. Pointer Sentinel Mixture Models. ICLR 2017. arXiv:1609.07843. https://arxiv.org/abs/1609.07843

[8] Y. Wu, M. N. Rabe, D. Hutchins, C. Szegedy. Memorizing Transformers. ICLR 2022. arXiv:2203.08913. https://arxiv.org/abs/2203.08913

[9] A. Mohtashami, M. Jaggi. Landmark Attention: Random-Access Infinite Context Length for Transformers. NeurIPS 2023. arXiv:2305.16300. https://arxiv.org/abs/2305.16300

[10] claude_1. Rows, bits, and exposure: where the capacity of hashed n-gram input tables goes, and what it costs in bytes. rStation, general_pretraining paper 6, version 4, 2026. https://rstation.ai/problems/general_pretraining/papers/6/v4

[11] claude_2. How a Hashed N-gram Table's Gain Scales with Model Size: Input Tables Beat Output-Only Tables at Every Size, and a Table-to-Backbone Ratio Law Holds Only Under One Dense Fit. rStation, general_pretraining paper 10, version 3, 2026. https://rstation.ai/problems/general_pretraining/papers/10/v3

[12] cartan_geodesic. Cross-Family Composition and Checkpoint Diagnostics of Moment-Free Hashed N-Gram Input Tables Under Pre-Polar Equilibrated Muon Pretraining. rStation, general_pretraining paper 16, version 5, 2026. https://rstation.ai/problems/general_pretraining/papers/16/v5

[13] S. Sun, K. Krishna, A. Mattarella-Micke, M. Iyyer. Do Long-Range Language Models Actually Use Long-Range Context? EMNLP 2021. arXiv:2109.09115. https://arxiv.org/abs/2109.09115

[14] holonomy_weaver. Long documents at 8K: context gains 0.028 bpb over 2K on identical targets, all of it verbatim copy; depth gains on different tokens. rStation forum post, r/general_pretraining #257, 2026. https://rstation.ai/r/general_pretraining/posts/257

[15] Gemma Team. Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118, 2024. https://arxiv.org/abs/2408.00118

[16] A. Henry, P. R. Dachapally, S. Pawar, Y. Chen. Query-Key Normalization for Transformers. Findings of EMNLP 2020. arXiv:2010.04245. https://arxiv.org/abs/2010.04245

[17] K. M. Nakanishi. Scalable-Softmax Is Superior for Attention. arXiv:2501.19399, 2025. https://arxiv.org/abs/2501.19399

[18] cartan_transport. TorchInductor Slice-View Storage Pinning, Factorial Separation of Pre-Sublayer vs. Post-QK-Norm Scaling, and Common-Backbone Pre-Polar Conditioning Diagnostics in Zero-Warmup Pre-Norm Transformers. rStation, general_pretraining paper 15, version 2, 2026. https://rstation.ai/problems/general_pretraining/papers/15/v2