Context-Demodulated Multi-Pathway Phrase Memory and Ridge-Regularized L2 Readout Calibration for Low-Bit N-Gram Language Models

u/cartan_geodesic · · v1

General improvements to language-model pretraining · Download Markdown · BibTeX

Abstract

Hashed N-gram embedding tables expand pretraining capacity at O(1) lookup cost, yet narrow-slice tables suffer from severe hash collisions, static Layer-0-only injection, and 3.35-4.90 mBPB loss under 2-bit post-training quantization. Across 385 ClimbMix-400B runs at depths L in {6, 8} (widths d in {384, 512}, V=8192), we study Context-Demodulated Multi-Pathway Phrase Memory (CoDeM-Ngram) and Multi-Readout Ridge-Regularized L2 Calibration. CoDeM-Ngram modulates the raw 4-slice lookup with a gate driven by a channel-rolled 3-token unigram summary, applies a residual causal depthwise Conv1d(k=4), projects to Layer 0 via a post-gated Muon-trained matrix W_proj, and routes the pre-projection demodulated phrase vector into alternating attention Value-Embedding (VE) skips via Muon-trained adapters W_pve^(l) at zero extra lookups or table bytes. At exact memory-parameter parity (57.93M params at L=6; 77.24M at L=8), full-horizon 953-step (499.65M-token) CoDeM-Ngram improves validation BPB over OE-Ember by -6.273 mBPB at L=8 (0.970154 vs. 0.976427; -6.378 mBPB on an untouched 20.97M-token confirmation split) and -7.902 mBPB at L=6, while beating equal-time OE-Ember (-2.554 mBPB at L=8; -3.183 mBPB at L=6), parameter-matched FactorEngram-style basis gating (-2.964 and -3.438 mBPB), and gate-detachment-matched Unigram-VE (-1.619 and -2.316 mBPB). Finally, folding a closed-form L2 calibration matrix into W_proj and {W_pve^(l)} improves 2-bit BPB by -0.874 to -0.880 mBPB, enabling 2-bit CoDeM-Ngram (6.52-6.73x module compression) to beat 16-bit OE-Ember by -3.14 to -3.86 mBPB.

Context-Demodulated Multi-Pathway Phrase Memory and Ridge-Regularized L2 Readout Calibration for Low-Bit N-Gram Language Models

1. Introduction and Related Work

Scaling input vocabulary and N-gram lookup tables decouples knowledge capacity from per-token FLOPs. Over-Tokenized Transformers (OE) [1] and SCONE [2] show that large multi-hash N-gram input tables substantially improve language modeling loss at fixed backbone FLOPs, while Engram [3], TN-gram [4], and FactorEngram [6] introduce structured compression and contextual routing for hashed phrase memories. Because storing two FP32 Adam moments triples table memory footprint, recent optimizer work replaces dense Adam states on token interfaces: Frequency-aware SGD [9] scales sparse embedding updates by per-row frequency, Adafactor [8] factors second-moment matrices into row and column moving averages, and Shah's Ember (OE-Ember) [5] adapts kilobyte-state factored row/column RMS normalization specifically to over-tokenized tables. Concurrently, Chung and Du [7] demonstrate post-training and quantization-aware low-bit N-gram embeddings (LowBit-OE).

Within the rStation general_pretraining benchmark, four published studies establish important constraints that inform our design:

  • Capacity vs. precision trade-offs (general_pretraining_6/v4 [10]): Demonstrates that hashed N-gram capacity scales primarily with row count and token exposure while tolerating aggressive bit-width reduction (2–4 bits), motivating our focus on combining parameter-matched architectural routing with 2-bit/3-bit post-training quantization.
  • Count-normalized moment-free writes vs. factored states (general_pretraining_11/v3 [11]): Shows that hashed N-gram rows require minimal optimizer history once updates are properly scaled by row exposure, and analyzes the gap between pure moment-free writes and factored-state alternatives (OE-Ember).
  • Shared value embeddings (general_pretraining_9/v1 [12]): Separates initialization scale, learned routing coefficients, and lookup cost when sharing value embeddings across attention layers. This directly motivates our $2\times 2$ factorial control (Section 4.3) separating $0.25I_d$ initialization and joint backpropagation from hashed N-gram phrase content.
  • Cross-family composition and unigram/N-gram routing (general_pretraining_16/v5 [13]): Analyzes how hashed N-gram input tables interact with backbone representations and unigram pathways under Pre-Polar Equilibrated Muon training.

Relationship to FactorEngram [6] and Engram [3]. The closest external architectural precedent is Yang et al.'s FactorEngram [6] (public September 28, 2026), which retrieves low-rank factorized N-gram coefficients, applies context-dependent basis-level gating before dictionary reconstruction, and smooths representations with a causal depthwise convolution, building on Engram's [3] context-gated lookup and convolution. Because FactorEngram [6] combines three ingredients—low-rank table factorization, basis-level contextual gating prior to projection, and causal depthwise convolution—we isolate its gating-and-convolution pattern directly on unfactorized OE-Ember narrow-slice tables at exact parameter parity (FE_basis_gate_isoparam and FE_basis_postconv_isoparam, Section 2.3 and Section 4.1), alongside full-table Engram (L4) and rank-$640$ factorized TN-gram (L3). Our contribution builds on and extends FactorEngram [6] in three specific, empirically isolated ways:

  1. Channel-rolled 3-token unigram-lag summary and dual pre/post gating (NG6-IsoParam vs. FE_basis_gate_isoparam, L4_engram, and L3_tngram): Replacing FactorEngram-style single-token basis gating (FE_basis_gate_isoparam, which gates raw slices by $2\sigma(W_{\text{fe}}w_{0,t})$ before $\text{Conv1d}_{\text{dw}}(k=4)$ and $W_{\text{proj}}$) with our channel-rolled 3-token unigram summary $c_t$ and dual pre/post gates (NG6-IsoParam) at exact parameter parity improves full-horizon 953-step validation BPB on all 3 seeds by $-1.597\text{ mBPB}$ at $L=6$ (1.013828 vs. 1.015425, CI $[-3.15, -0.05]$) and $-1.265\text{ mBPB}$ at $L=8$ (0.971853 vs. 0.973118, CI $[-1.68, -0.85]$; $-1.298\text{ mBPB}$ on fresh_rg42_dedup). Moreover, avoiding low-rank table factorization at the 110.5 MiB budget gives CoDeM-Ngram an $-11.05\text{ mBPB}$ advantage over Engram (L4, 1.023127) and $-11.66\text{ mBPB}$ over rank-$640$ TN-gram (L3, 1.023739).
  2. Zero-lookup multi-pathway reuse into deeper Value Embeddings (NG7-IsoParam vs. NG6-IsoParam and C2b_gate_det_isoparam): Neither Engram nor FactorEngram routes its Layer-0 demodulated phrase representation into deeper attention Value-Embedding (VE) skips. Routing the pre-projection demodulated phrase vector $\tilde{u}_t$ into alternating VE layers via $W_{\text{pve}}^{(l)}$ adds zero hash lookups and zero table bytes while improving full-horizon validation BPB by $-1.614\text{ to }-1.841\text{ mBPB}$ over Layer-0-only NG6-IsoParam at $L=6$ and $-1.684\text{ to }-1.699\text{ mBPB}$ at $L=8$ (totaling $-3.211\text{ to }-3.438\text{ mBPB}$ over FE_basis_gate_isoparam at $L=6$ and $-2.949\text{ to }-2.964\text{ mBPB}$ at $L=8$), and outperforming strictly gate-detachment-matched Unigram-VE (C2b_gate_det_isoparam) by $-2.090\text{ to }-2.316\text{ mBPB}$ at $L=6$ and $-1.603\text{ to }-1.619\text{ mBPB}$ at $L=8$ (Section 4.3).
  3. Closed-form multi-readout ridge-regularized L2 calibration folded into $W_{\text{proj}}$ and $\{W_{\text{pve}}^{(l)}\}$: Because CoDeM-Ngram feeds $\tilde{u}_t$ into linear readouts $W_{\text{proj}}$ (Layer 0) and $\{W_{\text{pve}}^{(l)}\}$ (VE skips), 2-bit post-training quantization error across the 4 concatenated hash slices is partially compensated at zero deployment overhead by folding a single ridge-regularized least-squares matrix $(M^*)^\top$ into both readout pathways (Section 2.4 and Section 4.4).

2. Exact Executable Specification

All equations below match the exact executed computation in ngram_bench.py, run_ngram_nextgen.py, run_ngram_v8_suite.py, and run_ngram_v9_fixes.py (included in the supplementary archive). Throughout, $L \in \{6, 8\}$ denotes Transformer depth (number of layers) and $d = 64L \in \{384, 512\}$ denotes hidden model width.

2.1 Base-$(V+1)$ Polynomial N-Gram Hashing and Exact Parameter Parity (GPT.oe_ids)

Let $x = (x_0, \dots, x_{T-1}) \in \{0, \dots, V-1\}^T$ be a token sequence of length $T=2048$ with vocabulary size $V = 8192$ (gpt-neo-125M BPE). To distinguish sequence boundaries from token ID $0$, GPT.oe_ids (ngram_bench.py) pads out-of-bounds left positions with the sentinel value $V$: $$\tilde{x}_{t-\ell} = \begin{cases} x_{t-\ell} & \text{if } t - \ell \ge 0, \\ V & \text{if } t - \ell < 0, \end{cases} \qquad \ell \in \{1, 2\}. \tag{1}$$ With base $B = V + 1 = 8193$, each position $t$ forms exact base-$(V+1)$ polynomial integers for bigrams ($i=2$) and trigrams ($i=3$): $$I_t^{(2)} = x_t + B \tilde{x}_{t-1}, \qquad I_t^{(3)} = x_t + B \tilde{x}_{t-1} + B^2 \tilde{x}_{t-2}. \tag{2}$$ For $K=2$ hash functions per order, the 4 narrow slices are indexed by $j = (i-2)K + s \in \{0, 1, 2, 3\}$ for $i \in \{2, 3\}$ and $s \in \{0, 1\}$. Given base modulus $m$, slice $j$ has $m_j = m + 2j$ rows (m, m+2, m+4, m+6; no prime-rounding step is performed) and width $d_s = d / 4$, with bucket index: $$h_{t,j} = I_t^{(i)} \bmod (m + 2j), \qquad E_j \in \mathbb{R}^{(m+2j)\times d_s}. \tag{3}$$ The concatenated raw lookup vector is $e_{\text{raw},t} = \operatorname{Concat}_{j=0}^3\big(E_j[h_{t,j}]\big) \in \mathbb{R}^d$. In baseline OE-Ember (learned_proj_backoff=False), Layer 0 adds $e_{\text{raw},t}$ directly to the unigram embedding before the shared input RMSNorm: $x_t^{(0)} = \operatorname{RMSNorm}\big(E_{\text{uni}}[x_t] + e_{\text{raw},t}\big)$ (ngram_bench.py).

Exact Parameter Parity (NG7-IsoParam and Parameter-Matched Controls). Because contextual gating, depthwise convolution, readout, and VE routing add learned mixer parameters, we reduce the base modulus $m$ so that total memory-subsystem parameters (table_params + mixer_router_params) match baseline OE-Ember ($m=150,855$) to the single parameter:

  • Depth $L=6$ (width $d=384, d_s=96$, 3 VE layers $l \in \{1, 3, 5\}$): Baseline OE-Ember ($m=150,855$) has $\sum_{j=0}^3 (150,855 + 2j)\times 96 = (4\times 150,855 + 12)\times 96 = 57,929,472$ table parameters ($110.492\text{ MiB}$ BF16 module; $84,275,244$ whole-model parameters $= 160.742\text{ MiB}$ BF16 with the $26,345,772$-parameter / $50.251\text{ MiB}$ backbone).
    • CoDeM-Ngram (NG7-IsoParam) and Unigram-VE controls (C2b, C2b_gate_det, C2d) use $m = 148,547$, giving $(4\times 148,547 + 12)\times 96 = 57,043,200$ table parameters plus $886,272$ mixer/router parameters ($W_{\text{ctx}} \in \mathbb{R}^{2d\times d}$ $= 2d^2$, $W_{\text{proj}} \in \mathbb{R}^{d\times d}$ $= d^2$, $3\times W_{\text{pve}}^{(l)} \in \mathbb{R}^{d\times d}$ $= 3d^2$, and phrase_dw_conv $\in \mathbb{R}^{d\times 1\times 4}$ $= 4d$, all bias-free, totaling $6d^2 + 4d = 886,272$) for $57,929,472$ total memory parameters ($110.492\text{ MiB}$ BF16 module; $160.742\text{ MiB}$ BF16 whole model).
    • Layer-0-only NG6-IsoParam ($W_{\text{ctx}} \in \mathbb{R}^{2d\times d}$, $W_{\text{proj}} \in \mathbb{R}^{d\times d}$, phrase_dw_conv $\in \mathbb{R}^{d\times 1\times 4}$: $3d^2 + 4d = 443,904$ mixer parameters) uses $m = 149,699$ ($57,485,568$ table parameters) for $57,929,472$ total memory parameters.
    • Single-gate FactorEngram-style basis-gate controls FE_basis_gate_isoparam and FE_basis_postconv_isoparam ($W_{\text{fe}} \in \mathbb{R}^{d\times d}$, $W_{\text{proj}} \in \mathbb{R}^{d\times d}$, phrase_dw_conv $\in \mathbb{R}^{d\times 1\times 4}$: $2d^2 + 4d = 296,448$ mixer parameters) use $m = 150,083$ ($57,633,024$ table parameters) for $57,929,472$ total memory parameters.
  • Depth $L=8$ (width $d=512, d_s=128$, 4 VE layers $l \in \{1, 3, 5, 7\}$): Baseline OE-Ember ($m=150,855$) has $(4\times 150,855 + 12)\times 128 = 77,239,296$ table parameters ($147.322\text{ MiB}$ BF16 module; $127,571,472$ whole-model parameters $= 243.323\text{ MiB}$ BF16 with the $50,332,176$-parameter / $96.001\text{ MiB}$ backbone). NG7-IsoParam, C2b, and C2b_gate_det use $m = 147,267$ ($75,402,240$ table parameters $+ (7d^2 + 4d = 1,837,056)$ mixer parameters $= \mathbf{77,239,296}$); NG6-IsoParam uses $m = 149,315$ ($76,450,816 + (3d^2 + 4d = 788,480) = \mathbf{77,239,296}$); and FE_basis_gate_isoparam uses $m = 149,827$ ($76,712,960 + (2d^2 + 4d = 526,336) = \mathbf{77,239,296}$).

2.2 Factored Second-Moment (OE-Ember) and Empirical-Bayes (EB1) Table Updates

Let $G \in \mathbb{R}^{m_j \times d_s}$ be the accumulated FP32 gradient on slice table $E_j$ at step $t$, let $n_b = \sum_{\tau} \mathbb{I}[h_{\tau,j} = b]$ be the batch hit count for row $b$, and let $\mathcal{A}_j = \{b : n_b > 0\}$ be the active row set.

  • Implemented OE-Ember (oe_mfree_step in ngram_bench.py): Each table maintains FP32 factored buffers $r \in \mathbb{R}^{m_j}$ and $c \in \mathbb{R}^{d_s}$ ($2.303\text{ MiB}$ total across all 4 tables at $m=150,855$). At each step, row_mean_sq is non-zero on active rows ($b \in \mathcal{A}_j$) and zero on inactive rows ($b \notin \mathcal{A}_j$), while col_mean_sq averages over all $m_j$ rows: $$s_b^{\text{row}} = \begin{cases} \frac{1}{d_s}\sum_{k=1}^{d_s} G_{b,k}^2 & \text{if } b \in \mathcal{A}_j, \\ 0 & \text{if } b \notin \mathcal{A}_j, \end{cases} \qquad s_k^{\text{col}} = \frac{1}{m_j}\sum_{b=0}^{m_j-1} G_{b,k}^2 \quad (k \in \{1,\dots,d_s\}). \tag{4}$$ With $\beta_2 = 0.95$ (adam_betas = (0.8, 0.95)) and global step bias correction $\text{bias}_2 = 1 - \beta_2^t$, both buffers update over all indices via lerp_(..., 1 - beta2): $r_b \leftarrow \beta_2 r_b + (1-\beta_2) s_b^{\text{row}}$ ($\forall b \in \{0,\dots,m_j-1\}$, so inactive rows decay by $\beta_2$) and $c_k \leftarrow \beta_2 c_k + (1-\beta_2) s_k^{\text{col}}$. Defining $\hat{r}_b = \max(r_b / \text{bias}_2, 10^{-30})$ ($b \in \mathcal{A}_j$), $\hat{c}_k = \max(c_k / \text{bias}_2, 10^{-30})$, and $\tilde{c}_k = \hat{c}_k / \max(\frac{1}{d_s}\sum_{k'} \hat{c}_{k'}, 10^{-30})$, the active-row update places $\epsilon = 10^{-10}$ outside the square root: $$E_j[b, k] \leftarrow E_j[b, k] - \eta_{\text{oe}}(t) \frac{G_{b,k}}{\sqrt{\hat{r}_b \, \tilde{c}_k} + \epsilon}, \qquad b \in \mathcal{A}_j, \tag{5}$$ where the configured base learning rate $\eta_{\text{oe},0}$ is scaled by dmodel_lr_scale $= (d/768)^{-0.5}$ (ngram_bench.py: $\sqrt{2} \approx 1.4142$ at $d=384$, mapping 0.20 -> 0.2828 and 0.30 -> 0.4243; $\sqrt{1.5} \approx 1.2247$ at $d=512$, mapping 0.20 -> 0.2449 and 0.30 -> 0.3674) and follows the linear warmdown schedule.
  • Implemented Moment-Free Empirical-Bayes (EB1 in ngram_bench.py): Configured with homo_mfree=True, homo_alpha=0.50, homo_row_beta=0.0, homo_coord_gamma=0.0 and order-specific prior pseudo-counts $\kappa_2 = 2.0$ (bigrams) and $\kappa_3 = 5.0$ (trigrams), active gradients are damped by $\tilde{g}_b = G_b / \sqrt{n_b + \kappa_i}$ ($b \in \mathcal{A}_j$) and normalized by the global active-row RMS $s_{\text{glob}} = \sqrt{\max(\operatorname{mean}_{b\in\mathcal{A}_j, k} \tilde{g}_{b,k}^2, 10^{-30})}$: $$E_j[b] \leftarrow E_j[b] - \eta_{\text{oe}}(t) \frac{\tilde{g}_b}{s_{\text{glob}}}, \qquad b \in \mathcal{A}_j. \tag{6}$$

2.3 Context-Demodulated Multi-Pathway Phrase Memory (CoDeM-Ngram / NG7-IsoParam)

As implemented in NextGenGPT (run_ngram_nextgen.py), V8NextGenGPT (run_ngram_v8_suite.py), and V9NextGenGPT (run_ngram_v9_fixes.py), CoDeM-Ngram transforms the raw 4-slice lookup $e_{\text{raw},t} = \operatorname{Concat}_{j=0}^3(E_j[h_{t,j}]) \in \mathbb{R}^d$ through five stages:

  1. Channel-Rolled 3-Token Unigram Lag Summary: Let $w_{0,t} = \operatorname{RMSNorm}(E_{\text{uni}}[x_t]) \in \mathbb{R}^d$ be the RMS-normalized unigram embedding (wte), and let $w_{1,t}, w_{2,t}$ be causal temporal lags (zero-padded at $t < \ell$) circularly rolled along the channel dimension by $+1$ and $+2$ positions (torch.roll(..., shifts=1, dims=-1) and shifts=2) to break symmetry across lags without extra parameters: $$w_{1,t} = \operatorname{Roll}_{\text{chan},1}(w_{0,t-1}), \qquad w_{2,t} = \operatorname{Roll}_{\text{chan},2}(w_{0,t-2}), \qquad c_t^{\text{raw}} = \frac{1}{\sqrt{3}}\!\left(w_{0,t} + \frac{1}{\sqrt{2}}\,w_{1,t} + \frac{1}{2}\,w_{2,t}\right) \in \mathbb{R}^d. \tag{7}$$ To prevent Layer-0 gate gradients from perturbing wte, the gate input uses $c_t = \operatorname{stopgrad}(c_t^{\text{raw}})$ (gate_detach_wte=True).
  2. Dual Context Gate (bcd_ctx_proj) and Pre-Projection Demodulation: A single bias-free linear projection $W_{\text{ctx}} \in \mathbb{R}^{2d\times d}$ (nn.Linear(d, 2 * d, bias=False)) maps $c_t$ into pre- and post-projection gates around unity, and $g_t^{\text{pre}}$ directly modulates the raw lookup $e_{\text{raw},t}$: $$\big(g_t^{\text{pre}},\, g_t^{\text{post}}\big) = \operatorname{Chunk}_2\!\big(2\,\sigma(W_{\text{ctx}} c_t)\big) \in (0, 2)^d \times (0, 2)^d, \qquad \tilde{u}_t^{(0)} = e_{\text{raw},t} \odot g_t^{\text{pre}}. \tag{8}$$
  3. Causal Depthwise Local Smoothing (phrase_dw_conv): A depthwise 1D convolution $\text{Conv1d}_{\text{dw}}$ with kernel size $k=4$, groups $= d$ ($4d$ parameters, no bias), causal left zero-padding of 3 positions on $\operatorname{RMSNorm}(\tilde{u}^{(0)})$, and SiLU nonlinearity refines the demodulated slices: $$\tilde{u}_t = \tilde{u}_t^{(0)} + \operatorname{SiLU}\!\Big(\operatorname{Conv1d}_{\text{dw}}\!\big(\operatorname{RMSNorm}(\tilde{u}^{(0)})_{t-3:t}\big)\Big) \in \mathbb{R}^d. \tag{9}$$
  4. Cross-Slice Linear Readout (oe_proj) and Post-Gated Layer-0 Injection: A bias-free readout matrix $W_{\text{proj}} \in \mathbb{R}^{d\times d}$ mixes all 4 narrow slices into the backbone coordinate frame, applies $g_t^{\text{post}}$, and adds the result to $E_{\text{uni}}[x_t]$ prior to the shared input RMSNorm: $$e_{\text{L0},t} = \big(W_{\text{proj}} \tilde{u}_t\big) \odot g_t^{\text{post}}, \qquad x_t^{(0)} = \operatorname{RMSNorm}\!\big(E_{\text{uni}}[x_t] + e_{\text{L0},t}\big). \tag{10}$$
  5. Zero-Lookup Multi-Pathway Value-Embedding (Phrase-VE) Routing (pve_projs): In alternating attention layers $l \in \mathcal{L}_{\text{ve}}$ ($\{1, 3, 5\}$ at $L=6$; $\{1, 3, 5, 7\}$ at $L=8$), the pre-projection demodulated phrase vector $\tilde{u}_t$ (before $W_{\text{proj}}$ and $g_t^{\text{post}}$) is routed into the attention Value tensor $v_t^{(l)}$ through a layer-specific bias-free matrix $W_{\text{pve}}^{(l)} \in \mathbb{R}^{d\times d}$ modulated by the existing per-head attention value gate: $$v_t^{(l)} \leftarrow v_t^{(l)} + 2\,\sigma\!\big(g_{\text{ve}}^{(l)}(x_t^{(l,\text{in})})\big) \odot \big(W_{\text{pve}}^{(l)} \tilde{u}_t\big). \tag{11}$$ In NG7-IsoParam, gradients from all $l \in \mathcal{L}_{\text{ve}}$ flow jointly through $\tilde{u}_t$ into $\text{Conv1d}_{\text{dw}}$, $W_{\text{ctx}}$, and the N-gram tables $\{E_j\}_{j=0}^3$ (note that Phrase-VE branches from $\tilde{u}_t$ before $W_{\text{proj}}$, so VE gradients update the shared tables and Layer-0 gate/conv directly without passing through $W_{\text{proj}}$).

Controlled FactorEngram [6] Basis-Gate Baselines and Unigram-VE Controls (V9NextGenGPT in run_ngram_v9_fixes.py). To isolate our 3-token channel-rolled lag summary $c_t$ and dual pre/post gates against Yang et al.'s FactorEngram [6] basis-level gating on the exact same OE-Ember tables and parameter budget, FE_basis_gate_isoparam replaces Eqs. (7)–(8) with a single-token unigram gate $g_t^{\text{fe}} = 2\sigma(W_{\text{fe}}\operatorname{stopgrad}(w_{0,t}))$ ($W_{\text{fe}} \in \mathbb{R}^{d\times d}$), forms $\tilde{u}_t^{(0)} = e_{\text{raw},t} \odot g_t^{\text{fe}}$, applies Eq. (9), and projects via $e_{\text{L0},t} = W_{\text{proj}}\tilde{u}_t$; FE_basis_postconv_isoparam places the causal depthwise $\text{Conv1d}_{\text{dw}}(k=4)$ after $W_{\text{proj}}$. For the unigram-VE controls, C2d_isoparam (pve_detach=True) and C2b_gate_det_isoparam (pve_detach=False) route $\operatorname{stopgrad}(c_t^{\text{raw}})$ or $c_t^{\text{raw}}$ into $W_{\text{pve}}^{(l)}$ while holding gate_detach_wte=True ($c_t = \operatorname{stopgrad}(c_t^{\text{raw}})$ in Eq. 8) strictly fixed across C2d, C2b_gate_det, and NG7. (In V8NextGenGPT's C2b, setting pve_detach=False also left $c_t$ attached to wte inside Eq. 8; comparing C2b against C2b_gate_det and NG7 in Section 4.3 isolates both effects.)

Exact Initialization and Optimizer Assignment (NextGenGPT.init_weights and V9NextGenGPT.init_weights). All N-gram tables $E_j$ are zero-initialized. With $s = \sqrt{3/d}$, $W_{\text{proj}}$ uses standard uniform initialization $\mathcal{U}[-s, +s]$, $W_{\text{ctx}}$ (and $W_{\text{fe}}$) is initialized uniformly in $\mathcal{U}[-0.25s, +0.25s]$ (torch.nn.init.uniform_(self.bcd_ctx_proj.weight, -0.25 * s, 0.25 * s)), $\text{Conv1d}_{\text{dw}}$ (phrase_dw_conv) is zero-initialized (so $\tilde{u}_t = \tilde{u}_t^{(0)}$ at step 0), and $W_{\text{pve}}^{(l)} = 0.25\,I_d$ in NG7/C2b/C2b_gate_det/C2d (or $0$ in zero-init controls C2/C2c). All 2D mixer/router matrices ($W_{\text{ctx}}, W_{\text{proj}}, W_{\text{pve}}^{(l)}$) are stored in FP32 and optimized by Pre-Polar Equilibrated Muon (matrix_lr = 0.04, momentum $0.95$, Nesterov, 5 Polar Express steps); phrase_dw_conv (ndim == 3) is optimized by AdamW in the x0_params group (lr = 0.04, $\beta=(0.96, 0.95)$).

2.4 Non-Uniform Lloyd-Max PTQ and Multi-Readout Ridge-Regularized L2 Calibration

For low-bit post-training quantization (PTQ) to $b \in \{2, 3\}$ bits (quantize_tables_lloyd_max in run_ngram_suite3.py and run_ngram_v9_fixes.py, following Chung & Du [7]), each active row $r$ of slice table $E_j$ is RMS-normalized to form a 1D empirical distribution, clustered into $2^b$ non-uniform centroids ($4$ levels for 2-bit, $8$ levels for 3-bit) via 20 iterations of 1D Lloyd-Max k-means (initialized at empirical quantiles), and refined with 5 alternating least-squares steps updating per-row BF16 scales $s_{j,r} = (\sum_k E_j[r,k] c_{q_{r,k}}) / (\sum_k c_{q_{r,k}}^2)$ and nearest centroid assignments $q_{r,k}$, with centroids and row scales stored in BF16.

On $N_{\text{cal}} = \max(1, (24\times 32)/\text{mb})$ training batches ($768 \times 2048 = 1,572,864$ calibration tokens), let $\tilde{e}_t \in \mathbb{R}^d$ and $\tilde{q}_t \in \mathbb{R}^d$ denote the full-precision and quantized pre-projection features produced by Eq. (9), and let $d_t = \tilde{q}_t - \tilde{e}_t$ be the pre-projection quantization error vector. Accumulating the second-moment matrices $\Sigma_{qq} = \frac{1}{N}\sum_t \tilde{q}_t \tilde{q}_t^\top$, $\Sigma_{qe} = \frac{1}{N}\sum_t \tilde{q}_t \tilde{e}_t^\top$, $\Sigma_{ee} = \frac{1}{N}\sum_t \tilde{e}_t \tilde{e}_t^\top$, and $\Sigma_{dd} = \frac{1}{N}\sum_t d_t d_t^\top$ in FP64 with zero-centered ridge penalty $\lambda = \lambda_{\text{rel}} \frac{1}{d}\operatorname{tr}(\Sigma_{qq})$ ($\lambda_{\text{rel}} = 10^{-4}$), we solve: $$M^* = \arg\min_{M \in \mathcal{S}} \left\{ \frac{1}{N}\sum_{t=1}^N \big\|\tilde{e}_t - M^\top \tilde{q}_t\big\|_2^2 + \lambda \|M\|_F^2 \right\} = \big(\Sigma_{qq} + \lambda I_d\big)^{-1}\Sigma_{qe}. \tag{12}$$ Because $\tilde{q}_t$ feeds both the Layer-0 readout $W_{\text{proj}} \tilde{q}_t$ (Eq. 10) and the Phrase-VE projections $W_{\text{pve}}^{(l)} \tilde{q}_t$ (Eq. 11), folding $M^*$ into any readout matrix $W$ via $\widehat{W} \leftarrow W (M^*)^\top$ replaces $W \tilde{q}_t$ with $W (M^*)^\top \tilde{q}_t$ at zero inference overhead. In fit_and_eval_ptq_readout_suite (run_ngram_v9_fixes.py), we decompose this calibration along two axes:

  1. Structural Constraint Space $\mathcal{S}$: (a) Diag ($d$ parameters: $M_{\text{diag}} = \operatorname{diag}(\operatorname{diag}(\Sigma_{qe}) / (\operatorname{diag}(\Sigma_{qq}) + \lambda))$), (b) Block-Diag ($4 d_s^2$ parameters: $M_{\text{blk}} = \operatorname{block\_diag}(M^{(0)}, M^{(1)}, M^{(2)}, M^{(3)})$ with $M^{(j)} = (\Sigma_{qq}^{(j,j)} + \lambda I_{d_s})^{-1}\Sigma_{qe}^{(j,j)}$), and (c) Full ($d^2$ parameters: dense $M_{\text{full}} \in \mathbb{R}^{d\times d}$).
  2. Readout Pathway Isolation (full_l0_only, full_ve_only, full_combined): For models with Phrase-VE (NG7-IsoParam), we separately evaluate folding $M_{\text{full}}^\top$ into $W_{\text{proj}}$ only (full_l0_only), into $\{W_{\text{pve}}^{(l)}\}$ only (full_ve_only), and into both pathways simultaneously (full_combined), reporting both Layer-0 readout MSE reduction and VE readout MSE reduction alongside full-precision feature cross-slice correlation $R^2(\tilde{e}^{(a)}, \tilde{e}^{(b)})$ (from $\Sigma_{ee}$) and quantization-error cross-slice correlation $R^2(d^{(a)}, d^{(b)})$ (from $\Sigma_{dd}$).

Figure 1: CoDeM-Ngram architecture and FactorEngram/Unigram-VE controls, full-horizon L=6 and L=8 scaling and equal-time comparisons, common-modulus purity-stratified gains, and multi-readout 2-bit L2 calibration.


3. Experimental Setup, Data Provenance, and Resource Accounting

Data Provenance and Untouched Confirmation Split (verify_fresh_rg_provenance). All runs use the ClimbMix-400B benchmark dataset (V=8192):

  • Training set (train): Exact shards shard_00000.parquet through shard_00008.parquet ($9$ shards).
  • Validation set (val): Exact shard shard_06542.parquet, evaluated over 20,971,520 tokens ($320$ batches; inputs_int32_sha256 = 1e696a20964bfbe6...) on full-horizon runs and 5,242,880 tokens ($80$ batches; inputs_int32_sha256 = ebbee9f2a5526f13...) on 240-step runs.
  • Deduplicated held-out split (heldout_dedup): Constructed from the first half of shard_00009.parquet (row groups 0..41 out of 83 total row groups 0..82; 43,008 documents, 43,007 unique SHA1 hashes) by excluding all documents present in _HELDOUT_CROSS_SPLIT_DUP_HASHES (26 raw train–val overlaps, 23 train–heldout overlaps, and 2 val–heldout overlaps across the shard; 11 excluded inside rg 0..41) before deterministic token packing (build_eval_loader in ngram_bench.py; inputs_int32_sha256 = ebea7e3170d83b5e... at $20.97\text{M}$ tokens and 62b973a88dc286df... at $5.24\text{M}$ tokens).
  • Untouched final confirmation split (fresh_rg42_dedup): Constructed from the second half of shard_00009.parquet (row groups 42..82; 41,984 total documents). As verified programmatically by verify_fresh_rg_provenance(42, 83) (run_ngram_v9_fixes.py) and logged in exp_ngram_v9_*.json, rg 42..82 contains 16 documents matching _HELDOUT_CROSS_SPLIT_DUP_HASHES (train/val overlaps) and 1 document matching rg 0..41; that single rg 0..41 overlap is already among the 16 train/val duplicates (overlap_with_rg0_to_41_not_already_in_train_or_val = 0), so both cross-split filtering and strict union filtering against train ∪ val ∪ rg0..41 retain the exact same 41,968 documents (cross_and_union_filters_identical = true; inputs_int32_sha256 = 7fb047bff60c054a... at $20.97\text{M}$ tokens and 3dd8e72ed8132d52... at $5.24\text{M}$ tokens).

Full-Budget Depth $L=8$ ($d=512$) Microbatch Protocol Check (mb=64 vs. mb=32). The benchmark reference specifies depth $L=8$ ($d=512$, 8 layers, 4 heads, 50,332,176 backbone parameters), 953 steps at 524,288 tokens/step (499.65M tokens), and microbatch device_batch_size = 64 (grad_accum = 4). On a 24 GiB GPU, d=8 plain with mb=64 consumes 22,804 MiB peak CUDA VRAM, leaving insufficient headroom for a 147.32 MiB N-gram subsystem and its activations. Setting mb=32, grad_accum=8 preserves the 524,288 tokens/step effective batch size while halving activation VRAM to 11,701–12,534 MiB (minor bfloat16 gradient-accumulation rounding differences remain between 8 vs. 4 microbatches). On Seed 42 (exp_ngram_v8_d8full.json), d8_full953_plain_mb64 achieves 0.994026 val BPB (0.994763 heldout_dedup, 0.994343 fresh_rg42_dedup, 1075.3 s train) and d8_full953_plain_mb32 achieves 0.993637 val BPB (0.994301 heldout_dedup, 0.993985 fresh_rg42_dedup, 1088.6 s train)—an empirical difference of $0.39\text{ mBPB}$ ($6\times$ smaller than the $-23.80\text{ mBPB}$ N-gram gain and $16\times$ smaller than the $-6.27\text{ mBPB}$ gain of CoDeM-Ngram over OE-Ember). All compared $L=8$ models use the identical mb=32, grad_accum=8 setting.

Forward + Backward FLOPs Accounting (GPT.estimate_flops). Under the benchmark's per-token FLOPs formula GPT.estimate_flops() (ngram_bench.py: $6\times$ active non-embedding parameters $+$ causal sliding-window attention FLOPs $\sum_{l=1}^L 12\,n_{\text{head}}\,d_{\text{head}}\min(w_l, T)$ under window pattern SSSL with $w_l \in \{1024, 2048\}$):

  • Depth $L=6$ ($d=384, L_{\text{ve}}=3$): Baseline plain (and OE-Ember, whose table lookups are $O(1)$ indexing) executes $6\times 14,942,496 + 30,670,848 = \mathbf{120,325,824\text{ FLOPs/token}}$ (120.33 MFLOPs/token). CoDeM-Ngram (NG7-IsoParam) adds $886,272$ active mixer/router parameters ($6d^2 + 4d$), contributing $6\times 886,272 = \mathbf{5,317,632\text{ FLOPs/token}}$ (+5.32 MFLOPs/token), an increase of +4.42% (125,643,456 vs. 120,325,824 FLOPs/token).
  • Depth $L=8$ ($d=512, L_{\text{ve}}=4$): Baseline plain / OE-Ember executes $6\times 31,457,792 + 50,331,648 = \mathbf{239,078,400\text{ FLOPs/token}}$ (239.08 MFLOPs/token). CoDeM-Ngram adds $1,837,056$ active mixer/router parameters ($7d^2 + 4d$), contributing $6\times 1,837,056 = \mathbf{11,022,336\text{ FLOPs/token}}$ (+11.02 MFLOPs/token), an increase of +4.61% (250,100,736 vs. 239,078,400 FLOPs/token).

Timing Conventions and Reproduction Entry Point. In all tables, Train s (training_seconds) measures synchronized steady-state training time over steps 10..N-1 (excluding the first 10 compilation/warmup steps and periodic validation checks), Loop s (training_loop_seconds) measures the complete training loop from step 0..N-1 including compilation and the 6 periodic $2\text{M}$-token validation curve evaluations (every 190 steps), and Wall s (wall_clock_s) includes post-training full-split (20.97M-token) evaluations and Lloyd-Max PTQ suites. The supplementary archive is self-contained: ngram_bench.py provides the tokenizer, ClimbMix-400B dataloaders, GPT backbone, Muon/Ember optimizers, and provenance verifiers; run_ngram_suite3.py provides quantize_tables_lloyd_max; run_ngram_nextgen.py, run_ngram_v8_suite.py, and run_ngram_v9_fixes.py execute the training and PTQ suites; summarize_all_ngram_results.py recomputes ngram_authoritative_summary.json from the raw run JSONs; and generate_ngram_figure.py generates figure-1.png.

Training Memory Audit vs. Simulated Deployment Storage. We distinguish three memory metrics in Table 1 (audit_exact_memory_bytes in run_ngram_v8_suite.py):

  1. Training memory subsystem (Train Sub MiB) and Peak CUDA VRAM: During training, OE-Ember stores BF16 tables ($110.492\text{ MiB}$ at $L=6$; $147.322\text{ MiB}$ at $L=8$) plus FP32 factored second moments ($2.303\text{ MiB}$), totaling $112.795\text{ MiB}$ ($L=6$) and $149.626\text{ MiB}$ ($L=8$). CoDeM-Ngram (NG7-IsoParam) stores smaller BF16 tables ($108.802\text{ MiB}$ at $L=6$; $143.818\text{ MiB}$ at $L=8$), FP32 Ember row/col buffers ($2.268\text{ MiB}$ at $L=6$; $2.249\text{ MiB}$ at $L=8$), FP32 mixer/router master weights ($3.381\text{ MiB}$ at $L=6$; $7.008\text{ MiB}$ at $L=8$), and FP32 Muon/AdamW momentum buffers ($2.823\text{--}3.387\text{ MiB}$ at $L=6$; $6.027\text{ MiB}$ at $L=8$), totaling $117.275\text{--}117.838\text{ MiB}$ at $L=6$ (14,741 MiB peak CUDA VRAM vs. 13,964 MiB for OE-Ember) and $159.103\text{ MiB}$ at $L=8$ (12,532 MiB peak CUDA VRAM vs. 12,003 MiB for OE-Ember).
  2. BF16 Deployment Storage (Mod MiB / Whole MiB): Casting mixer/router weights to BF16 yields exact storage parity between CoDeM-Ngram, FE_basis_gate_isoparam, NG6-IsoParam, C2b, C2b_gate_det, and OE-Ember: $110.492\text{ MiB}$ module / $160.742\text{ MiB}$ whole model at $L=6$ (including the $50.251\text{ MiB}$ BF16 backbone), and $147.322\text{ MiB}$ module / $243.323\text{ MiB}$ whole model at $L=8$ (including the $96.001\text{ MiB}$ BF16 backbone).
  3. Simulated Low-Bit Storage (Mod MiB / Whole MiB): Low-bit evaluation uses exact dequantized Lloyd-Max simulation (quantize_tables_lloyd_max) and reports theoretical bit-packed table storage (2 or 3 bits/weight + BF16 per-row scales + $2^b$ centroids + BF16 mixer/router weights): at 2 bits, CoDeM-Ngram occupies $16.424\text{ MiB}$ module / $66.675\text{ MiB}$ whole model at $L=6$ ($6.73\times$ module-only compression; $2.41\times$ whole-model compression) and $22.605\text{ MiB}$ module / $118.606\text{ MiB}$ whole model at $L=8$ ($6.52\times$ module-only compression; $2.05\times$ whole-model compression).

4. Empirical Results, Causal Controls, and Mechanistic Analysis

4.1 Full-Horizon Comparisons at Depth $L=8$ and $L=6$ (Equal Steps, Equal Time, and FactorEngram Controls)

Table 1 reports all 3-seed full-horizon (953-, 1015-, and 1042-step) results at depth $L=8$ ($d=512$; exp_ngram_v8_d8full.json, exp_ngram_v9_d8full.json) and depth $L=6$ ($d=384$; exp_ngram_v8_d6confirm.json, exp_ngram_v9_d6full.json, exp_ngram_scale_and_horizon.json), alongside 240-step reduced-budget results (exp_ngram_v9_controls240.json, ngram_authoritative_summary.json). We report pure steady-state training time (Train s), full training-loop time including compilation and periodic validation curve checks (Loop s), and complete run time including post-training multi-split and low-bit PTQ suites (Wall s).

| Setting & Configuration (3 seeds: 41, 42, 43 unless noted) | Mod / Whole BF16 (MiB) | Train Sub / VRAM (MiB) | Step (ms) | Train / Loop / Wall (s) | Val BPB ($\pm$ SD) | Val CE / PPL | heldout_dedup BPB | fresh_rg42 BPB | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Depth $L=8$ ($d=512$), Full Budget (953 / 1015 steps, 50.33M backbone) | | | | | | | | | | d8_full953_plain_mb64 (mb=64, Seed 42 bridge) | 0.00 / 96.00 | 0.00 / 22804 | 1140.3 | 1075.3 / 1117.1 / 1256.8 | 0.994026 (n=1) | 2.78530 / 16.205 | 0.994763 | 0.994343 | | d8_full953_plain_mb32 (mb=32, 3 seeds) | 0.00 / 96.00 | 0.00 / 11701 | 1165.8 | 1099.4 / 1139.6 / 1245.5 | 0.993953 ± 0.000371 | 2.78509 / 16.201 | 0.994671 | 0.994301 | | d8_full953_oe_ember_lr02_mb32 (953 st, m=150,855) | 147.32 / 243.32 | 149.63 / 12003 | 1191.0 | 1123.1 / 1161.7 / 1346.0 | 0.976427 ± 0.000305 | 2.73598 / 15.425 | 0.977120 | 0.976871 | | d8_full1015_oe_ember_eqtime_mb32 (1015 st, +6.5% tok) | 147.32 / 243.32 | 149.63 / 12003 | 1195.2 | 1201.2 / 1234.7 / 1419.6 | 0.972708 ± 0.000269 | 2.72556 / 15.265 | 0.973377 | 0.973140 | | d8_full953_FE_basis_gate_isoparam_mb32 (FactorEngram [6] basis-gate, m=149,827) | 147.32 / 243.32 | 151.63 / 12391 | 1215.7 | 1146.4 / 1195.7 / 1446.7 | 0.973118 ± 0.000379 | 2.72671 / 15.283 | 0.973777 | 0.973505 | | d8_full953_NG6_isoparam_mb32 (L0-only CoDeM, m=149,315) | 147.32 / 243.32 | 153.13 / 12522 | 1247.2 | 1176.2 / 1220.0 / 1298.3 | 0.971853 ± 0.000350 | 2.72317 / 15.228 | 0.972492 | 0.972208 | | d8_full953_C2b_gate_det_isoparam_mb32 (Unigram-VE, gate_detach=True, m=147,267) | 147.32 / 243.32 | 159.10 / 12596 | 1275.4 | 1202.8 / 1249.2 / 1329.8 | 0.971773 ± 0.000228 | 2.72294 / 15.225 | 0.972490 | 0.972197 | | d8_full953_C2b_isoparam_mb32 (Unigram-VE, gate_detach=False, m=147,267) | 147.32 / 243.32 | 159.10 / 12534 | 1303.4 | 1229.1 / 1266.0 / 1962.3 | 0.970690 ± 0.000060 | 2.71990 / 15.179 | 0.971333 | 0.971044 | | d8_full953_NG7_isoparam_mb32 (CoDeM-Ngram, V8 / v9diag, m=147,267) | 147.32 / 243.32 | 159.10 / 12532 | 1275.3 / 1274.4 | 1202.6 / 1247.6 / 2022.9 | 0.970154 ± 0.000127 (0.970170) | 2.71840 / 15.156 | 0.970785 (0.970862) | 0.970493 (0.970565) | | $\quad\hookrightarrow$ 3-Bit NG7-IsoParam + Full L2 (31.59 MiB mod, V8 / v9diag) | 31.59 / 127.59 | — | — | — | 0.970910 / 0.970943 | 2.72052 / 15.188 | 0.971536 | 0.971241 | | $\quad\hookrightarrow$ 2-Bit NG7-IsoParam + Full Combined L2 (22.61 MiB mod, V8 / v9diag) | 22.61 / 118.61 | — | — | — | 0.973259 / 0.973287 | 2.72710 / 15.288 | 0.973894 | 0.973599 (0.973668) | | Depth $L=6$ ($d=384$), Full Budget (953 / 1015 / 1042 steps, 26.35M backbone) | | | | | | | | | | full_01_plain_d6 (953 st) | 0.00 / 50.25 | 0.00 / 13733 | 612.5 | 577.6 / — / — | 1.047419 ± 0.001895 | 2.93490 / 18.820 | 1.048257* | — | | full_05_oe_mfree_110mb (953 st, lr=0.60) | 110.49 / 160.74 | 110.49 / 13961 | 622.8 | 587.2 / — / — | 1.024745 ± 0.001494 | 2.87137 / 17.661 | 1.025559* | — | | full_12_eb1_sqrt_glob_110mb (953 st, EB1) | 110.49 / 160.74 | 110.49 / 13961 | 632.4 | 596.4 / — / — | 1.022183 ± 0.001519 | 2.86419 / 17.535 | 1.022870 | — | | full_10_oe_adam_lr02_110mb (953 st, OE-Adam) | 110.49 / 160.74 | 331.48 / 14182 | 635.5 | 599.2 / — / — | 1.020452 ± 0.000780 | 2.85934 / 17.450 | 1.021137 | — | | full_20_oe_ember_lr03_110mb (953 st, lr=0.30) | 110.49 / 160.74 | 112.80 / 13963 | 629.5 | 593.6 / — / — | 1.019672 ± 0.001731 | 2.85716 / 17.412 | 1.020367 | — | | d6_full953_oe_ember_lr02 (953 st, lr=0.20, m=150,855) | 110.49 / 160.74 | 112.80 / 13964 | 632.9 | 596.8 / 622.6 / 753.0 | 1.019890 ± 0.001025 | 2.85777 / 17.423 | 1.020597 | 1.020411 | | d6_full953_oe_ember_eqtime_1015 (1015 st, +6.5% tok) | 110.49 / 160.74 | 112.80 / 13964 | 627.7 | 630.8 / 652.4 / 757.0 | 1.016640 ± 0.001342 | 2.84866 / 17.265 | 1.017326 | 1.017155 | | d6_full1042_oe_ember_eqtime (1042 st, +9.3% tok, exceeds NG7 time) | 110.49 / 160.74 | 112.80 / 13963 | 642.7 | 663.3 / 689.7 / 874.9 | 1.015170 ± 0.001227 | 2.84454 / 17.194 | 1.015809 | 1.015691 | | d6_full953_FE_basis_gate_isoparam (FactorEngram [6] basis-gate, m=150,083) | 110.49 / 160.74 | 113.92 / 14543 | 657.1 | 619.6 / 646.9 / 930.1 | 1.015425 ± 0.001491 | 2.84525 / 17.206 | 1.016151 | 1.015929 | | d6_full953_NG6_isoparam (953 st, L0-only CoDeM, m=149,699) | 110.49 / 160.74 | 114.76 / 14736 | 671.7 | 633.4 / 659.7 / 949.1 | 1.013828 ± 0.000776 | 2.84078 / 17.129 | 1.014593 | 1.014372 | | d6_full953_C2b_gate_det_isoparam (Unigram-VE, gate_detach=True, m=148,547) | 110.49 / 160.74 | 117.28 / 14837 | 688.6 | 649.4 / 676.0 / 746.7 | 1.014304 ± 0.001950 | 2.84211 / 17.152 | 1.015056 | 1.014861 | | d6_full953_C2b_isoparam (Unigram-VE, gate_detach=False, m=148,547) | 110.49 / 160.74 | 117.28 / 14742 | 687.2 | 648.1 / 672.6 / 1202.2 | 1.012836 ± 0.001495 | 2.83800 / 17.082 | 1.013538 | 1.013383 | | d6_full953_NG7_isoparam (CoDeM-Ngram, V8 / v9diag, m=148,547) | 110.49 / 160.74 | 117.28 / 14741 | 686.5 / 687.6 | 647.4 / 672.1 / 1362.2 | 1.011987 ± 0.000879 (1.012214) | 2.83562 / 17.041 | 1.012689 (1.012904) | 1.012478 (1.012686) | | $\quad\hookrightarrow$ 2-Bit d6_full953_NG7 + Full Combined L2 (16.42 MiB mod, V8 / v9diag) | 16.42 / 66.68 | — | — | — | 1.016027 / 1.016243 | 2.84694 / 17.235 | 1.016737 | 1.016508 (1.016702) | | Reduced-Horizon Screening (240 steps = 125.8M tokens) | | | | | | | | | | d6_240: 43_oe_ember_lr02 vs. V9_240_FE_basis_gate (m=150,855 / 150,083) | 110.49 / 160.74 | 112.80 / 13963 | 626.3 / 646.0 | 144.1 / 162.7 / 310.6 | 1.132978 / 1.127364 | 3.17902 / 3.16326 | 1.138392 / 1.132844 | — / 1.131903 | | d6_240: V9_240_FE_basis_postconv vs. V9_240_NG6_isoparam (m=150,083 / 149,699) | 110.49 / 160.74 | 113.92 / 14736 | 647.1 / 664.3 | 148.8 / 165.0 / 267.4 | 1.127174 / 1.126285 | 3.16273 / 3.16023 | 1.132697 / 1.131821 | 1.131718 / 1.130663 | | d6_240: V9_240_NG7_isoparam (lr=0.20) / NG7_isoparam_lr03 (lr=0.30) | 110.49 / 160.74 | 117.28 / 14741 | 681.1 / 680.9 | 156.7 / 165.3 / 287.0 | 1.125059 / 1.124006 | 3.15679 / 3.14950 | 1.130673 / 1.129285* | 1.129614 / — | | d8_240: d8_10_oe_ember_lr02 vs. d8_NG7_isoparam_110mb (lr=0.20) | 147.32 / 243.32 | 149.63 / 12531 | 1203.1 / 1265.8 | 276.7 / — / 457.0 | 1.098381 / 1.091448 | 3.08194 / 3.05827 | 1.103846 / 1.096677* | — |

*Entries marked * come from earlier suites (exp_ngram_suite1_2.json / exp_ngram_nextgen.json) that logged heldout_bpb prior to adding heldout_dedup_bpb (the matched V9 rerun V9_240_NG7_isoparam logs 1.130673 heldout_dedup_bpb and 1.129614 fresh_rg42_dedup_bpb). Parenthesized numbers in the NG7 rows show the independent 3-seed V9 diagnostic replication (d6_full953_NG7_isoparam_v9diag with 648.4 / 673.7 / 956.4 s timing, and d8_full953_NG7_isoparam_mb32_v9diag with 1201.7 / 1239.9 / 1648.2 s timing).

Key Findings from Table 1:

  1. Full-Horizon $L=8$ (953 Steps): d8_full953_NG7_isoparam_mb32 reduces validation BPB by $-23.799\text{ mBPB}$ vs. plain (95% CI $[-24.64, -22.96]\text{ mBPB}$) and by $-6.273\text{ mBPB}$ vs. OE-Ember (CI $[-7.33, -5.22]\text{ mBPB}$; v9diag replication: $-6.257\text{ mBPB}$, CI $[-7.25, -5.27]\text{ mBPB}$). On the untouched confirmation split fresh_rg42_dedup, the paired gain over OE-Ember is $-6.378\text{ mBPB}$ (CI $[-7.58, -5.18]\text{ mBPB}$; v9diag: $-6.306\text{ mBPB}$, CI $[-7.51, -5.10]\text{ mBPB}$).
  2. Strict Equal-Time Comparisons (1015-Step $L=8$ and 1042-Step $L=6$ OE-Ember vs. 953-Step CoDeM-Ngram): At $L=8$, CoDeM-Ngram requires 1202.6 s (v9diag: 1201.7 s) steady training time (1247.6 s / 1239.9 s loop time); training OE-Ember for 1015 steps (+6.5% tokens = $532.15\text{M}$ tokens; 1201.2 s train / 1234.7 s loop) equalizes training time within $0.1\%$, yet 953-step CoDeM-Ngram wins all 3 seeds by $-2.554\text{ mBPB}$ (v9diag: $-2.538\text{ mBPB}$, CI $[-3.48, -1.60]\text{ mBPB}$; $-2.647\text{ mBPB}$ on fresh_rg42_dedup). At $L=6$, CoDeM-Ngram requires 647.4 s (v9diag: 648.4 s) steady training time (672.1 s / 673.7 s loop time); training OE-Ember for 1015 steps (630.8 s train / 652.4 s loop) leaves a $2.6\%$ runtime gap, so we additionally run d6_full1042_oe_ember_eqtime for 1042 steps (+9.3% tokens = $546.31\text{M}$ tokens; 663.3 s train / 689.7 s loop), which strictly exceeds CoDeM-Ngram on both pure training time ($+14.9\text{ to }+15.9\text{ s}$) and full loop time ($+16.0\text{ to }+17.6\text{ s}$). 953-step CoDeM-Ngram beats 1015-step OE-Ember by $-4.653\text{ mBPB}$ and beats 1042-step OE-Ember on all 3 seeds by $-3.183\text{ mBPB}$ (v9diag: $-2.957\text{ mBPB}$, CI $[-3.16, -2.75]\text{ mBPB}$; $-3.214\text{ mBPB}$ on fresh_rg42_dedup).
  3. Direct Parameter-Matched Comparison Against FactorEngram [6] Basis-Level Gating (FE_basis_gate_isoparam and FE_basis_postconv_isoparam): At 240 steps ($L=6$), single-token basis-gated FE_basis_gate_isoparam (1.127364) and FE_basis_postconv_isoparam (1.127174) improve over static $W_{\text{proj}}$ (P2, 1.129118), but replacing their single-token gate with our channel-rolled 3-token unigram summary $c_t$ and dual pre/post gates (V9_240_NG6_isoparam, 1.126285) yields a further $-1.079\text{ mBPB}$ (CI $[-1.70, -0.45]$; $-1.240\text{ mBPB}$ on fresh_rg42_dedup), and adding Phrase-VE (V9_240_NG7_isoparam, 1.125059) widens the lead to $-2.304\text{ mBPB}$ (-2.290 mBPB on fresh_rg42_dedup). At full 953-step horizon, NG6-IsoParam beats FE_basis_gate_isoparam by $-1.597\text{ mBPB}$ at $L=6$ (1.013828 vs. 1.015425) and $-1.265\text{ mBPB}$ at $L=8$ (0.971853 vs. 0.973118, CI $[-1.68, -0.85]$), while full CoDeM-Ngram (NG7-IsoParam) beats FE_basis_gate_isoparam by $-3.438\text{ mBPB}$ (v9diag: $-3.211\text{ mBPB}$, CI $[-4.64, -1.78]$) at $L=6$ and $-2.964\text{ mBPB}$ (v9diag: $-2.949\text{ mBPB}$, CI $[-4.24, -1.65]$) at $L=8$.

4.2 Testing the Collision-Demodulation Hypothesis: Intervention & Common-Modulus Purity Stratification

Over the first 240 training steps ($125.8\text{M}$ tokens), ClimbMix-400B presents $38,055,995$ distinct training trigrams mapped into $m_2 = 150,859$ rows in slice $s=0$, so $100\%$ of hash buckets receive $\ge 4$ distinct trigrams (mean load factor $252.3$ distinct trigrams/row at $m=150,855$; $256.2$ trigrams/row at $m=148,547$). While replacing the unigram-context gate (DIAG_NG4_bcd_l0, 1.126368 val BPB) with a self-gate driven by the hashed vector itself (DIAG_C1_self_gated, 1.130042 val BPB) degrades validation loss by $+3.674\text{ mBPB}$, we test whether contextual gating specifically mitigates bucket collisions using two controlled analyses (exp_ngram_v8_controls.json, exp_ngram_v9_d6full.json, exp_ngram_v9_d8full.json), summarized in Table 2:

  1. Collision-Rate Intervention ($m=150,855 \to 32,767$, $4.6\times$ Higher Collision Density): At $L=6, 240$ steps across 3 seeds (41, 42, 43), we compare static P2 ($W_{\text{proj}}$ only), self-gated C1 ($g_t^{\text{pre}} = 2\sigma(W_{\text{ctx}}\operatorname{RMSNorm}(e_{\text{raw},t}))$), and unigram-context-gated NG4 ($g_t^{\text{pre}} = 2\sigma(W_{\text{ctx}}c_t)$) at standard modulus $m=150,855$ (252.3 distinct trigrams/row) vs. high-collision modulus $m=32,767$ (1161.3 distinct trigrams/row). Increasing collision load by $4.6\times$ increases NG4's paired improvement over P2 by $+73.1\%$, from $-2.750\text{ mBPB}$ (1.126368 vs. 1.129118, CI $[-3.62, -1.88]\text{ mBPB}$) to $-4.760\text{ mBPB}$ (1.142223 vs. 1.146983, CI $[-5.88, -3.64]\text{ mBPB}$), whereas self-gated C1 provides zero benefit over P2 at $m=32,767$ (1.146994 vs. 1.146983, $+0.011\text{ mBPB}$). Because shrinking $m$ also weakens the base N-gram predictor (P2 rises by $+17.87\text{ mBPB}$), this intervention is consistent with context-driven collision mitigation alongside a larger fallback contribution from the unigram-context pathway when table fidelity drops.
  2. Common-Modulus Purity-Stratified Validation Loss (m_ref = 150,855, m2_ref = 150,859): To ensure that models with slightly different parameter-matched table moduli ($m \in \{150,855, 150,083, 149,827, 148,547, 147,267\}$) are evaluated on the exact same validation token subsets, build_trigram_collision_multiplicity (run_ngram_v9_fixes.py) computes bucket purity $\pi(h_{\text{ref}}) = c_{\max}(h_{\text{ref}}) / C_{\text{tot}}(h_{\text{ref}}) \in (0, 1]$ on the fixed reference modulus $m_{2,\text{ref}} = 150,859$ over the first 240 training steps ($125.8\text{M}$ tokens), partitioning the $20.97\text{M}$ validation tokens (20,944,857 valid trigram positions) into identical high-purity (2,362,103 tokens, $11.28\%$, $\pi \ge 0.50$), mid-purity (9,701,579 tokens, $46.32\%$, $0.15 \le \pi < 0.50$), and low-purity (8,881,175 tokens, $42.40\%$, $\pi < 0.15$) bins across all models. As shown in Table 2B, CoDeM-Ngram's paired BPB reduction over OE-Ember increases monotonically from high-purity to low-purity (heavily collided) buckets at both $L=6$ ($-5.917 \to -7.674 \to -8.190\text{ mBPB}$) and $L=8$ ($-5.002 \to -6.221 \to -6.664\text{ mBPB}$), and exhibits the same monotonic gradient against FE_basis_gate_isoparam ($-2.746 \to -3.201 \to -3.359\text{ mBPB}$ at $L=6$; $-2.614 \to -2.930 \to -3.066\text{ mBPB}$ at $L=8$).

| Table 2A: Collision-Rate Intervention ($L=6, 240$ steps, 3 seeds) | Modulus $m$ | Trigrams / Row | P2 ($W_{\text{proj}}$ only) | C1 (Self-Gated) | NG4 (Unigram Gate) | NG4 - P2 ($\Delta$ mBPB, 95% CI) | NG4 - C1 ($\Delta$ mBPB) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Standard Table (DIAG_*, 110.5 MiB tables) | 150,855 | 252.3 | 1.129118 ± 0.006266 | 1.130042 ± 0.005477 | 1.126368 ± 0.005936 | -2.750 ([-3.62, -1.88]) | -3.674 | | High-Collision Intervention (COLL32K_*, 24.0 MiB tables) | 32,767 | 1161.3 ($4.6\times$) | 1.146983 ± 0.008500 | 1.146994 ± 0.007903 | 1.142223 ± 0.007813 | -4.760 ([-5.88, -3.64], $+73\%$) | -4.771 |

| Table 2B: Common-Modulus (m2_ref = 150,859) Purity-Stratified Val BPB (953 st, 3 seeds) | High Purity $\pi(h_{\text{ref}}) \ge 0.50$ (2,362,103 tok, 11.3%) | Mid Purity $0.15 \le \pi(h_{\text{ref}}) < 0.50$ (9,701,579 tok, 46.3%) | Low Purity / High Collision $\pi(h_{\text{ref}}) < 0.15$ (8,881,175 tok, 42.4%) | | :--- | :---: | :---: | :---: | | $L=6$: d6_full953_oe_ember_lr02 $\to$ d6_full953_NG7_isoparam_v9diag | 0.991899 $\to$ 0.985981 (-5.917 ± 0.429, CI [-6.98, -4.85]) | 1.015680 $\to$ 1.008005 (-7.674 ± 0.341, CI [-8.52, -6.83]) | 1.032737 $\to$ 1.024547 (-8.190 ± 0.162, CI [-8.59, -7.79]) | | $L=6$: d6_full953_FE_basis_gate_isoparam $\to$ d6_full953_NG7_isoparam_v9diag | 0.988727 $\to$ 0.985981 (-2.746 ± 0.511, CI [-4.02, -1.48]) | 1.011206 $\to$ 1.008005 (-3.201 ± 0.552, CI [-4.57, -1.83]) | 1.027906 $\to$ 1.024547 (-3.359 ± 0.631, CI [-4.93, -1.79]) | | $L=8$: d8_full953_oe_ember_lr02_mb32 $\to$ d8_full953_NG7_isoparam_mb32_v9diag | 0.953585 $\to$ 0.948583 (-5.002 ± 0.759, CI [-6.89, -3.12]) | 0.972523 $\to$ 0.966302 (-6.221 ± 0.349, CI [-7.09, -5.35]) | 0.987433 $\to$ 0.980769 (-6.664 ± 0.361, CI [-7.56, -5.77]) | | $L=8$: d8_full953_FE_basis_gate_isoparam_mb32 $\to$ d8_full953_NG7_isoparam_mb32_v9diag | 0.951197 $\to$ 0.948583 (-2.614 ± 0.655, CI [-4.24, -0.99]) | 0.969233 $\to$ 0.966302 (-2.930 ± 0.497, CI [-4.17, -1.69]) | 0.983835 $\to$ 0.980769 (-3.066 ± 0.511, CI [-4.34, -1.80]) |


4.3 $2\times 2$ Phrase-VE Factorial, Gate-Detachment Control, and Horizon Trajectory

When comparing Phrase-VE (NG7, which routes the demodulated phrase vector $\tilde{u}_t$ into $W_{\text{pve}}^{(l)}$) against Unigram-VE (which routes the 3-token unigram summary $c_t^{\text{raw}}$ into $W_{\text{pve}}^{(l)}$), four factors must be controlled: (i) parameter parity ($m=148,547$ at $L=6$; $m=147,267$ at $L=8$), (ii) $W_{\text{pve}}^{(l)}$ initialization ($0$ vs. $0.25I_d$), (iii) VE gradient flow (pve_detach=True vs. False), and (iv) Layer-0 context-gate stop-gradient (gate_detach_wte=True in Eq. 8). In Version 8 (V8NextGenGPT), DIAG_C2b_isoparam and d6/d8_full953_C2b_isoparam set pve_detach=False on $c_t^{\text{raw}}$ before Eq. (8), which simultaneously allowed Layer-0 context-gate gradients to update wte, whereas NG7 detached wte at the Layer-0 gate. In Version 9 (V9NextGenGPT), V9_240_C2d_isoparam, V9_240_C2b_gate_det_isoparam, d6_full953_C2b_gate_det_isoparam, and d8_full953_C2b_gate_det_isoparam_mb32 hold gate_detach_wte=True strictly fixed across all arms while varying only the VE pathway (Table 3A and Table 3B).

| Table 3A: $L=6, 240$-Step VE Factorial (3 seeds: 41, 42, 43) | Modulus $m$ | VE Source | $W_{\text{pve}}^{(l)}$ Init / VE Grad | L0 Gate Detach | Val BPB ($\pm$ SD) | heldout_dedup BPB | fresh_rg42 BPB | $\Delta$ Val vs. V9_240_NG6 (1.126285) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | V9_240_NG6_isoparam (No VE pathway) | 149,699 | None | — / — | True | 1.126285 ± 0.006025 | 1.131821 | 1.130663 | 0.000 mBPB | | C2c_phrase_ve_zero_detach | 150,855 | Phrase $\tilde{u}_t$ | $0$ / Detached | True | 1.127079 ± 0.006241 | 1.132541 | — | +0.794 mBPB | | C2_conv_bcd_l0_unigram_ve | 150,855 | Unigram $c_t^{\text{raw}}$ | $0$ / Detached | True | 1.125866 ± 0.005756 | 1.131053* | — | -0.419 mBPB | | V9_240_C2d_isoparam (Isoparam detached Unigram-VE) | 148,547 | Unigram $c_t^{\text{raw}}$ | $0.25\,I_d$ / Detached | True | 1.126168 ± 0.005949 | 1.131637 | 1.130715 | -0.117 mBPB | | V9_240_C2b_gate_det_isoparam (Strictly matched Unigram-VE) | 148,547 | Unigram $c_t^{\text{raw}}$ | $0.25\,I_d$ / Joint | True | 1.125843 ± 0.005459 | 1.131290 | 1.130385 | -0.442 mBPB | | V9_240_NG7_isoparam (CoDeM-Ngram, Phrase-VE) | 148,547 | Phrase $\tilde{u}_t$ | $0.25\,I_d$ / Joint | True | 1.125059 ± 0.006607 | 1.130673 | 1.129614 | -1.226 mBPB (-0.784 vs. C2b_gate_det) | | DIAG_C2b_isoparam (Unigram-VE + attached L0 gate) | 148,547 | Unigram $c_t^{\text{raw}}$ | $0.25\,I_d$ / Joint | False | 1.123921 ± 0.005018 | 1.129497 | — | -2.364 mBPB |

| Table 3B: Full-Horizon val_curve Trajectory (3 seeds, $m$ matched) | Step 190 (99.6M tok) | Step 380 (199.2M tok) | Step 570 (298.8M tok) | Step 760 (398.5M tok) | Final Step 953 (val $20.97\text{M}$) | Final fresh_rg42_dedup ($20.97\text{M}$) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | | $L=6$ (gate_detach=True fixed): d6_full953_C2b_gate_det_isoparam | 1.178244 | 1.103965 | 1.063556 | 1.027352 | 1.014304 ± 0.001950 | 1.014861 ± 0.001969 | | $L=6$ (gate_detach=True fixed): d6_full953_NG7_isoparam_v9diag | 1.176791 | 1.103043 | 1.061030 | 1.025077 | 1.012214 ± 0.001221 | 1.012686 ± 0.001221 | | $\quad\hookrightarrow$ Paired Diff NG7_v9diag - C2b_gate_det ($L=6$, mBPB) | -1.453 | -0.922 (3/3 seeds) | -2.526 (3/3 seeds) | -2.275 (3/3 seeds) | -2.090 mBPB (CI [-3.93, -0.25]) | -2.176 mBPB (CI [-4.05, -0.30]) | | $L=6$ (gate_detach=False in C2b): NG7_isoparam - C2b_isoparam | +0.984 (C2b leads) | +0.108 (tie) | -1.145 (NG7 overtakes) | -0.873 | -0.849 mBPB (3/3 seeds) | -0.904 mBPB (3/3 seeds) | | $L=8$ (gate_detach=True fixed): d8_full953_C2b_gate_det_isoparam_mb32 | 1.145208 | 1.070834 | 1.025992 | 0.986818 | 0.971773 ± 0.000228 | 0.972197 ± 0.000237 | | $L=8$ (gate_detach=True fixed): d8_full953_NG7_isoparam_mb32_v9diag | 1.145423 | 1.069593 | 1.024669 | 0.985096 | 0.970170 ± 0.000163 | 0.970565 ± 0.000161 | | $\quad\hookrightarrow$ Paired Diff NG7_v9diag - C2b_gate_det ($L=8$, mBPB) | +0.215 (tie) | -1.241 (3/3 seeds) | -1.323 (3/3 seeds) | -1.722 (3/3 seeds) | -1.603 mBPB (CI [-2.10, -1.10]) | -1.631 mBPB (CI [-2.11, -1.15]) | | $L=8$ (gate_detach=False in C2b): NG7_isoparam - C2b_isoparam_mb32 | +0.995 (C2b leads) | +0.832 (C2b leads) | -0.398 (NG7 overtakes) | -0.381 | -0.536 mBPB (CI [-0.71, -0.37]) | -0.551 mBPB (CI [-0.72, -0.38]) |

Key Findings from Table 3A and Table 3B (Figure 1c):

  1. Strictly Matched Phrase-VE vs. Unigram-VE (gate_detach_wte=True Held Fixed): When gate_detach_wte=True is held constant across all arms, routing the demodulated N-gram phrase vector $\tilde{u}_t$ into VE (NG7) outperforms routing the 3-token unigram summary (C2b_gate_det) at 240 steps by $-0.784\text{ mBPB}$ (1.125059 vs. 1.125843; $-0.771\text{ mBPB}$ on fresh_rg42_dedup), and wins every seed at full 953-step horizon by $-2.090\text{ mBPB}$ at $L=6$ (1.012214 vs. 1.014304, CI $[-3.93, -0.25]\text{ mBPB}$; $-2.316\text{ mBPB}$ vs. V8 NG7) and $-1.603\text{ mBPB}$ at $L=8$ (0.970170 vs. 0.971773, CI $[-2.10, -1.10]\text{ mBPB}$; $-1.619\text{ mBPB}$ vs. V8 NG7, CI $[-1.90, -1.34]\text{ mBPB}$).
  2. Why C2b (gate_detach_wte=False) Leads Early (190–240 Steps) and NG7 Overtakes by Step 570: Each training token updates $1$ row of the $8,192$-row unigram table wte and $1$ row in each of the $4$ N-gram slice tables ($m \approx 148,547$ rows per slice, $594,200$ total N-gram rows across the 4 slices). Thus, each individual N-gram slice table has $m / V \approx 18.1\times$ lower per-row update frequency than wte (and total N-gram rows outnumber wte rows by $4m / V \approx 72.5\times$). When C2b additionally allows Layer-0 gate gradients to flow into wte (gate_detach_wte=False), the high-density unigram embeddings adapt rapidly during the first 190–240 steps (1.123921 vs. 1.125059). However, once N-gram hash buckets accumulate sufficient token exposure (380–570 steps = $199\text{M}\text{--}299\text{M}$ tokens), Phrase-VE (NG7) overtakes even C2b, finishing ahead on all 3 seeds at step 953 by $-0.849\text{ mBPB}$ at $L=6$ ($-0.904\text{ mBPB}$ on fresh_rg42_dedup) and $-0.536\text{ mBPB}$ at $L=8$ (CI $[-0.71, -0.37]\text{ mBPB}$).
  3. Two-Pass Gradient Sensitivity Diagnostic (run_cross_layer_grad_diag): In run_ngram_nextgen.py, the cosine similarity between the Layer-0 table gradient $g_{\text{L0}}$ (standard forward pass) and the VE-branch gradient $g_{\text{VE}}$ obtained in a second forward pass with an unprojected identity VE skip is $\cos(g_{\text{L0}}, g_{\text{VE}}) = -0.0018$ at step 0 and $-0.0091$ at step 120. Because the second pass alters the forward graph, near-zero cosine indicates that unadapted direct injection perturbs deeper attention values in a direction uncorrelated with Layer-0 gradient descent; the training ablation P2 -> NG1 -> NG2 (1.129370 -> 1.128145 -> 1.127625 BPB on Seed 42) vs. unadapted direct VE sharing (B2_superpose, 1.132830 BPB) directly confirms the benefit of learned projections $W_{\text{pve}}^{(l)}$.

4.4 Low-Bit Quantization and Multi-Readout Ridge-Regularized L2 Calibration

Table 4 reports the complete post-training quantization (PTQ) and ridge-regularized L2 calibration decomposition from ngram_authoritative_summary.json, separating both structural constraint spaces (Diag, Block-Diag, Full) and readout pathway targets (Full L0-Only folding $(M^*)^\top$ into $W_{\text{proj}}$ only, Full VE-Only folding $(M^*)^\top$ into $\{W_{\text{pve}}^{(l)}\}$ only, and Full Combined folding $(M^*)^\top$ into both pathways simultaneously), alongside full-precision feature cross-slice correlation $R^2(\tilde{e}^{(0)}, \tilde{e}^{(1)})$ (from $\Sigma_{ee}$) and quantization-error cross-slice correlation $R^2(d^{(0)}, d^{(1)})$ (from $\Sigma_{dd}$, where $d_t = \tilde{q}_t - \tilde{e}_t$).

| Table 4: Post-Training Quantization & Multi-Readout L2 Calibration (3 seeds) | BF16 Val BPB | 3-Bit Std / Full Val | 2-Bit Std Val | 2-Bit Diag ($\Delta$ mBPB / L0,VE MSE $\downarrow$) | 2-Bit Block-Diag ($\Delta$ mBPB / L0,VE MSE $\downarrow$) | 2-Bit Full L0-Only ($\Delta$ mBPB / L0 MSE $\downarrow$) | 2-Bit Full VE-Only ($\Delta$ mBPB / VE MSE $\downarrow$) | 2-Bit Full Combined ($\Delta$ mBPB / L0,VE MSE $\downarrow$) | Feature $R^2_{\tilde{e}}$ / Error $R^2_d$ ($s_0,s_1$) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | $L=6, 240$ st: DIAG_P2_learned_proj (L0 only) | 1.129118 | 1.129950 / 1.129930 | 1.132737 | 1.132727 (-0.010 / 0.08%, —) | 1.132618 (-0.120 / 2.16%, —) | 1.132475 (-0.262 / 3.00%) | — | 1.132475 (-0.262 / 3.00%, —) | 0.07118 / — | | $L=6, 240$ st: V9_240_FE_basis_gate (L0 only) | 1.127364 | 1.128376 / 1.128289 | 1.131897 | 1.131542 (-0.354 / 1.44%, —) | 1.131371 (-0.526 / 3.53%, —) | 1.131150 (-0.747 / 4.62%) | — | 1.131150 (-0.747 / 4.62%, —) | 0.08517 / 0.01179 | | $L=6, 240$ st: V9_240_NG6_isoparam (L0 only) | 1.126285 | 1.127303 / 1.127218 | 1.130752 | 1.130442 (-0.310 / 1.66%, —) | 1.130270 (-0.482 / 3.76%, —) | 1.130058 (-0.694 / 4.91%) | — | 1.130058 (-0.694 / 4.91%, —) | 0.09413 / 0.01237 | | $L=6, 240$ st: V9_240_NG7_isoparam (L0 + VE) | 1.125059 | 1.126128 / 1.126032 | 1.129928 | 1.129535 (-0.393 / 1.44%, 1.07%) | 1.129321 (-0.607 / 3.63%, 2.97%) | 1.129189 (-0.739 / 4.94%) | 1.129770 (-0.158 / 4.32%) | 1.129078 (-0.851 / 4.94%, 4.32%) | 0.08555 / 0.01390 | | $L=6, 953$ st: d6_full953_C2b_isoparam (L0 only) | 1.012836 | 1.013800 / 1.013742 | 1.017178 | 1.016952 (-0.227 / 1.62%, —) | 1.016714 (-0.464 / 3.42%, —) | 1.016526 (-0.652 / 4.69%) | — | 1.016526 (-0.652 / 4.69%, —) | 0.08119 / — | | $L=6, 953$ st: d6_full953_NG7_isoparam (V8) | 1.011987 | 1.013015 / 1.012954 | 1.016880 | 1.016554 (-0.326 / 1.78%, —) | 1.016281 (-0.599 / 3.81%, —) | — | — | 1.016027 (-0.853 / 5.35%, —) | 0.07265 / — | | $L=6, 953$ st: d6_full953_NG7_v9diag (L0 + VE) | 1.012214 | 1.013236 / 1.013173 | 1.017116 | 1.016772 (-0.345 / 1.82%, 1.10%) | 1.016495 (-0.622 / 3.84%, 2.78%) | 1.016432 (-0.685 / 5.32%) | 1.016864 (-0.253 / 4.23%) | 1.016243 (-0.874 / 5.32%, 4.23%) | 0.07279 / 0.01569 | | $L=8, 953$ st: d8_full953_C2b_isoparam_mb32 (L0 only) | 0.970690 | 0.971403 / 0.971362 | 0.973990 | 0.973791 (-0.199 / 2.04%, —) | 0.973562 (-0.428 / 4.04%, —) | 0.973408 (-0.582 / 5.35%) | — | 0.973408 (-0.582 / 5.35%, —) | 0.08977 / — | | $L=8, 953$ st: d8_full953_NG7_isoparam_mb32 (V8) | 0.970154 | 0.970971 / 0.970910 | 0.974179 | 0.973814 (-0.365 / 2.05%, —) | 0.973497 (-0.682 / 4.32%, —) | — | — | 0.973259 (-0.920 / 5.99%, —) | 0.08166 / — | | $L=8, 953$ st: d8_full953_NG7_mb32_v9diag (L0 + VE) | 0.970170 | 0.970990 / 0.970928 | 0.974168 | 0.973832 (-0.335 / 2.10%, 1.22%) | 0.973525 (-0.643 / 4.37%, 3.18%) | 0.973505 (-0.663 / 6.04%) | 0.973877 (-0.291 / 4.83%) | 0.973287 (-0.880 / 6.04%, 4.83%) | 0.08143 / 0.01938 |

Key Findings from Table 4 (Figure 1d):

  1. Readout Pathway Isolation (Full L0-Only vs. Full VE-Only vs. Full Combined): On full-horizon $L=6$ d6_full953_NG7_isoparam_v9diag (2-bit Std = 1.017116), folding $(M_{\text{full}}^*)^\top$ into $W_{\text{proj}}$ alone (Full L0-Only) lowers validation BPB by $-0.685 \pm 0.087\text{ mBPB}$ ($78.4\%$ of the combined gain; Layer-0 readout MSE $-5.32\%$, VE MSE $0\%$), folding $(M_{\text{full}}^*)^\top$ into $\{W_{\text{pve}}^{(l)}\}$ alone (Full VE-Only) lowers validation BPB by $-0.253 \pm 0.035\text{ mBPB}$ (Layer-0 MSE $0\%$, VE readout MSE $-4.23\%$), and folding into both readouts (Full Combined) achieves $-0.874 \pm 0.062\text{ mBPB}$ (95% CI $[-1.03, -0.72]\text{ mBPB}$, nearly additive vs. $-0.685 - 0.253 = -0.938\text{ mBPB}$). On full-horizon $L=8$ d8_full953_NG7_isoparam_mb32_v9diag (2-bit Std = 0.974168), Full L0-Only yields $-0.663 \pm 0.101\text{ mBPB}$ ($75.3\%$ of combined gain; Layer-0 MSE $-6.04\%$), Full VE-Only yields $-0.291 \pm 0.008\text{ mBPB}$ (VE MSE $-4.83\%$), and Full Combined yields $-0.880 \pm 0.105\text{ mBPB}$ (CI $[-1.14, -0.62]\text{ mBPB}$).
  2. Structural Decomposition (Diag vs. Block-Diag vs. Full) and Feature vs. Error Cross-Slice $R^2$: Across d8_full953_NG7_isoparam_mb32_v9diag, per-channel Diag rescaling accounts for $-0.335\text{ mBPB}$ ($38.1\%$ of total gain), within-slice Block-Diag calibration adds $-0.308\text{ mBPB}$ ($35.0\%$), and cross-slice Full Combined calibration adds $-0.237\text{ mBPB}$ ($26.9\%$). Why does off-diagonal cross-slice calibration help when table quantization is independent per row? First, the shared unigram context gate $g_t^{\text{pre}}$ and local convolution correlate the full-precision pre-projection slice features ($R^2(\tilde{e}^{(0)}, \tilde{e}^{(1)}) = 0.07279$ at $L=6$ and $0.08143$ at $L=8$; between bigram and trigram slices $R^2(\tilde{e}^{(0)}, \tilde{e}^{(2)}) \approx 0.011$), allowing a linear combination of neighbor slices to acts as a shrinkage estimator for the target slice $\tilde{e}_t$. Second, because two hash slices of the same N-gram order index the same token sequence with moduli $m$ and $m+2$, their pre-projection quantization errors $d_t = \tilde{q}_t - \tilde{e}_t$ also exhibit non-zero within-order correlation ($R^2(d^{(0)}, d^{(1)}) = 0.01569$ at $L=6$, range $[0.0112, 0.0214]$; $0.01938$ at $L=8$, range $[0.0165, 0.0236]$).
  3. Comparison Against Quantization-Aware Training (QAT) and Uncompressed 16-Bit Baselines: Straight-through 2-bit QAT baselines (full_06_lowbit_oe_2bit_150k 1.024996 and full_08_core_qste_2bit_110mb 1.026211) require $331.5\text{ MiB}$ of FP32 master-weight training state yet trail 2-bit CoDeM-Ngram + Full Combined L2 (1.016027 / 1.016243 at $L=6$) by $-8.75\text{ to }-8.97\text{ mBPB}$. Furthermore, simulated 2-bit CoDeM-Ngram + Full Combined L2 ($16.42\text{ MiB}$ module / $66.68\text{ MiB}$ whole model at $L=6$; $22.61\text{ MiB}$ module / $118.61\text{ MiB}$ whole model at $L=8$) outperforms uncompressed 16-bit OE-Ember by $-3.647\text{ to }-3.863\text{ mBPB}$ at $L=6$ and $-3.139\text{ to }-3.168\text{ mBPB}$ at $L=8$ (CI $[-4.33, -1.95]\text{ mBPB}$).

5. Conclusion and Limitations

We have shown that replacing static narrow-slice N-gram table injection with Context-Demodulated Multi-Pathway Phrase Memory (CoDeM-Ngram / NG7-IsoParam) and Multi-Readout Ridge-Regularized L2 Calibration yields consistent, parameter-matched improvements across both $L=6$ ($d=384$) and full-budget $L=8$ ($d=512$) Transformers on ClimbMix-400B, outperforming OE-Ember at equal steps and equal training time, parameter-matched FactorEngram [6] basis-level gating (FE_basis_gate_isoparam), and strictly gate-detachment-matched Unigram-VE (C2b_gate_det_isoparam). Three limitations bound our claims: (1) our collision-rate intervention ($m=150,855 \to 32,767$) and common-modulus purity stratification ($m_{2,\text{ref}}=150,859$) support context-driven collision mitigation alongside unigram-context fallback when table fidelity drops, but do not imply that trained hash rows are exact linear superpositions of isolated phrase embeddings; (2) ridge-regularized L2 readout folding minimizes pre-projection reconstruction error rather than exact token cross-entropy; and (3) low-bit memory figures reflect exact dequantized Lloyd-Max simulation (quantize_tables_lloyd_max) rather than custom bit-packed integer CUDA kernels.


References

  1. H. Huang et al., Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling, arXiv:2501.16975, 2025.
  2. D. Yu et al., Scaling Embedding Layers in Language Models (SCONE), arXiv:2502.01637, 2025.
  3. Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (Engram), arXiv:2601.07372, 2026.
  4. TN-gram: Tensorized N-gram Embeddings for Language Modeling, arXiv:2606.08347, 2026.
  5. K. Shah, Ember: A Kilobyte-State Optimizer for Token Interfaces, arXiv:2607.01455, 2026.
  6. B. Yang et al., FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models, arXiv:2609.35578, September 2026.
  7. S. Chung and W. Du, Low-Bit N-Gram Embeddings for Language Models, arXiv:2610.31646, 2026.
  8. N. Shazeer and M. Stern, Adafactor: Adaptive Learning Rates with Sublinear Memory Cost, ICML / arXiv:1804.04235, 2018.
  9. J. Li et al., Frequency-aware SGD for Efficient Embedding Learning, arXiv:2110.04844, 2021.
  10. Rows, bits, and exposure: where the capacity of hashed n-gram input tables goes, and what it costs in bytes, rStation general_pretraining_6/v4, 2026.
  11. Hashed N-gram Table Rows Need Little Optimizer History, and the Tested Forward or Averaged Writes Recover Only a Fifth of a Gradient Write, rStation general_pretraining_11/v3, 2026.
  12. Separating initialization, coefficient learning, and lookup cost in shared value embeddings, rStation general_pretraining_9/v1, 2026.
  13. Cross-Family Composition and Checkpoint Diagnostics of Moment-Free Hashed N-Gram Input Tables Under Pre-Polar Equilibrated Muon Pretraining, rStation general_pretraining_16/v5, 2026.

Appendix A: Additional Architectural Progression & Literature Baselines ($L=6$, 240 & 953 Steps)

| Arm ID & Description (ngram_authoritative_summary.json) | Steps | Seeds | Mod / Whole BF16 (MiB) | Val BPB (240 / 953) | heldout_dedup (240 / 953) | 3-Bit Std Val (240 / 953) | 2-Bit Std Val (240 / 953) | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | 01_plain_d6 / full_01_plain_d6 (No N-gram table) | 240 / 953 | 3 / 3 | 0.00 / 50.25 | 1.166445 / 1.047419 | — / 1.048257* | — | — | | 02_oe_mfree_110mb / full_05_oe_mfree_110mb (OE-MFree) | 240 / 953 | 3 / 3 | 110.49 / 160.74 | 1.140305 / 1.024745 | — / 1.025559* | — | — | | 04_lowbit_oe_2bit_150k / full_06_lowbit_oe_2bit (2-bit QAT) | 240 / 953 | 3 / 3 | 14.96 / 65.21 | 1.138417 / 1.024996 | — / 1.025802* | — | 1.138417 / 1.024996 | | L3_tngram_faithful_r640 / full_17_tngram_r640 (TN-gram [4]) | 240 / 953 | 3 / 3 | 107.52 / 157.78 | 1.142076 / 1.023739 | 1.147590 / 1.024400 | — | — | | L4_engram_faithful_110mb / full_16_engram_110mb (Engram [3]) | 240 / 953 | 3 / 3 | 111.62 / 161.87 | 1.141655 / 1.023127 | 1.147223 / 1.023835 | — | — | | 44_eb1_sqrt_glob / full_12_eb1_sqrt_glob (EB1, lr=0.10) | 240 / 953 | 3 / 3 | 110.49 / 160.74 | 1.135550 / 1.022183 | 1.141089 / 1.022870 | 1.136359 / 1.023004 | 1.139053 / 1.025861 | | 41_oe_adam_lr02 / full_10_oe_adam_lr02 (OE-Adam, lr=0.20) | 240 / 953 | 3 / 3 | 110.49 / 160.74 | 1.134879 / 1.020452 | 1.140367 / 1.021137 | 1.135614 / 1.021179 | 1.137994 / 1.023672 | | 43_oe_ember_lr02 / d6_full953_oe_ember_lr02 (OE-Ember, lr=0.20) | 240 / 953 | 3 / 3 | 110.49 / 160.74 | 1.132978 / 1.019890 | 1.138392 / 1.020597 | 1.133745 / 1.020667 | 1.136328 / 1.023309 | | B3_ember_cyclic04 / full_14_b3_ember_cyclic04 (Fixed cyclic VE) | 240 / 953 | 3 / 3 | 110.49 / 160.74 | 1.131443 / 1.019126 | 1.136871 / 1.019848 | 1.132185 / 1.019849 | 1.134662 / 1.022411 | | P2_ember_learned_proj / full_19_ember_learned_proj ($W_{\text{proj}}$) | 240 / 953 | 3 / 3 | 110.77 / 161.02 | 1.129370 / 1.017638 | 1.134834 / 1.018392 | 1.130191 / 1.018426 | 1.132990 / 1.021205 | | V9_240_FE_basis_gate / d6_full953_FE_basis_gate_isoparam (FactorEngram [6] basis-gate) | 240 / 953 | 3 / 3 | 110.49 / 160.74 | 1.127364 / 1.015425 | 1.132844 / 1.016151 | 1.128376 / — | 1.131897 / — (Full: 1.131150) | | V9_240_NG6_isoparam / d6_full953_NG6_isoparam (L0-only CoDeM, isoparam) | 240 / 953 | 3 / 3 | 110.49 / 160.74 | 1.126285 / 1.013828 | 1.131821 / 1.014593 | 1.127303 / — | 1.130752 / — (Full: 1.130058) | | V9_240_NG7_isoparam / d6_full953_NG7_isoparam (CoDeM-Ngram) | 240 / 953 | 3 / 3 | 110.49 / 160.74 | 1.125059 / 1.011987 | 1.130673 / 1.012689 | 1.126128 / 1.013015 | 1.129928 / 1.016880 (Full: 1.129078 / 1.016027) |

Reviewer report

Accept

The paper establishes a useful, bounded improvement to hashed n-gram memory in the supplied pretraining setting. Its strongest contribution is the controlled evidence for combining context-dependent input injection with reuse of the resulting phrase features in attention-value pathways. The calibration study adds a practical application of ordinary ridge regression to these shared features. Acceptance rests on these empirical findings, rather than a new general theory of collision removal or quantization.

The evidence goes beyond an isolated favorable run. At both backbone sizes, the paper includes parameter-matched comparisons, three training seeds, full-horizon evaluation, learning curves, and longer-training OE controls. Independently recomputing differences from the retained V9 records gives improvements over the longer-training OE control of $-2.957$ mBPB at $L=6$ and $-2.538$ mBPB at $L=8$, with paired 95% Student-$t$ intervals $[-3.158,-2.755]$ and $[-3.478,-1.598]$. Against the gate-detachment-matched unigram pathway, the corresponding improvements are $-2.090$ and $-1.603$ mBPB. These comparisons substantively connect the contribution to the task's objective of improving prediction for comparable resources.

The main forward computation is understandable from Eqs. (7)–(11), including the important distinction between the pre-projection phrase features and the post-gated Layer-0 output. The full-space ridge solution and its folding into multiple linear readouts are mathematically sound. The retained records support the reported additional 2-bit calibration gains of approximately $0.874$ and $0.880$ mBPB. My verification consisted of code inspection, independent arithmetic checks on retained records, and literature comparison; I did not reproduce GPU training or checkpoint evaluation.

The following qualifications and corrections should accompany the contribution:

  1. Keep the novelty and competitor claims precise. The relevant rStation papers [10]–[13] already establish low-bit table capacity, low-state writes, shared value embeddings, and unigram/n-gram routing. FactorEngram [6] already establishes componentwise contextual gating of retrieved coefficients and causal convolution. The present combination and its controlled measurements provide an incremental contribution beyond those results. However, FE_basis_gate_isoparam is a simplified, inspired comparator, not a reproduction of FactorEngram: the published method normalizes a projected backbone query, uses the reconstruction dictionary itself for gating, includes coefficient sparsity regularization, and studies internal insertion sites. The independent single-token gate used here omits those features. The reported advantage therefore establishes superiority to the specified control, not to the complete published method. Likewise, the joint change from that control to NG6 does not separately identify the effects of channel rolls, additional context, and the second gate.

  2. Add the closest broader precedents. MoVE [14] already shares memory across attention-value pathways with contextual routing; QwT [15] already uses closed-form linear regression to compensate quantization error. Neither establishes this paper's exact shared, demodulated n-gram architecture or its measured calibration decomposition, but both materially inform the attribution. The novelty of Section 2.4 is the application and simultaneous folding into existing readouts, not least-squares quantization compensation itself. References [6] and these precedents predate the submission well beyond the concurrency window.

  3. Correct the claimed “exact executable specification.” Section 2.3 calls the matrix optimizer “Pre-Polar Equilibrated Muon,” but ngram_bench.py:muon_step_fused performs global normalization before Polar Express and adaptive rescaling afterward; it does not implement the pre-polar row/column equilibration described in [13]. The runners also ramp momentum from $0.85$ to $0.95$. The convolution inherits a separate AdamW group with $\beta=(0.9,0.95)$ and $\epsilon=10^{-8}$ from NextGenGPT.setup_optimizer, rather than the stated x0_params assignment and $\beta=(0.96,0.95)$. Section 2.4 should specify one codebook per table fitted to pooled normalized entries, with a sample cap of 500,000 and initialization by equally spaced points in $[-1.5,1.5]$; the code does not initialize at empirical quantiles or fit separate codebooks per row. These are localized specification corrections to the inspectable implementation underlying the comparisons.

  4. Separate evidence for efficacy from evidence for mechanism. Shrinking the modulus changes capacity and predictor quality as well as collision density. Common-reference purity bins ensure comparable evaluation populations, but they do not measure each differently hashed model's own collision severity. The findings are consistent with collision mitigation and contextual fallback, as the limitations acknowledge; they do not uniquely establish demodulation as the cause. Similarly, average row-exposure ratios alone do not establish why phrase routing overtakes unigram routing. Keep that explanation explicitly hypothetical.

  5. Clarify confirmation and reproducibility. Document deduplication establishes source separation, not independence from model selection. The same fresh_rg42 split appears in both V8 and V9, so explain which decisions were frozen before its first use. Also, the released heldout_dedup iterator traverses all row groups, rather than explicitly enforcing the stated 0..41 restriction; record the actual consumed document range to substantiate that description. The supplement depends on baseline/prepare.py and baseline/attention_backend.py, so replace the self-contained claim with concrete setup instructions connecting these imports to the supplied task archive. Three-seed intervals should be described as exploratory, especially after extensive configuration screening.

Clarity is adequate for acceptance: the research question, computation, decisive comparisons, and limitations are identifiable. Nevertheless, the main text would be substantially easier to read with short descriptive arm names, one primary result table, and version-specific repetitions moved to an appendix. The abstract is informative but overcrowded. Preserve the distinction between module and whole-model compression, and between simulated packed storage and measured execution. The evidence supports two small backbone sizes on one corpus and recipe; it does not establish broader backbone transfer or native low-bit speedups. The disclosed microbatch change and bridge comparison also make the $L=8$ results an adaptation of the prescribed protocol, rather than a strict microbatch-64 comparison.

Additional references, continuing the manuscript's numbering:

[14] Yangyan Li. MoVE: Mixture of Value Embeddings — A New Axis for Scaling Parametric Memory in Autoregressive Models. 2026. https://arxiv.org/abs/2601.22887

[15] Minghao Fu et al. Quantization without Tears. arXiv:2411.13918, 2024; revised 2025. https://arxiv.org/abs/2411.13918

Loading…