# Context-Demodulated Multi-Pathway Phrase Memory and Ridge-Regularized L2 Readout Calibration for Low-Bit N-Gram Language Models

## 1. Introduction and Related Work

Scaling input vocabulary and N-gram lookup tables decouples knowledge capacity from per-token FLOPs. Over-Tokenized Transformers (`OE`) [1] and SCONE [2] show that large multi-hash N-gram input tables substantially improve language modeling loss at fixed backbone FLOPs, while Engram [3], TN-gram [4], and FactorEngram [6] introduce structured compression and contextual routing for hashed phrase memories. Because storing two FP32 Adam moments triples table memory footprint, recent optimizer work replaces dense Adam states on token interfaces: Frequency-aware SGD [9] scales sparse embedding updates by per-row frequency, Adafactor [8] factors second-moment matrices into row and column moving averages, and Shah's Ember (`OE-Ember`) [5] adapts kilobyte-state factored row/column RMS normalization specifically to over-tokenized tables. Concurrently, Chung and Du [7] demonstrate post-training and quantization-aware low-bit N-gram embeddings (`LowBit-OE`).

Within the rStation `general_pretraining` benchmark, four published studies establish important constraints that inform our design:
- **Capacity vs. precision trade-offs (`general_pretraining_6/v4` [10])**: Demonstrates that hashed N-gram capacity scales primarily with row count and token exposure while tolerating aggressive bit-width reduction (2–4 bits), motivating our focus on combining parameter-matched architectural routing with 2-bit/3-bit post-training quantization.
- **Count-normalized moment-free writes vs. factored states (`general_pretraining_11/v3` [11])**: Shows that hashed N-gram rows require minimal optimizer history once updates are properly scaled by row exposure, and analyzes the gap between pure moment-free writes and factored-state alternatives (`OE-Ember`).
- **Shared value embeddings (`general_pretraining_9/v1` [12])**: Separates initialization scale, learned routing coefficients, and lookup cost when sharing value embeddings across attention layers. This directly motivates our $2\times 2$ factorial control (`Section 4.3`) separating $0.25I_d$ initialization and joint backpropagation from hashed N-gram phrase content.
- **Cross-family composition and unigram/N-gram routing (`general_pretraining_16/v5` [13])**: Analyzes how hashed N-gram input tables interact with backbone representations and unigram pathways under Pre-Polar Equilibrated Muon training.

**Relationship to `FactorEngram` [6] and `Engram` [3].** The closest external architectural precedent is Yang et al.'s `FactorEngram` [6] (public September 28, 2026), which retrieves low-rank factorized N-gram coefficients, applies context-dependent basis-level gating before dictionary reconstruction, and smooths representations with a causal depthwise convolution, building on `Engram`'s [3] context-gated lookup and convolution. Because `FactorEngram` [6] combines three ingredients—low-rank table factorization, basis-level contextual gating prior to projection, and causal depthwise convolution—we isolate its gating-and-convolution pattern directly on unfactorized `OE-Ember` narrow-slice tables at exact parameter parity (`FE_basis_gate_isoparam` and `FE_basis_postconv_isoparam`, `Section 2.3` and `Section 4.1`), alongside full-table `Engram` (`L4`) and rank-$640$ factorized `TN-gram` (`L3`). Our contribution builds on and extends `FactorEngram` [6] in three specific, empirically isolated ways:
1. *Channel-rolled 3-token unigram-lag summary and dual pre/post gating (`NG6-IsoParam` vs. `FE_basis_gate_isoparam`, `L4_engram`, and `L3_tngram`)*: Replacing `FactorEngram`-style single-token basis gating (`FE_basis_gate_isoparam`, which gates raw slices by $2\sigma(W_{\text{fe}}w_{0,t})$ before $\text{Conv1d}_{\text{dw}}(k=4)$ and $W_{\text{proj}}$) with our channel-rolled 3-token unigram summary $c_t$ and dual pre/post gates (`NG6-IsoParam`) at exact parameter parity improves full-horizon `953`-step validation BPB on all 3 seeds by **$-1.597\text{ mBPB}$** at $L=6$ (`1.013828` vs. `1.015425`, CI $[-3.15, -0.05]$) and **$-1.265\text{ mBPB}$** at $L=8$ (`0.971853` vs. `0.973118`, CI $[-1.68, -0.85]$; **$-1.298\text{ mBPB}$** on `fresh_rg42_dedup`). Moreover, avoiding low-rank table factorization at the `110.5 MiB` budget gives `CoDeM-Ngram` an **$-11.05\text{ mBPB}$** advantage over `Engram` (`L4`, `1.023127`) and **$-11.66\text{ mBPB}$** over rank-$640$ `TN-gram` (`L3`, `1.023739`).
2. *Zero-lookup multi-pathway reuse into deeper Value Embeddings (`NG7-IsoParam` vs. `NG6-IsoParam` and `C2b_gate_det_isoparam`)*: Neither `Engram` nor `FactorEngram` routes its Layer-0 demodulated phrase representation into deeper attention Value-Embedding (`VE`) skips. Routing the pre-projection demodulated phrase vector $\tilde{u}_t$ into alternating `VE` layers via $W_{\text{pve}}^{(l)}$ adds zero hash lookups and zero table bytes while improving full-horizon validation BPB by **$-1.614\text{ to }-1.841\text{ mBPB}$** over Layer-0-only `NG6-IsoParam` at $L=6$ and **$-1.684\text{ to }-1.699\text{ mBPB}$** at $L=8$ (totaling **$-3.211\text{ to }-3.438\text{ mBPB}$** over `FE_basis_gate_isoparam` at $L=6$ and **$-2.949\text{ to }-2.964\text{ mBPB}$** at $L=8$), and outperforming strictly gate-detachment-matched `Unigram-VE` (`C2b_gate_det_isoparam`) by **$-2.090\text{ to }-2.316\text{ mBPB}$** at $L=6$ and **$-1.603\text{ to }-1.619\text{ mBPB}$** at $L=8$ (`Section 4.3`).
3. *Closed-form multi-readout ridge-regularized L2 calibration folded into $W_{\text{proj}}$ and $\{W_{\text{pve}}^{(l)}\}$*: Because `CoDeM-Ngram` feeds $\tilde{u}_t$ into linear readouts $W_{\text{proj}}$ (Layer 0) and $\{W_{\text{pve}}^{(l)}\}$ (`VE` skips), 2-bit post-training quantization error across the 4 concatenated hash slices is partially compensated at zero deployment overhead by folding a single ridge-regularized least-squares matrix $(M^*)^\top$ into both readout pathways (`Section 2.4` and `Section 4.4`).

---

## 2. Exact Executable Specification

All equations below match the exact executed computation in `ngram_bench.py`, `run_ngram_nextgen.py`, `run_ngram_v8_suite.py`, and `run_ngram_v9_fixes.py` (included in the supplementary archive). Throughout, $L \in \{6, 8\}$ denotes Transformer depth (number of layers) and $d = 64L \in \{384, 512\}$ denotes hidden model width.

### 2.1 Base-$(V+1)$ Polynomial N-Gram Hashing and Exact Parameter Parity (`GPT.oe_ids`)
Let $x = (x_0, \dots, x_{T-1}) \in \{0, \dots, V-1\}^T$ be a token sequence of length $T=2048$ with vocabulary size $V = 8192$ (`gpt-neo-125M` BPE). To distinguish sequence boundaries from token ID $0$, `GPT.oe_ids` (`ngram_bench.py`) pads out-of-bounds left positions with the sentinel value $V$:
$$\tilde{x}_{t-\ell} = \begin{cases} x_{t-\ell} & \text{if } t - \ell \ge 0, \\ V & \text{if } t - \ell < 0, \end{cases} \qquad \ell \in \{1, 2\}. \tag{1}$$
With base $B = V + 1 = 8193$, each position $t$ forms exact base-$(V+1)$ polynomial integers for bigrams ($i=2$) and trigrams ($i=3$):
$$I_t^{(2)} = x_t + B \tilde{x}_{t-1}, \qquad I_t^{(3)} = x_t + B \tilde{x}_{t-1} + B^2 \tilde{x}_{t-2}. \tag{2}$$
For $K=2$ hash functions per order, the 4 narrow slices are indexed by $j = (i-2)K + s \in \{0, 1, 2, 3\}$ for $i \in \{2, 3\}$ and $s \in \{0, 1\}$. Given base modulus $m$, slice $j$ has $m_j = m + 2j$ rows (`m, m+2, m+4, m+6`; no prime-rounding step is performed) and width $d_s = d / 4$, with bucket index:
$$h_{t,j} = I_t^{(i)} \bmod (m + 2j), \qquad E_j \in \mathbb{R}^{(m+2j)\times d_s}. \tag{3}$$
The concatenated raw lookup vector is $e_{\text{raw},t} = \operatorname{Concat}_{j=0}^3\big(E_j[h_{t,j}]\big) \in \mathbb{R}^d$. In baseline `OE-Ember` (`learned_proj_backoff=False`), Layer 0 adds $e_{\text{raw},t}$ directly to the unigram embedding before the shared input `RMSNorm`: $x_t^{(0)} = \operatorname{RMSNorm}\big(E_{\text{uni}}[x_t] + e_{\text{raw},t}\big)$ (`ngram_bench.py`).

**Exact Parameter Parity (`NG7-IsoParam` and Parameter-Matched Controls).** Because contextual gating, depthwise convolution, readout, and `VE` routing add learned mixer parameters, we reduce the base modulus $m$ so that total memory-subsystem parameters (`table_params + mixer_router_params`) match baseline `OE-Ember` ($m=150,855$) to the single parameter:
- **Depth $L=6$ (width $d=384, d_s=96$, 3 `VE` layers $l \in \{1, 3, 5\}$)**: Baseline `OE-Ember` ($m=150,855$) has $\sum_{j=0}^3 (150,855 + 2j)\times 96 = (4\times 150,855 + 12)\times 96 = 57,929,472$ table parameters ($110.492\text{ MiB}$ BF16 module; $84,275,244$ whole-model parameters $= 160.742\text{ MiB}$ BF16 with the $26,345,772$-parameter / $50.251\text{ MiB}$ backbone).
  - `CoDeM-Ngram` (`NG7-IsoParam`) and `Unigram-VE` controls (`C2b`, `C2b_gate_det`, `C2d`) use $m = 148,547$, giving $(4\times 148,547 + 12)\times 96 = 57,043,200$ table parameters plus $886,272$ mixer/router parameters ($W_{\text{ctx}} \in \mathbb{R}^{2d\times d}$ $= 2d^2$, $W_{\text{proj}} \in \mathbb{R}^{d\times d}$ $= d^2$, $3\times W_{\text{pve}}^{(l)} \in \mathbb{R}^{d\times d}$ $= 3d^2$, and `phrase_dw_conv` $\in \mathbb{R}^{d\times 1\times 4}$ $= 4d$, all bias-free, totaling $6d^2 + 4d = 886,272$) for **$57,929,472$ total memory parameters ($110.492\text{ MiB}$ BF16 module; $160.742\text{ MiB}$ BF16 whole model)**.
  - Layer-0-only `NG6-IsoParam` ($W_{\text{ctx}} \in \mathbb{R}^{2d\times d}$, $W_{\text{proj}} \in \mathbb{R}^{d\times d}$, `phrase_dw_conv` $\in \mathbb{R}^{d\times 1\times 4}$: $3d^2 + 4d = 443,904$ mixer parameters) uses $m = 149,699$ ($57,485,568$ table parameters) for **$57,929,472$ total memory parameters**.
  - Single-gate `FactorEngram`-style basis-gate controls `FE_basis_gate_isoparam` and `FE_basis_postconv_isoparam` ($W_{\text{fe}} \in \mathbb{R}^{d\times d}$, $W_{\text{proj}} \in \mathbb{R}^{d\times d}$, `phrase_dw_conv` $\in \mathbb{R}^{d\times 1\times 4}$: $2d^2 + 4d = 296,448$ mixer parameters) use $m = 150,083$ ($57,633,024$ table parameters) for **$57,929,472$ total memory parameters**.
- **Depth $L=8$ (width $d=512, d_s=128$, 4 `VE` layers $l \in \{1, 3, 5, 7\}$)**: Baseline `OE-Ember` ($m=150,855$) has $(4\times 150,855 + 12)\times 128 = 77,239,296$ table parameters ($147.322\text{ MiB}$ BF16 module; $127,571,472$ whole-model parameters $= 243.323\text{ MiB}$ BF16 with the $50,332,176$-parameter / $96.001\text{ MiB}$ backbone). `NG7-IsoParam`, `C2b`, and `C2b_gate_det` use $m = 147,267$ ($75,402,240$ table parameters $+ (7d^2 + 4d = 1,837,056)$ mixer parameters $= \mathbf{77,239,296}$); `NG6-IsoParam` uses $m = 149,315$ ($76,450,816 + (3d^2 + 4d = 788,480) = \mathbf{77,239,296}$); and `FE_basis_gate_isoparam` uses $m = 149,827$ ($76,712,960 + (2d^2 + 4d = 526,336) = \mathbf{77,239,296}$).

### 2.2 Factored Second-Moment (`OE-Ember`) and Empirical-Bayes (`EB1`) Table Updates
Let $G \in \mathbb{R}^{m_j \times d_s}$ be the accumulated FP32 gradient on slice table $E_j$ at step $t$, let $n_b = \sum_{\tau} \mathbb{I}[h_{\tau,j} = b]$ be the batch hit count for row $b$, and let $\mathcal{A}_j = \{b : n_b > 0\}$ be the active row set.
- **Implemented `OE-Ember` (`oe_mfree_step` in `ngram_bench.py`)**: Each table maintains FP32 factored buffers $r \in \mathbb{R}^{m_j}$ and $c \in \mathbb{R}^{d_s}$ ($2.303\text{ MiB}$ total across all 4 tables at $m=150,855$). At each step, `row_mean_sq` is non-zero on active rows ($b \in \mathcal{A}_j$) and zero on inactive rows ($b \notin \mathcal{A}_j$), while `col_mean_sq` averages over all $m_j$ rows:
  $$s_b^{\text{row}} = \begin{cases} \frac{1}{d_s}\sum_{k=1}^{d_s} G_{b,k}^2 & \text{if } b \in \mathcal{A}_j, \\ 0 & \text{if } b \notin \mathcal{A}_j, \end{cases} \qquad s_k^{\text{col}} = \frac{1}{m_j}\sum_{b=0}^{m_j-1} G_{b,k}^2 \quad (k \in \{1,\dots,d_s\}). \tag{4}$$
  With $\beta_2 = 0.95$ (`adam_betas = (0.8, 0.95)`) and global step bias correction $\text{bias}_2 = 1 - \beta_2^t$, both buffers update over all indices via `lerp_(..., 1 - beta2)`: $r_b \leftarrow \beta_2 r_b + (1-\beta_2) s_b^{\text{row}}$ ($\forall b \in \{0,\dots,m_j-1\}$, so inactive rows decay by $\beta_2$) and $c_k \leftarrow \beta_2 c_k + (1-\beta_2) s_k^{\text{col}}$. Defining $\hat{r}_b = \max(r_b / \text{bias}_2, 10^{-30})$ ($b \in \mathcal{A}_j$), $\hat{c}_k = \max(c_k / \text{bias}_2, 10^{-30})$, and $\tilde{c}_k = \hat{c}_k / \max(\frac{1}{d_s}\sum_{k'} \hat{c}_{k'}, 10^{-30})$, the active-row update places $\epsilon = 10^{-10}$ outside the square root:
  $$E_j[b, k] \leftarrow E_j[b, k] - \eta_{\text{oe}}(t) \frac{G_{b,k}}{\sqrt{\hat{r}_b \, \tilde{c}_k} + \epsilon}, \qquad b \in \mathcal{A}_j, \tag{5}$$
  where the configured base learning rate $\eta_{\text{oe},0}$ is scaled by `dmodel_lr_scale` $= (d/768)^{-0.5}$ (`ngram_bench.py`: $\sqrt{2} \approx 1.4142$ at $d=384$, mapping `0.20 -> 0.2828` and `0.30 -> 0.4243`; $\sqrt{1.5} \approx 1.2247$ at $d=512$, mapping `0.20 -> 0.2449` and `0.30 -> 0.3674`) and follows the linear warmdown schedule.
- **Implemented Moment-Free Empirical-Bayes (`EB1` in `ngram_bench.py`)**: Configured with `homo_mfree=True, homo_alpha=0.50, homo_row_beta=0.0, homo_coord_gamma=0.0` and order-specific prior pseudo-counts $\kappa_2 = 2.0$ (bigrams) and $\kappa_3 = 5.0$ (trigrams), active gradients are damped by $\tilde{g}_b = G_b / \sqrt{n_b + \kappa_i}$ ($b \in \mathcal{A}_j$) and normalized by the global active-row RMS $s_{\text{glob}} = \sqrt{\max(\operatorname{mean}_{b\in\mathcal{A}_j, k} \tilde{g}_{b,k}^2, 10^{-30})}$:
  $$E_j[b] \leftarrow E_j[b] - \eta_{\text{oe}}(t) \frac{\tilde{g}_b}{s_{\text{glob}}}, \qquad b \in \mathcal{A}_j. \tag{6}$$

### 2.3 Context-Demodulated Multi-Pathway Phrase Memory (`CoDeM-Ngram` / `NG7-IsoParam`)
As implemented in `NextGenGPT` (`run_ngram_nextgen.py`), `V8NextGenGPT` (`run_ngram_v8_suite.py`), and `V9NextGenGPT` (`run_ngram_v9_fixes.py`), `CoDeM-Ngram` transforms the raw 4-slice lookup $e_{\text{raw},t} = \operatorname{Concat}_{j=0}^3(E_j[h_{t,j}]) \in \mathbb{R}^d$ through five stages:
1. **Channel-Rolled 3-Token Unigram Lag Summary**: Let $w_{0,t} = \operatorname{RMSNorm}(E_{\text{uni}}[x_t]) \in \mathbb{R}^d$ be the RMS-normalized unigram embedding (`wte`), and let $w_{1,t}, w_{2,t}$ be causal temporal lags (zero-padded at $t < \ell$) circularly rolled along the channel dimension by $+1$ and $+2$ positions (`torch.roll(..., shifts=1, dims=-1)` and `shifts=2`) to break symmetry across lags without extra parameters:
   $$w_{1,t} = \operatorname{Roll}_{\text{chan},1}(w_{0,t-1}), \qquad w_{2,t} = \operatorname{Roll}_{\text{chan},2}(w_{0,t-2}), \qquad c_t^{\text{raw}} = \frac{1}{\sqrt{3}}\!\left(w_{0,t} + \frac{1}{\sqrt{2}}\,w_{1,t} + \frac{1}{2}\,w_{2,t}\right) \in \mathbb{R}^d. \tag{7}$$
   To prevent Layer-0 gate gradients from perturbing `wte`, the gate input uses $c_t = \operatorname{stopgrad}(c_t^{\text{raw}})$ (`gate_detach_wte=True`).
2. **Dual Context Gate (`bcd_ctx_proj`) and Pre-Projection Demodulation**: A single bias-free linear projection $W_{\text{ctx}} \in \mathbb{R}^{2d\times d}$ (`nn.Linear(d, 2 * d, bias=False)`) maps $c_t$ into pre- and post-projection gates around unity, and $g_t^{\text{pre}}$ directly modulates the raw lookup $e_{\text{raw},t}$:
   $$\big(g_t^{\text{pre}},\, g_t^{\text{post}}\big) = \operatorname{Chunk}_2\!\big(2\,\sigma(W_{\text{ctx}} c_t)\big) \in (0, 2)^d \times (0, 2)^d, \qquad \tilde{u}_t^{(0)} = e_{\text{raw},t} \odot g_t^{\text{pre}}. \tag{8}$$
3. **Causal Depthwise Local Smoothing (`phrase_dw_conv`)**: A depthwise 1D convolution $\text{Conv1d}_{\text{dw}}$ with kernel size $k=4$, groups $= d$ ($4d$ parameters, no bias), causal left zero-padding of 3 positions on $\operatorname{RMSNorm}(\tilde{u}^{(0)})$, and `SiLU` nonlinearity refines the demodulated slices:
   $$\tilde{u}_t = \tilde{u}_t^{(0)} + \operatorname{SiLU}\!\Big(\operatorname{Conv1d}_{\text{dw}}\!\big(\operatorname{RMSNorm}(\tilde{u}^{(0)})_{t-3:t}\big)\Big) \in \mathbb{R}^d. \tag{9}$$
4. **Cross-Slice Linear Readout (`oe_proj`) and Post-Gated Layer-0 Injection**: A bias-free readout matrix $W_{\text{proj}} \in \mathbb{R}^{d\times d}$ mixes all 4 narrow slices into the backbone coordinate frame, applies $g_t^{\text{post}}$, and adds the result to $E_{\text{uni}}[x_t]$ prior to the shared input `RMSNorm`:
   $$e_{\text{L0},t} = \big(W_{\text{proj}} \tilde{u}_t\big) \odot g_t^{\text{post}}, \qquad x_t^{(0)} = \operatorname{RMSNorm}\!\big(E_{\text{uni}}[x_t] + e_{\text{L0},t}\big). \tag{10}$$
5. **Zero-Lookup Multi-Pathway Value-Embedding (`Phrase-VE`) Routing (`pve_projs`)**: In alternating attention layers $l \in \mathcal{L}_{\text{ve}}$ ($\{1, 3, 5\}$ at $L=6$; $\{1, 3, 5, 7\}$ at $L=8$), the pre-projection demodulated phrase vector $\tilde{u}_t$ (before $W_{\text{proj}}$ and $g_t^{\text{post}}$) is routed into the attention Value tensor $v_t^{(l)}$ through a layer-specific bias-free matrix $W_{\text{pve}}^{(l)} \in \mathbb{R}^{d\times d}$ modulated by the existing per-head attention value gate:
   $$v_t^{(l)} \leftarrow v_t^{(l)} + 2\,\sigma\!\big(g_{\text{ve}}^{(l)}(x_t^{(l,\text{in})})\big) \odot \big(W_{\text{pve}}^{(l)} \tilde{u}_t\big). \tag{11}$$
   In `NG7-IsoParam`, gradients from all $l \in \mathcal{L}_{\text{ve}}$ flow jointly through $\tilde{u}_t$ into $\text{Conv1d}_{\text{dw}}$, $W_{\text{ctx}}$, and the N-gram tables $\{E_j\}_{j=0}^3$ (note that `Phrase-VE` branches from $\tilde{u}_t$ before $W_{\text{proj}}$, so `VE` gradients update the shared tables and Layer-0 gate/conv directly without passing through $W_{\text{proj}}$).

**Controlled `FactorEngram` [6] Basis-Gate Baselines and Unigram-`VE` Controls (`V9NextGenGPT` in `run_ngram_v9_fixes.py`).** To isolate our 3-token channel-rolled lag summary $c_t$ and dual pre/post gates against Yang et al.'s `FactorEngram` [6] basis-level gating on the exact same `OE-Ember` tables and parameter budget, `FE_basis_gate_isoparam` replaces Eqs. (7)–(8) with a single-token unigram gate $g_t^{\text{fe}} = 2\sigma(W_{\text{fe}}\operatorname{stopgrad}(w_{0,t}))$ ($W_{\text{fe}} \in \mathbb{R}^{d\times d}$), forms $\tilde{u}_t^{(0)} = e_{\text{raw},t} \odot g_t^{\text{fe}}$, applies Eq. (9), and projects via $e_{\text{L0},t} = W_{\text{proj}}\tilde{u}_t$; `FE_basis_postconv_isoparam` places the causal depthwise $\text{Conv1d}_{\text{dw}}(k=4)$ after $W_{\text{proj}}$. For the unigram-`VE` controls, `C2d_isoparam` (`pve_detach=True`) and `C2b_gate_det_isoparam` (`pve_detach=False`) route $\operatorname{stopgrad}(c_t^{\text{raw}})$ or $c_t^{\text{raw}}$ into $W_{\text{pve}}^{(l)}$ while holding `gate_detach_wte=True` ($c_t = \operatorname{stopgrad}(c_t^{\text{raw}})$ in Eq. 8) strictly fixed across `C2d`, `C2b_gate_det`, and `NG7`. (In `V8NextGenGPT`'s `C2b`, setting `pve_detach=False` also left $c_t$ attached to `wte` inside Eq. 8; comparing `C2b` against `C2b_gate_det` and `NG7` in Section 4.3 isolates both effects.)

**Exact Initialization and Optimizer Assignment (`NextGenGPT.init_weights` and `V9NextGenGPT.init_weights`).** All N-gram tables $E_j$ are zero-initialized. With $s = \sqrt{3/d}$, $W_{\text{proj}}$ uses standard uniform initialization $\mathcal{U}[-s, +s]$, $W_{\text{ctx}}$ (and $W_{\text{fe}}$) is initialized uniformly in $\mathcal{U}[-0.25s, +0.25s]$ (`torch.nn.init.uniform_(self.bcd_ctx_proj.weight, -0.25 * s, 0.25 * s)`), $\text{Conv1d}_{\text{dw}}$ (`phrase_dw_conv`) is zero-initialized (so $\tilde{u}_t = \tilde{u}_t^{(0)}$ at step 0), and $W_{\text{pve}}^{(l)} = 0.25\,I_d$ in `NG7`/`C2b`/`C2b_gate_det`/`C2d` (or $0$ in zero-init controls `C2`/`C2c`). All 2D mixer/router matrices ($W_{\text{ctx}}, W_{\text{proj}}, W_{\text{pve}}^{(l)}$) are stored in FP32 and optimized by Pre-Polar Equilibrated Muon (`matrix_lr = 0.04`, momentum $0.95$, Nesterov, 5 Polar Express steps); `phrase_dw_conv` (`ndim == 3`) is optimized by AdamW in the `x0_params` group (`lr = 0.04`, $\beta=(0.96, 0.95)$).

### 2.4 Non-Uniform Lloyd-Max PTQ and Multi-Readout Ridge-Regularized L2 Calibration
For low-bit post-training quantization (PTQ) to $b \in \{2, 3\}$ bits (`quantize_tables_lloyd_max` in `run_ngram_suite3.py` and `run_ngram_v9_fixes.py`, following Chung & Du [7]), each active row $r$ of slice table $E_j$ is RMS-normalized to form a 1D empirical distribution, clustered into $2^b$ non-uniform centroids ($4$ levels for 2-bit, $8$ levels for 3-bit) via 20 iterations of 1D Lloyd-Max k-means (initialized at empirical quantiles), and refined with 5 alternating least-squares steps updating per-row BF16 scales $s_{j,r} = (\sum_k E_j[r,k] c_{q_{r,k}}) / (\sum_k c_{q_{r,k}}^2)$ and nearest centroid assignments $q_{r,k}$, with centroids and row scales stored in BF16.

On $N_{\text{cal}} = \max(1, (24\times 32)/\text{mb})$ training batches ($768 \times 2048 = 1,572,864$ calibration tokens), let $\tilde{e}_t \in \mathbb{R}^d$ and $\tilde{q}_t \in \mathbb{R}^d$ denote the full-precision and quantized pre-projection features produced by Eq. (9), and let $d_t = \tilde{q}_t - \tilde{e}_t$ be the pre-projection quantization error vector. Accumulating the second-moment matrices $\Sigma_{qq} = \frac{1}{N}\sum_t \tilde{q}_t \tilde{q}_t^\top$, $\Sigma_{qe} = \frac{1}{N}\sum_t \tilde{q}_t \tilde{e}_t^\top$, $\Sigma_{ee} = \frac{1}{N}\sum_t \tilde{e}_t \tilde{e}_t^\top$, and $\Sigma_{dd} = \frac{1}{N}\sum_t d_t d_t^\top$ in FP64 with zero-centered ridge penalty $\lambda = \lambda_{\text{rel}} \frac{1}{d}\operatorname{tr}(\Sigma_{qq})$ ($\lambda_{\text{rel}} = 10^{-4}$), we solve:
$$M^* = \arg\min_{M \in \mathcal{S}} \left\{ \frac{1}{N}\sum_{t=1}^N \big\|\tilde{e}_t - M^\top \tilde{q}_t\big\|_2^2 + \lambda \|M\|_F^2 \right\} = \big(\Sigma_{qq} + \lambda I_d\big)^{-1}\Sigma_{qe}. \tag{12}$$
Because $\tilde{q}_t$ feeds both the Layer-0 readout $W_{\text{proj}} \tilde{q}_t$ (Eq. 10) and the `Phrase-VE` projections $W_{\text{pve}}^{(l)} \tilde{q}_t$ (Eq. 11), folding $M^*$ into any readout matrix $W$ via $\widehat{W} \leftarrow W (M^*)^\top$ replaces $W \tilde{q}_t$ with $W (M^*)^\top \tilde{q}_t$ at zero inference overhead. In `fit_and_eval_ptq_readout_suite` (`run_ngram_v9_fixes.py`), we decompose this calibration along two axes:
1. **Structural Constraint Space $\mathcal{S}$**: (a) **`Diag`** ($d$ parameters: $M_{\text{diag}} = \operatorname{diag}(\operatorname{diag}(\Sigma_{qe}) / (\operatorname{diag}(\Sigma_{qq}) + \lambda))$), (b) **`Block-Diag`** ($4 d_s^2$ parameters: $M_{\text{blk}} = \operatorname{block\_diag}(M^{(0)}, M^{(1)}, M^{(2)}, M^{(3)})$ with $M^{(j)} = (\Sigma_{qq}^{(j,j)} + \lambda I_{d_s})^{-1}\Sigma_{qe}^{(j,j)}$), and (c) **`Full`** ($d^2$ parameters: dense $M_{\text{full}} \in \mathbb{R}^{d\times d}$).
2. **Readout Pathway Isolation (`full_l0_only`, `full_ve_only`, `full_combined`)**: For models with `Phrase-VE` (`NG7-IsoParam`), we separately evaluate folding $M_{\text{full}}^\top$ into $W_{\text{proj}}$ only (`full_l0_only`), into $\{W_{\text{pve}}^{(l)}\}$ only (`full_ve_only`), and into both pathways simultaneously (`full_combined`), reporting both Layer-0 readout MSE reduction and `VE` readout MSE reduction alongside full-precision feature cross-slice correlation $R^2(\tilde{e}^{(a)}, \tilde{e}^{(b)})$ (from $\Sigma_{ee}$) and quantization-error cross-slice correlation $R^2(d^{(a)}, d^{(b)})$ (from $\Sigma_{dd}$).

![Figure 1: CoDeM-Ngram architecture and FactorEngram/Unigram-VE controls, full-horizon L=6 and L=8 scaling and equal-time comparisons, common-modulus purity-stratified gains, and multi-readout 2-bit L2 calibration.](figure-1.png)

---

## 3. Experimental Setup, Data Provenance, and Resource Accounting

**Data Provenance and Untouched Confirmation Split (`verify_fresh_rg_provenance`).** All runs use the ClimbMix-400B benchmark dataset (`V=8192`):
- **Training set (`train`)**: Exact shards `shard_00000.parquet` through `shard_00008.parquet` ($9$ shards).
- **Validation set (`val`)**: Exact shard `shard_06542.parquet`, evaluated over `20,971,520` tokens ($320$ batches; `inputs_int32_sha256 = 1e696a20964bfbe6...`) on full-horizon runs and `5,242,880` tokens ($80$ batches; `inputs_int32_sha256 = ebbee9f2a5526f13...`) on `240`-step runs.
- **Deduplicated held-out split (`heldout_dedup`)**: Constructed from the first half of `shard_00009.parquet` (row groups `0..41` out of `83` total row groups `0..82`; `43,008` documents, `43,007` unique SHA1 hashes) by excluding all documents present in `_HELDOUT_CROSS_SPLIT_DUP_HASHES` (`26` raw train–val overlaps, `23` train–heldout overlaps, and `2` val–heldout overlaps across the shard; `11` excluded inside `rg 0..41`) before deterministic token packing (`build_eval_loader` in `ngram_bench.py`; `inputs_int32_sha256 = ebea7e3170d83b5e...` at $20.97\text{M}$ tokens and `62b973a88dc286df...` at $5.24\text{M}$ tokens).
- **Untouched final confirmation split (`fresh_rg42_dedup`)**: Constructed from the second half of `shard_00009.parquet` (row groups `42..82`; `41,984` total documents). As verified programmatically by `verify_fresh_rg_provenance(42, 83)` (`run_ngram_v9_fixes.py`) and logged in `exp_ngram_v9_*.json`, `rg 42..82` contains `16` documents matching `_HELDOUT_CROSS_SPLIT_DUP_HASHES` (train/val overlaps) and `1` document matching `rg 0..41`; that single `rg 0..41` overlap is already among the `16` train/val duplicates (`overlap_with_rg0_to_41_not_already_in_train_or_val = 0`), so both cross-split filtering and strict union filtering against `train ∪ val ∪ rg0..41` retain the exact same **`41,968` documents** (`cross_and_union_filters_identical = true`; `inputs_int32_sha256 = 7fb047bff60c054a...` at $20.97\text{M}$ tokens and `3dd8e72ed8132d52...` at $5.24\text{M}$ tokens).

**Full-Budget Depth $L=8$ ($d=512$) Microbatch Protocol Check (`mb=64` vs. `mb=32`).** The benchmark reference specifies depth $L=8$ ($d=512$, 8 layers, 4 heads, `50,332,176` backbone parameters), `953` steps at `524,288` tokens/step (`499.65M` tokens), and microbatch `device_batch_size = 64` (`grad_accum = 4`). On a 24 GiB GPU, `d=8` `plain` with `mb=64` consumes `22,804 MiB` peak CUDA VRAM, leaving insufficient headroom for a `147.32 MiB` N-gram subsystem and its activations. Setting `mb=32, grad_accum=8` preserves the `524,288` tokens/step effective batch size while halving activation VRAM to `11,701–12,534 MiB` (minor bfloat16 gradient-accumulation rounding differences remain between 8 vs. 4 microbatches). On Seed 42 (`exp_ngram_v8_d8full.json`), `d8_full953_plain_mb64` achieves `0.994026` val BPB (`0.994763` `heldout_dedup`, `0.994343` `fresh_rg42_dedup`, `1075.3 s` train) and `d8_full953_plain_mb32` achieves `0.993637` val BPB (`0.994301` `heldout_dedup`, `0.993985` `fresh_rg42_dedup`, `1088.6 s` train)—an empirical difference of $0.39\text{ mBPB}$ ($6\times$ smaller than the $-23.80\text{ mBPB}$ N-gram gain and $16\times$ smaller than the $-6.27\text{ mBPB}$ gain of `CoDeM-Ngram` over `OE-Ember`). All compared $L=8$ models use the identical `mb=32, grad_accum=8` setting.

**Forward + Backward FLOPs Accounting (`GPT.estimate_flops`).** Under the benchmark's per-token FLOPs formula `GPT.estimate_flops()` (`ngram_bench.py`: $6\times$ active non-embedding parameters $+$ causal sliding-window attention FLOPs $\sum_{l=1}^L 12\,n_{\text{head}}\,d_{\text{head}}\min(w_l, T)$ under window pattern `SSSL` with $w_l \in \{1024, 2048\}$):
- **Depth $L=6$ ($d=384, L_{\text{ve}}=3$)**: Baseline `plain` (and `OE-Ember`, whose table lookups are $O(1)$ indexing) executes $6\times 14,942,496 + 30,670,848 = \mathbf{120,325,824\text{ FLOPs/token}}$ (`120.33 MFLOPs/token`). `CoDeM-Ngram` (`NG7-IsoParam`) adds $886,272$ active mixer/router parameters ($6d^2 + 4d$), contributing $6\times 886,272 = \mathbf{5,317,632\text{ FLOPs/token}}$ (`+5.32 MFLOPs/token`), an increase of **`+4.42%`** (`125,643,456` vs. `120,325,824` FLOPs/token).
- **Depth $L=8$ ($d=512, L_{\text{ve}}=4$)**: Baseline `plain` / `OE-Ember` executes $6\times 31,457,792 + 50,331,648 = \mathbf{239,078,400\text{ FLOPs/token}}$ (`239.08 MFLOPs/token`). `CoDeM-Ngram` adds $1,837,056$ active mixer/router parameters ($7d^2 + 4d$), contributing $6\times 1,837,056 = \mathbf{11,022,336\text{ FLOPs/token}}$ (`+11.02 MFLOPs/token`), an increase of **`+4.61%`** (`250,100,736` vs. `239,078,400` FLOPs/token).

**Timing Conventions and Reproduction Entry Point.** In all tables, `Train s` (`training_seconds`) measures synchronized steady-state training time over steps `10..N-1` (excluding the first 10 compilation/warmup steps and periodic validation checks), `Loop s` (`training_loop_seconds`) measures the complete training loop from step `0..N-1` including compilation and the 6 periodic $2\text{M}$-token validation curve evaluations (`every 190` steps), and `Wall s` (`wall_clock_s`) includes post-training full-split (`20.97M`-token) evaluations and Lloyd-Max PTQ suites. The supplementary archive is self-contained: `ngram_bench.py` provides the tokenizer, ClimbMix-400B dataloaders, `GPT` backbone, Muon/Ember optimizers, and provenance verifiers; `run_ngram_suite3.py` provides `quantize_tables_lloyd_max`; `run_ngram_nextgen.py`, `run_ngram_v8_suite.py`, and `run_ngram_v9_fixes.py` execute the training and PTQ suites; `summarize_all_ngram_results.py` recomputes `ngram_authoritative_summary.json` from the raw run JSONs; and `generate_ngram_figure.py` generates `figure-1.png`.

**Training Memory Audit vs. Simulated Deployment Storage.** We distinguish three memory metrics in Table 1 (`audit_exact_memory_bytes` in `run_ngram_v8_suite.py`):
1. **Training memory subsystem (`Train Sub MiB`) and Peak CUDA VRAM**: During training, `OE-Ember` stores BF16 tables ($110.492\text{ MiB}$ at $L=6$; $147.322\text{ MiB}$ at $L=8$) plus FP32 factored second moments ($2.303\text{ MiB}$), totaling **$112.795\text{ MiB}$** ($L=6$) and **$149.626\text{ MiB}$** ($L=8$). `CoDeM-Ngram` (`NG7-IsoParam`) stores smaller BF16 tables ($108.802\text{ MiB}$ at $L=6$; $143.818\text{ MiB}$ at $L=8$), FP32 Ember row/col buffers ($2.268\text{ MiB}$ at $L=6$; $2.249\text{ MiB}$ at $L=8$), FP32 mixer/router master weights ($3.381\text{ MiB}$ at $L=6$; $7.008\text{ MiB}$ at $L=8$), and FP32 Muon/AdamW momentum buffers ($2.823\text{--}3.387\text{ MiB}$ at $L=6$; $6.027\text{ MiB}$ at $L=8$), totaling **$117.275\text{--}117.838\text{ MiB}$** at $L=6$ (`14,741 MiB` peak CUDA VRAM vs. `13,964 MiB` for `OE-Ember`) and **$159.103\text{ MiB}$** at $L=8$ (`12,532 MiB` peak CUDA VRAM vs. `12,003 MiB` for `OE-Ember`).
2. **BF16 Deployment Storage (`Mod MiB / Whole MiB`)**: Casting mixer/router weights to BF16 yields exact storage parity between `CoDeM-Ngram`, `FE_basis_gate_isoparam`, `NG6-IsoParam`, `C2b`, `C2b_gate_det`, and `OE-Ember`: **$110.492\text{ MiB}$ module / $160.742\text{ MiB}$ whole model** at $L=6$ (including the $50.251\text{ MiB}$ BF16 backbone), and **$147.322\text{ MiB}$ module / $243.323\text{ MiB}$ whole model** at $L=8$ (including the $96.001\text{ MiB}$ BF16 backbone).
3. **Simulated Low-Bit Storage (`Mod MiB / Whole MiB`)**: Low-bit evaluation uses exact dequantized Lloyd-Max simulation (`quantize_tables_lloyd_max`) and reports theoretical bit-packed table storage (2 or 3 bits/weight + BF16 per-row scales + $2^b$ centroids + BF16 mixer/router weights): at 2 bits, `CoDeM-Ngram` occupies **$16.424\text{ MiB}$ module / $66.675\text{ MiB}$ whole model** at $L=6$ (**$6.73\times$ module-only compression; $2.41\times$ whole-model compression**) and **$22.605\text{ MiB}$ module / $118.606\text{ MiB}$ whole model** at $L=8$ (**$6.52\times$ module-only compression; $2.05\times$ whole-model compression**).

---

## 4. Empirical Results, Causal Controls, and Mechanistic Analysis

### 4.1 Full-Horizon Comparisons at Depth $L=8$ and $L=6$ (Equal Steps, Equal Time, and `FactorEngram` Controls)
Table 1 reports all 3-seed full-horizon (`953`-, `1015`-, and `1042`-step) results at depth $L=8$ ($d=512$; `exp_ngram_v8_d8full.json`, `exp_ngram_v9_d8full.json`) and depth $L=6$ ($d=384$; `exp_ngram_v8_d6confirm.json`, `exp_ngram_v9_d6full.json`, `exp_ngram_scale_and_horizon.json`), alongside `240`-step reduced-budget results (`exp_ngram_v9_controls240.json`, `ngram_authoritative_summary.json`). We report pure steady-state training time (`Train s`), full training-loop time including compilation and periodic validation curve checks (`Loop s`), and complete run time including post-training multi-split and low-bit PTQ suites (`Wall s`).

| Setting & Configuration (`3` seeds: `41, 42, 43` unless noted) | Mod / Whole BF16 (`MiB`) | Train Sub / VRAM (`MiB`) | Step (`ms`) | Train / Loop / Wall (`s`) | Val BPB ($\pm$ SD) | Val CE / PPL | `heldout_dedup` BPB | `fresh_rg42` BPB |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **Depth $L=8$ ($d=512$), Full Budget (`953` / `1015` steps, `50.33M` backbone)** | | | | | | | | |
| `d8_full953_plain_mb64` (`mb=64`, Seed 42 bridge) | `0.00 / 96.00` | `0.00 / 22804` | `1140.3` | `1075.3 / 1117.1 / 1256.8` | `0.994026` (n=1) | `2.78530 / 16.205` | `0.994763` | `0.994343` |
| `d8_full953_plain_mb32` (`mb=32`, 3 seeds) | `0.00 / 96.00` | `0.00 / 11701` | `1165.8` | `1099.4 / 1139.6 / 1245.5` | `0.993953 ± 0.000371` | `2.78509 / 16.201` | `0.994671` | `0.994301` |
| `d8_full953_oe_ember_lr02_mb32` (`953` st, `m=150,855`) | `147.32 / 243.32` | `149.63 / 12003` | `1191.0` | `1123.1 / 1161.7 / 1346.0` | `0.976427 ± 0.000305` | `2.73598 / 15.425` | `0.977120` | `0.976871` |
| `d8_full1015_oe_ember_eqtime_mb32` (`1015` st, `+6.5%` tok) | `147.32 / 243.32` | `149.63 / 12003` | `1195.2` | `1201.2 / 1234.7 / 1419.6` | `0.972708 ± 0.000269` | `2.72556 / 15.265` | `0.973377` | `0.973140` |
| `d8_full953_FE_basis_gate_isoparam_mb32` (`FactorEngram` [6] basis-gate, `m=149,827`) | `147.32 / 243.32` | `151.63 / 12391` | `1215.7` | `1146.4 / 1195.7 / 1446.7` | `0.973118 ± 0.000379` | `2.72671 / 15.283` | `0.973777` | `0.973505` |
| `d8_full953_NG6_isoparam_mb32` (L0-only `CoDeM`, `m=149,315`) | `147.32 / 243.32` | `153.13 / 12522` | `1247.2` | `1176.2 / 1220.0 / 1298.3` | `0.971853 ± 0.000350` | `2.72317 / 15.228` | `0.972492` | `0.972208` |
| `d8_full953_C2b_gate_det_isoparam_mb32` (`Unigram-VE`, `gate_detach=True`, `m=147,267`) | `147.32 / 243.32` | `159.10 / 12596` | `1275.4` | `1202.8 / 1249.2 / 1329.8` | `0.971773 ± 0.000228` | `2.72294 / 15.225` | `0.972490` | `0.972197` |
| `d8_full953_C2b_isoparam_mb32` (`Unigram-VE`, `gate_detach=False`, `m=147,267`) | `147.32 / 243.32` | `159.10 / 12534` | `1303.4` | `1229.1 / 1266.0 / 1962.3` | `0.970690 ± 0.000060` | `2.71990 / 15.179` | `0.971333` | `0.971044` |
| **`d8_full953_NG7_isoparam_mb32` (`CoDeM-Ngram`, V8 / `v9diag`, `m=147,267`)** | **`147.32 / 243.32`** | **`159.10 / 12532`** | **`1275.3 / 1274.4`** | **`1202.6 / 1247.6 / 2022.9`** | **`0.970154 ± 0.000127`** (`0.970170`) | **`2.71840 / 15.156`** | **`0.970785`** (`0.970862`) | **`0.970493`** (`0.970565`) |
| $\quad\hookrightarrow$ **3-Bit `NG7-IsoParam + Full L2`** (`31.59 MiB` mod, V8 / `v9diag`) | `31.59 / 127.59` | — | — | — | **`0.970910`** / **`0.970943`** | `2.72052 / 15.188` | **`0.971536`** | **`0.971241`** |
| $\quad\hookrightarrow$ **2-Bit `NG7-IsoParam + Full Combined L2`** (`22.61 MiB` mod, V8 / `v9diag`) | `22.61 / 118.61` | — | — | — | **`0.973259`** / **`0.973287`** | `2.72710 / 15.288` | **`0.973894`** | **`0.973599`** (`0.973668`) |
| **Depth $L=6$ ($d=384$), Full Budget (`953` / `1015` / `1042` steps, `26.35M` backbone)** | | | | | | | | |
| `full_01_plain_d6` (`953` st) | `0.00 / 50.25` | `0.00 / 13733` | `612.5` | `577.6 / — / —` | `1.047419 ± 0.001895` | `2.93490 / 18.820` | `1.048257*` | — |
| `full_05_oe_mfree_110mb` (`953` st, `lr=0.60`) | `110.49 / 160.74` | `110.49 / 13961` | `622.8` | `587.2 / — / —` | `1.024745 ± 0.001494` | `2.87137 / 17.661` | `1.025559*` | — |
| `full_12_eb1_sqrt_glob_110mb` (`953` st, `EB1`) | `110.49 / 160.74` | `110.49 / 13961` | `632.4` | `596.4 / — / —` | `1.022183 ± 0.001519` | `2.86419 / 17.535` | `1.022870` | — |
| `full_10_oe_adam_lr02_110mb` (`953` st, `OE-Adam`) | `110.49 / 160.74` | `331.48 / 14182` | `635.5` | `599.2 / — / —` | `1.020452 ± 0.000780` | `2.85934 / 17.450` | `1.021137` | — |
| `full_20_oe_ember_lr03_110mb` (`953` st, `lr=0.30`) | `110.49 / 160.74` | `112.80 / 13963` | `629.5` | `593.6 / — / —` | `1.019672 ± 0.001731` | `2.85716 / 17.412` | `1.020367` | — |
| `d6_full953_oe_ember_lr02` (`953` st, `lr=0.20`, `m=150,855`) | `110.49 / 160.74` | `112.80 / 13964` | `632.9` | `596.8 / 622.6 / 753.0` | `1.019890 ± 0.001025` | `2.85777 / 17.423` | `1.020597` | `1.020411` |
| `d6_full953_oe_ember_eqtime_1015` (`1015` st, `+6.5%` tok) | `110.49 / 160.74` | `112.80 / 13964` | `627.7` | `630.8 / 652.4 / 757.0` | `1.016640 ± 0.001342` | `2.84866 / 17.265` | `1.017326` | `1.017155` |
| `d6_full1042_oe_ember_eqtime` (`1042` st, `+9.3%` tok, exceeds `NG7` time) | `110.49 / 160.74` | `112.80 / 13963` | `642.7` | `663.3 / 689.7 / 874.9` | `1.015170 ± 0.001227` | `2.84454 / 17.194` | `1.015809` | `1.015691` |
| `d6_full953_FE_basis_gate_isoparam` (`FactorEngram` [6] basis-gate, `m=150,083`) | `110.49 / 160.74` | `113.92 / 14543` | `657.1` | `619.6 / 646.9 / 930.1` | `1.015425 ± 0.001491` | `2.84525 / 17.206` | `1.016151` | `1.015929` |
| `d6_full953_NG6_isoparam` (`953` st, L0-only `CoDeM`, `m=149,699`) | `110.49 / 160.74` | `114.76 / 14736` | `671.7` | `633.4 / 659.7 / 949.1` | `1.013828 ± 0.000776` | `2.84078 / 17.129` | `1.014593` | `1.014372` |
| `d6_full953_C2b_gate_det_isoparam` (`Unigram-VE`, `gate_detach=True`, `m=148,547`) | `110.49 / 160.74` | `117.28 / 14837` | `688.6` | `649.4 / 676.0 / 746.7` | `1.014304 ± 0.001950` | `2.84211 / 17.152` | `1.015056` | `1.014861` |
| `d6_full953_C2b_isoparam` (`Unigram-VE`, `gate_detach=False`, `m=148,547`) | `110.49 / 160.74` | `117.28 / 14742` | `687.2` | `648.1 / 672.6 / 1202.2` | `1.012836 ± 0.001495` | `2.83800 / 17.082` | `1.013538` | `1.013383` |
| **`d6_full953_NG7_isoparam` (`CoDeM-Ngram`, V8 / `v9diag`, `m=148,547`)** | **`110.49 / 160.74`** | **`117.28 / 14741`** | **`686.5 / 687.6`** | **`647.4 / 672.1 / 1362.2`** | **`1.011987 ± 0.000879`** (`1.012214`) | **`2.83562 / 17.041`** | **`1.012689`** (`1.012904`) | **`1.012478`** (`1.012686`) |
| $\quad\hookrightarrow$ **2-Bit `d6_full953_NG7 + Full Combined L2`** (`16.42 MiB` mod, V8 / `v9diag`) | `16.42 / 66.68` | — | — | — | **`1.016027`** / **`1.016243`** | `2.84694 / 17.235` | **`1.016737`** | **`1.016508`** (`1.016702`) |
| **Reduced-Horizon Screening (`240` steps = `125.8M` tokens)** | | | | | | | | |
| `d6_240`: `43_oe_ember_lr02` vs. `V9_240_FE_basis_gate` (`m=150,855` / `150,083`) | `110.49 / 160.74` | `112.80 / 13963` | `626.3 / 646.0` | `144.1 / 162.7 / 310.6` | `1.132978` / `1.127364` | `3.17902` / `3.16326` | `1.138392` / `1.132844` | `— / 1.131903` |
| `d6_240`: `V9_240_FE_basis_postconv` vs. `V9_240_NG6_isoparam` (`m=150,083` / `149,699`) | `110.49 / 160.74` | `113.92 / 14736` | `647.1 / 664.3` | `148.8 / 165.0 / 267.4` | `1.127174` / `1.126285` | `3.16273` / `3.16023` | `1.132697` / `1.131821` | `1.131718 / 1.130663` |
| `d6_240`: **`V9_240_NG7_isoparam`** (`lr=0.20`) / **`NG7_isoparam_lr03`** (`lr=0.30`) | **`110.49 / 160.74`** | **`117.28 / 14741`** | **`681.1 / 680.9`** | **`156.7 / 165.3 / 287.0`** | **`1.125059`** / **`1.124006`** | **`3.15679` / `3.14950`** | **`1.130673`** / **`1.129285*`** | **`1.129614 / —`** |
| `d8_240`: `d8_10_oe_ember_lr02` vs. **`d8_NG7_isoparam_110mb`** (`lr=0.20`) | `147.32 / 243.32` | `149.63 / 12531` | `1203.1 / 1265.8` | `276.7 / — / 457.0` | `1.098381` / **`1.091448`** | `3.08194` / **`3.05827`** | `1.103846` / **`1.096677*`** | — |

*\*Entries marked `*` come from earlier suites (`exp_ngram_suite1_2.json` / `exp_ngram_nextgen.json`) that logged `heldout_bpb` prior to adding `heldout_dedup_bpb` (the matched V9 rerun `V9_240_NG7_isoparam` logs `1.130673` `heldout_dedup_bpb` and `1.129614` `fresh_rg42_dedup_bpb`). Parenthesized numbers in the `NG7` rows show the independent 3-seed V9 diagnostic replication (`d6_full953_NG7_isoparam_v9diag` with `648.4 / 673.7 / 956.4 s` timing, and `d8_full953_NG7_isoparam_mb32_v9diag` with `1201.7 / 1239.9 / 1648.2 s` timing).*

**Key Findings from Table 1:**
1. **Full-Horizon $L=8$ (`953` Steps)**: `d8_full953_NG7_isoparam_mb32` reduces validation BPB by **$-23.799\text{ mBPB}$** vs. `plain` (95% CI $[-24.64, -22.96]\text{ mBPB}$) and by **$-6.273\text{ mBPB}$** vs. `OE-Ember` (CI $[-7.33, -5.22]\text{ mBPB}$; `v9diag` replication: **$-6.257\text{ mBPB}$**, CI $[-7.25, -5.27]\text{ mBPB}$). On the untouched confirmation split `fresh_rg42_dedup`, the paired gain over `OE-Ember` is **$-6.378\text{ mBPB}$** (CI $[-7.58, -5.18]\text{ mBPB}$; `v9diag`: **$-6.306\text{ mBPB}$**, CI $[-7.51, -5.10]\text{ mBPB}$).
2. **Strict Equal-Time Comparisons (`1015`-Step $L=8$ and `1042`-Step $L=6$ `OE-Ember` vs. `953`-Step `CoDeM-Ngram`)**: At $L=8$, `CoDeM-Ngram` requires `1202.6 s` (`v9diag`: `1201.7 s`) steady training time (`1247.6 s` / `1239.9 s` loop time); training `OE-Ember` for `1015` steps (`+6.5%` tokens = $532.15\text{M}$ tokens; `1201.2 s` train / `1234.7 s` loop) equalizes training time within $0.1\%$, yet `953`-step `CoDeM-Ngram` wins all 3 seeds by **$-2.554\text{ mBPB}$** (`v9diag`: **$-2.538\text{ mBPB}$**, CI $[-3.48, -1.60]\text{ mBPB}$; **$-2.647\text{ mBPB}$** on `fresh_rg42_dedup`). At $L=6$, `CoDeM-Ngram` requires `647.4 s` (`v9diag`: `648.4 s`) steady training time (`672.1 s` / `673.7 s` loop time); training `OE-Ember` for `1015` steps (`630.8 s` train / `652.4 s` loop) leaves a $2.6\%$ runtime gap, so we additionally run `d6_full1042_oe_ember_eqtime` for `1042` steps (`+9.3%` tokens = $546.31\text{M}$ tokens; `663.3 s` train / `689.7 s` loop), which strictly exceeds `CoDeM-Ngram` on both pure training time ($+14.9\text{ to }+15.9\text{ s}$) and full loop time ($+16.0\text{ to }+17.6\text{ s}$). `953`-step `CoDeM-Ngram` beats `1015`-step `OE-Ember` by **$-4.653\text{ mBPB}$** and beats `1042`-step `OE-Ember` on all 3 seeds by **$-3.183\text{ mBPB}$** (`v9diag`: **$-2.957\text{ mBPB}$**, CI $[-3.16, -2.75]\text{ mBPB}$; **$-3.214\text{ mBPB}$** on `fresh_rg42_dedup`).
3. **Direct Parameter-Matched Comparison Against `FactorEngram` [6] Basis-Level Gating (`FE_basis_gate_isoparam` and `FE_basis_postconv_isoparam`)**: At `240` steps ($L=6$), single-token basis-gated `FE_basis_gate_isoparam` (`1.127364`) and `FE_basis_postconv_isoparam` (`1.127174`) improve over static $W_{\text{proj}}$ (`P2`, `1.129118`), but replacing their single-token gate with our channel-rolled 3-token unigram summary $c_t$ and dual pre/post gates (`V9_240_NG6_isoparam`, `1.126285`) yields a further **$-1.079\text{ mBPB}$** (CI $[-1.70, -0.45]$; **$-1.240\text{ mBPB}$** on `fresh_rg42_dedup`), and adding `Phrase-VE` (`V9_240_NG7_isoparam`, `1.125059`) widens the lead to **$-2.304\text{ mBPB}$** (`-2.290 mBPB` on `fresh_rg42_dedup`). At full `953`-step horizon, `NG6-IsoParam` beats `FE_basis_gate_isoparam` by **$-1.597\text{ mBPB}$** at $L=6$ (`1.013828` vs. `1.015425`) and **$-1.265\text{ mBPB}$** at $L=8$ (`0.971853` vs. `0.973118`, CI $[-1.68, -0.85]$), while full `CoDeM-Ngram` (`NG7-IsoParam`) beats `FE_basis_gate_isoparam` by **$-3.438\text{ mBPB}$** (`v9diag`: **$-3.211\text{ mBPB}$**, CI $[-4.64, -1.78]$) at $L=6$ and **$-2.964\text{ mBPB}$** (`v9diag`: **$-2.949\text{ mBPB}$**, CI $[-4.24, -1.65]$) at $L=8$.

---

### 4.2 Testing the Collision-Demodulation Hypothesis: Intervention & Common-Modulus Purity Stratification
Over the first `240` training steps ($125.8\text{M}$ tokens), ClimbMix-400B presents $38,055,995$ distinct training trigrams mapped into $m_2 = 150,859$ rows in slice $s=0$, so $100\%$ of hash buckets receive $\ge 4$ distinct trigrams (mean load factor **$252.3$ distinct trigrams/row** at $m=150,855$; **$256.2$ trigrams/row** at $m=148,547$). While replacing the unigram-context gate (`DIAG_NG4_bcd_l0`, `1.126368` val BPB) with a self-gate driven by the hashed vector itself (`DIAG_C1_self_gated`, `1.130042` val BPB) degrades validation loss by **$+3.674\text{ mBPB}$**, we test whether contextual gating specifically mitigates bucket collisions using two controlled analyses (`exp_ngram_v8_controls.json`, `exp_ngram_v9_d6full.json`, `exp_ngram_v9_d8full.json`), summarized in Table 2:
1. **Collision-Rate Intervention ($m=150,855 \to 32,767$, $4.6\times$ Higher Collision Density)**: At $L=6, 240$ steps across 3 seeds (`41, 42, 43`), we compare static `P2` ($W_{\text{proj}}$ only), self-gated `C1` ($g_t^{\text{pre}} = 2\sigma(W_{\text{ctx}}\operatorname{RMSNorm}(e_{\text{raw},t}))$), and unigram-context-gated `NG4` ($g_t^{\text{pre}} = 2\sigma(W_{\text{ctx}}c_t)$) at standard modulus $m=150,855$ (`252.3` distinct trigrams/row) vs. high-collision modulus $m=32,767$ (`1161.3` distinct trigrams/row). Increasing collision load by $4.6\times$ increases `NG4`'s paired improvement over `P2` by **$+73.1\%$**, from **$-2.750\text{ mBPB}$** (`1.126368` vs. `1.129118`, CI $[-3.62, -1.88]\text{ mBPB}$) to **$-4.760\text{ mBPB}$** (`1.142223` vs. `1.146983`, CI $[-5.88, -3.64]\text{ mBPB}$), whereas self-gated `C1` provides zero benefit over `P2` at $m=32,767$ (`1.146994` vs. `1.146983`, $+0.011\text{ mBPB}$). Because shrinking $m$ also weakens the base N-gram predictor (`P2` rises by $+17.87\text{ mBPB}$), this intervention is consistent with context-driven collision mitigation alongside a larger fallback contribution from the unigram-context pathway when table fidelity drops.
2. **Common-Modulus Purity-Stratified Validation Loss (`m_ref = 150,855`, `m2_ref = 150,859`)**: To ensure that models with slightly different parameter-matched table moduli ($m \in \{150,855, 150,083, 149,827, 148,547, 147,267\}$) are evaluated on the **exact same validation token subsets**, `build_trigram_collision_multiplicity` (`run_ngram_v9_fixes.py`) computes bucket purity $\pi(h_{\text{ref}}) = c_{\max}(h_{\text{ref}}) / C_{\text{tot}}(h_{\text{ref}}) \in (0, 1]$ on the fixed reference modulus $m_{2,\text{ref}} = 150,859$ over the first `240` training steps ($125.8\text{M}$ tokens), partitioning the $20.97\text{M}$ validation tokens (`20,944,857` valid trigram positions) into identical high-purity (`2,362,103` tokens, $11.28\%$, $\pi \ge 0.50$), mid-purity (`9,701,579` tokens, $46.32\%$, $0.15 \le \pi < 0.50$), and low-purity (`8,881,175` tokens, $42.40\%$, $\pi < 0.15$) bins across all models. As shown in Table 2B, `CoDeM-Ngram`'s paired BPB reduction over `OE-Ember` increases monotonically from high-purity to low-purity (heavily collided) buckets at both $L=6$ (**$-5.917 \to -7.674 \to -8.190\text{ mBPB}$**) and $L=8$ (**$-5.002 \to -6.221 \to -6.664\text{ mBPB}$**), and exhibits the same monotonic gradient against `FE_basis_gate_isoparam` (**$-2.746 \to -3.201 \to -3.359\text{ mBPB}$** at $L=6$; **$-2.614 \to -2.930 \to -3.066\text{ mBPB}$** at $L=8$).

| Table 2A: Collision-Rate Intervention ($L=6, 240$ steps, 3 seeds) | Modulus $m$ | Trigrams / Row | `P2` ($W_{\text{proj}}$ only) | `C1` (Self-Gated) | `NG4` (Unigram Gate) | `NG4 - P2` ($\Delta$ mBPB, 95% CI) | `NG4 - C1` ($\Delta$ mBPB) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **Standard Table (`DIAG_*`, `110.5 MiB` tables)** | `150,855` | `252.3` | `1.129118 ± 0.006266` | `1.130042 ± 0.005477` | **`1.126368 ± 0.005936`** | **`-2.750`** (`[-3.62, -1.88]`) | **`-3.674`** |
| **High-Collision Intervention (`COLL32K_*`, `24.0 MiB` tables)** | `32,767` | `1161.3` ($4.6\times$) | `1.146983 ± 0.008500` | `1.146994 ± 0.007903` | **`1.142223 ± 0.007813`** | **`-4.760`** (`[-5.88, -3.64]`, **$+73\%$**) | **`-4.771`** |

| Table 2B: Common-Modulus (`m2_ref = 150,859`) Purity-Stratified Val BPB (`953` st, 3 seeds) | High Purity $\pi(h_{\text{ref}}) \ge 0.50$ (`2,362,103` tok, `11.3%`) | Mid Purity $0.15 \le \pi(h_{\text{ref}}) < 0.50$ (`9,701,579` tok, `46.3%`) | Low Purity / High Collision $\pi(h_{\text{ref}}) < 0.15$ (`8,881,175` tok, `42.4%`) |
| :--- | :---: | :---: | :---: |
| **$L=6$**: `d6_full953_oe_ember_lr02` $\to$ **`d6_full953_NG7_isoparam_v9diag`** | `0.991899` $\to$ **`0.985981`** (**`-5.917 ± 0.429`**, CI `[-6.98, -4.85]`) | `1.015680` $\to$ **`1.008005`** (**`-7.674 ± 0.341`**, CI `[-8.52, -6.83]`) | `1.032737` $\to$ **`1.024547`** (**`-8.190 ± 0.162`**, CI `[-8.59, -7.79]`) |
| **$L=6$**: `d6_full953_FE_basis_gate_isoparam` $\to$ **`d6_full953_NG7_isoparam_v9diag`** | `0.988727` $\to$ **`0.985981`** (**`-2.746 ± 0.511`**, CI `[-4.02, -1.48]`) | `1.011206` $\to$ **`1.008005`** (**`-3.201 ± 0.552`**, CI `[-4.57, -1.83]`) | `1.027906` $\to$ **`1.024547`** (**`-3.359 ± 0.631`**, CI `[-4.93, -1.79]`) |
| **$L=8$**: `d8_full953_oe_ember_lr02_mb32` $\to$ **`d8_full953_NG7_isoparam_mb32_v9diag`** | `0.953585` $\to$ **`0.948583`** (**`-5.002 ± 0.759`**, CI `[-6.89, -3.12]`) | `0.972523` $\to$ **`0.966302`** (**`-6.221 ± 0.349`**, CI `[-7.09, -5.35]`) | `0.987433` $\to$ **`0.980769`** (**`-6.664 ± 0.361`**, CI `[-7.56, -5.77]`) |
| **$L=8$**: `d8_full953_FE_basis_gate_isoparam_mb32` $\to$ **`d8_full953_NG7_isoparam_mb32_v9diag`** | `0.951197` $\to$ **`0.948583`** (**`-2.614 ± 0.655`**, CI `[-4.24, -0.99]`) | `0.969233` $\to$ **`0.966302`** (**`-2.930 ± 0.497`**, CI `[-4.17, -1.69]`) | `0.983835` $\to$ **`0.980769`** (**`-3.066 ± 0.511`**, CI `[-4.34, -1.80]`) |

---

### 4.3 $2\times 2$ `Phrase-VE` Factorial, Gate-Detachment Control, and Horizon Trajectory
When comparing `Phrase-VE` (`NG7`, which routes the demodulated phrase vector $\tilde{u}_t$ into $W_{\text{pve}}^{(l)}$) against `Unigram-VE` (which routes the 3-token unigram summary $c_t^{\text{raw}}$ into $W_{\text{pve}}^{(l)}$), four factors must be controlled: (i) parameter parity ($m=148,547$ at $L=6$; $m=147,267$ at $L=8$), (ii) $W_{\text{pve}}^{(l)}$ initialization ($0$ vs. $0.25I_d$), (iii) `VE` gradient flow (`pve_detach=True` vs. `False`), and (iv) Layer-0 context-gate stop-gradient (`gate_detach_wte=True` in Eq. 8). In Version 8 (`V8NextGenGPT`), `DIAG_C2b_isoparam` and `d6/d8_full953_C2b_isoparam` set `pve_detach=False` on $c_t^{\text{raw}}$ before Eq. (8), which simultaneously allowed Layer-0 context-gate gradients to update `wte`, whereas `NG7` detached `wte` at the Layer-0 gate. In Version 9 (`V9NextGenGPT`), `V9_240_C2d_isoparam`, `V9_240_C2b_gate_det_isoparam`, `d6_full953_C2b_gate_det_isoparam`, and `d8_full953_C2b_gate_det_isoparam_mb32` hold `gate_detach_wte=True` strictly fixed across all arms while varying only the `VE` pathway (`Table 3A` and `Table 3B`).

| Table 3A: $L=6, 240$-Step `VE` Factorial (3 seeds: `41, 42, 43`) | Modulus $m$ | `VE` Source | $W_{\text{pve}}^{(l)}$ Init / `VE` Grad | L0 Gate Detach | Val BPB ($\pm$ SD) | `heldout_dedup` BPB | `fresh_rg42` BPB | $\Delta$ Val vs. `V9_240_NG6` (`1.126285`) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| `V9_240_NG6_isoparam` (No `VE` pathway) | `149,699` | None | — / — | `True` | `1.126285 ± 0.006025` | `1.131821` | `1.130663` | `0.000 mBPB` |
| `C2c_phrase_ve_zero_detach` | `150,855` | Phrase $\tilde{u}_t$ | $0$ / Detached | `True` | `1.127079 ± 0.006241` | `1.132541` | — | `+0.794 mBPB` |
| `C2_conv_bcd_l0_unigram_ve` | `150,855` | Unigram $c_t^{\text{raw}}$ | $0$ / Detached | `True` | `1.125866 ± 0.005756` | `1.131053*` | — | `-0.419 mBPB` |
| `V9_240_C2d_isoparam` (Isoparam detached `Unigram-VE`) | `148,547` | Unigram $c_t^{\text{raw}}$ | $0.25\,I_d$ / Detached | `True` | `1.126168 ± 0.005949` | `1.131637` | `1.130715` | `-0.117 mBPB` |
| `V9_240_C2b_gate_det_isoparam` (Strictly matched `Unigram-VE`) | `148,547` | Unigram $c_t^{\text{raw}}$ | $0.25\,I_d$ / Joint | `True` | `1.125843 ± 0.005459` | `1.131290` | `1.130385` | `-0.442 mBPB` |
| **`V9_240_NG7_isoparam` (`CoDeM-Ngram`, `Phrase-VE`)** | **`148,547`** | **Phrase $\tilde{u}_t$** | **$0.25\,I_d$ / Joint** | **`True`** | **`1.125059 ± 0.006607`** | **`1.130673`** | **`1.129614`** | **`-1.226 mBPB`** (`-0.784` vs. `C2b_gate_det`) |
| `DIAG_C2b_isoparam` (`Unigram-VE` + attached L0 gate) | `148,547` | Unigram $c_t^{\text{raw}}$ | $0.25\,I_d$ / Joint | `False` | `1.123921 ± 0.005018` | `1.129497` | — | `-2.364 mBPB` |

| Table 3B: Full-Horizon `val_curve` Trajectory (`3` seeds, $m$ matched) | Step `190` (`99.6M` tok) | Step `380` (`199.2M` tok) | Step `570` (`298.8M` tok) | Step `760` (`398.5M` tok) | Final Step `953` (`val` $20.97\text{M}$) | Final `fresh_rg42_dedup` ($20.97\text{M}$) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **$L=6$ (`gate_detach=True` fixed)**: `d6_full953_C2b_gate_det_isoparam` | `1.178244` | `1.103965` | `1.063556` | `1.027352` | `1.014304 ± 0.001950` | `1.014861 ± 0.001969` |
| **$L=6$ (`gate_detach=True` fixed)**: **`d6_full953_NG7_isoparam_v9diag`** | **`1.176791`** | **`1.103043`** | **`1.061030`** | **`1.025077`** | **`1.012214 ± 0.001221`** | **`1.012686 ± 0.001221`** |
| $\quad\hookrightarrow$ **Paired Diff `NG7_v9diag - C2b_gate_det` ($L=6$, mBPB)** | **`-1.453`** | **`-0.922`** (3/3 seeds) | **`-2.526`** (3/3 seeds) | **`-2.275`** (3/3 seeds) | **`-2.090 mBPB`** (`CI [-3.93, -0.25]`) | **`-2.176 mBPB`** (`CI [-4.05, -0.30]`) |
| **$L=6$ (`gate_detach=False` in `C2b`)**: `NG7_isoparam - C2b_isoparam` | `+0.984` (`C2b` leads) | `+0.108` (tie) | **`-1.145`** (`NG7` overtakes) | **`-0.873`** | **`-0.849 mBPB`** (3/3 seeds) | **`-0.904 mBPB`** (3/3 seeds) |
| **$L=8$ (`gate_detach=True` fixed)**: `d8_full953_C2b_gate_det_isoparam_mb32` | **`1.145208`** | `1.070834` | `1.025992` | `0.986818` | `0.971773 ± 0.000228` | `0.972197 ± 0.000237` |
| **$L=8$ (`gate_detach=True` fixed)**: **`d8_full953_NG7_isoparam_mb32_v9diag`** | `1.145423` | **`1.069593`** | **`1.024669`** | **`0.985096`** | **`0.970170 ± 0.000163`** | **`0.970565 ± 0.000161`** |
| $\quad\hookrightarrow$ **Paired Diff `NG7_v9diag - C2b_gate_det` ($L=8$, mBPB)** | `+0.215` (tie) | **`-1.241`** (3/3 seeds) | **`-1.323`** (3/3 seeds) | **`-1.722`** (3/3 seeds) | **`-1.603 mBPB`** (`CI [-2.10, -1.10]`) | **`-1.631 mBPB`** (`CI [-2.11, -1.15]`) |
| **$L=8$ (`gate_detach=False` in `C2b`)**: `NG7_isoparam - C2b_isoparam_mb32` | `+0.995` (`C2b` leads) | `+0.832` (`C2b` leads) | **`-0.398`** (`NG7` overtakes) | **`-0.381`** | **`-0.536 mBPB`** (`CI [-0.71, -0.37]`) | **`-0.551 mBPB`** (`CI [-0.72, -0.38]`) |

**Key Findings from Table 3A and Table 3B (`Figure 1c`):**
1. **Strictly Matched `Phrase-VE` vs. `Unigram-VE` (`gate_detach_wte=True` Held Fixed)**: When `gate_detach_wte=True` is held constant across all arms, routing the demodulated N-gram phrase vector $\tilde{u}_t$ into `VE` (`NG7`) outperforms routing the 3-token unigram summary (`C2b_gate_det`) at `240` steps by **$-0.784\text{ mBPB}$** (`1.125059` vs. `1.125843`; **$-0.771\text{ mBPB}$** on `fresh_rg42_dedup`), and wins every seed at full `953`-step horizon by **$-2.090\text{ mBPB}$** at $L=6$ (`1.012214` vs. `1.014304`, CI $[-3.93, -0.25]\text{ mBPB}$; **$-2.316\text{ mBPB}$** vs. V8 `NG7`) and **$-1.603\text{ mBPB}$** at $L=8$ (`0.970170` vs. `0.971773`, CI $[-2.10, -1.10]\text{ mBPB}$; **$-1.619\text{ mBPB}$** vs. V8 `NG7`, CI $[-1.90, -1.34]\text{ mBPB}$).
2. **Why `C2b` (`gate_detach_wte=False`) Leads Early (`190–240` Steps) and `NG7` Overtakes by Step `570`**: Each training token updates $1$ row of the $8,192$-row unigram table `wte` and $1$ row in each of the $4$ N-gram slice tables ($m \approx 148,547$ rows per slice, $594,200$ total N-gram rows across the 4 slices). Thus, each individual N-gram slice table has **$m / V \approx 18.1\times$ lower per-row update frequency** than `wte` (and total N-gram rows outnumber `wte` rows by $4m / V \approx 72.5\times$). When `C2b` additionally allows Layer-0 gate gradients to flow into `wte` (`gate_detach_wte=False`), the high-density unigram embeddings adapt rapidly during the first `190–240` steps (`1.123921` vs. `1.125059`). However, once N-gram hash buckets accumulate sufficient token exposure (`380–570` steps = $199\text{M}\text{--}299\text{M}$ tokens), `Phrase-VE` (`NG7`) overtakes even `C2b`, finishing ahead on all 3 seeds at step `953` by **$-0.849\text{ mBPB}$** at $L=6$ (**$-0.904\text{ mBPB}$** on `fresh_rg42_dedup`) and **$-0.536\text{ mBPB}$** at $L=8$ (CI $[-0.71, -0.37]\text{ mBPB}$).
3. **Two-Pass Gradient Sensitivity Diagnostic (`run_cross_layer_grad_diag`)**: In `run_ngram_nextgen.py`, the cosine similarity between the Layer-0 table gradient $g_{\text{L0}}$ (standard forward pass) and the `VE`-branch gradient $g_{\text{VE}}$ obtained in a second forward pass with an unprojected identity `VE` skip is $\cos(g_{\text{L0}}, g_{\text{VE}}) = -0.0018$ at step 0 and $-0.0091$ at step 120. Because the second pass alters the forward graph, near-zero cosine indicates that unadapted direct injection perturbs deeper attention values in a direction uncorrelated with Layer-0 gradient descent; the training ablation `P2 -> NG1 -> NG2` (`1.129370 -> 1.128145 -> 1.127625` BPB on Seed 42) vs. unadapted direct `VE` sharing (`B2_superpose`, `1.132830` BPB) directly confirms the benefit of learned projections $W_{\text{pve}}^{(l)}$.

---

### 4.4 Low-Bit Quantization and Multi-Readout Ridge-Regularized L2 Calibration
Table 4 reports the complete post-training quantization (PTQ) and ridge-regularized L2 calibration decomposition from `ngram_authoritative_summary.json`, separating both **structural constraint spaces** (`Diag`, `Block-Diag`, `Full`) and **readout pathway targets** (`Full L0-Only` folding $(M^*)^\top$ into $W_{\text{proj}}$ only, `Full VE-Only` folding $(M^*)^\top$ into $\{W_{\text{pve}}^{(l)}\}$ only, and `Full Combined` folding $(M^*)^\top$ into both pathways simultaneously), alongside full-precision feature cross-slice correlation $R^2(\tilde{e}^{(0)}, \tilde{e}^{(1)})$ (from $\Sigma_{ee}$) and quantization-error cross-slice correlation $R^2(d^{(0)}, d^{(1)})$ (from $\Sigma_{dd}$, where $d_t = \tilde{q}_t - \tilde{e}_t$).

| Table 4: Post-Training Quantization & Multi-Readout L2 Calibration (`3` seeds) | BF16 Val BPB | 3-Bit `Std` / `Full` Val | 2-Bit `Std` Val | 2-Bit `Diag` ($\Delta$ mBPB / L0,VE MSE $\downarrow$) | 2-Bit `Block-Diag` ($\Delta$ mBPB / L0,VE MSE $\downarrow$) | 2-Bit `Full L0-Only` ($\Delta$ mBPB / L0 MSE $\downarrow$) | 2-Bit `Full VE-Only` ($\Delta$ mBPB / VE MSE $\downarrow$) | 2-Bit `Full Combined` ($\Delta$ mBPB / L0,VE MSE $\downarrow$) | Feature $R^2_{\tilde{e}}$ / Error $R^2_d$ ($s_0,s_1$) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **$L=6, 240$ st**: `DIAG_P2_learned_proj` (L0 only) | `1.129118` | `1.129950 / 1.129930` | `1.132737` | `1.132727` (`-0.010` / `0.08%, —`) | `1.132618` (`-0.120` / `2.16%, —`) | `1.132475` (`-0.262` / `3.00%`) | — | `1.132475` (`-0.262` / `3.00%, —`) | `0.07118` / — |
| **$L=6, 240$ st**: `V9_240_FE_basis_gate` (L0 only) | `1.127364` | `1.128376 / 1.128289` | `1.131897` | `1.131542` (`-0.354` / `1.44%, —`) | `1.131371` (`-0.526` / `3.53%, —`) | `1.131150` (`-0.747` / `4.62%`) | — | `1.131150` (`-0.747` / `4.62%, —`) | `0.08517` / `0.01179` |
| **$L=6, 240$ st**: `V9_240_NG6_isoparam` (L0 only) | `1.126285` | `1.127303 / 1.127218` | `1.130752` | `1.130442` (`-0.310` / `1.66%, —`) | `1.130270` (`-0.482` / `3.76%, —`) | `1.130058` (`-0.694` / `4.91%`) | — | `1.130058` (`-0.694` / `4.91%, —`) | `0.09413` / `0.01237` |
| **$L=6, 240$ st**: **`V9_240_NG7_isoparam`** (`L0 + VE`) | **`1.125059`** | **`1.126128 / 1.126032`** | `1.129928` | `1.129535` (`-0.393` / `1.44%, 1.07%`) | `1.129321` (`-0.607` / `3.63%, 2.97%`) | `1.129189` (`-0.739` / `4.94%`) | `1.129770` (`-0.158` / `4.32%`) | **`1.129078`** (**`-0.851`** / **`4.94%, 4.32%`**) | `0.08555` / `0.01390` |
| **$L=6, 953$ st**: `d6_full953_C2b_isoparam` (L0 only) | `1.012836` | `1.013800 / 1.013742` | `1.017178` | `1.016952` (`-0.227` / `1.62%, —`) | `1.016714` (`-0.464` / `3.42%, —`) | `1.016526` (`-0.652` / `4.69%`) | — | `1.016526` (`-0.652` / `4.69%, —`) | `0.08119` / — |
| **$L=6, 953$ st**: **`d6_full953_NG7_isoparam`** (V8) | **`1.011987`** | **`1.013015 / 1.012954`** | `1.016880` | `1.016554` (`-0.326` / `1.78%, —`) | `1.016281` (`-0.599` / `3.81%, —`) | — | — | **`1.016027`** (**`-0.853`** / **`5.35%, —`**) | `0.07265` / — |
| **$L=6, 953$ st**: **`d6_full953_NG7_v9diag`** (`L0 + VE`) | **`1.012214`** | **`1.013236 / 1.013173`** | `1.017116` | `1.016772` (`-0.345` / `1.82%, 1.10%`) | `1.016495` (`-0.622` / `3.84%, 2.78%`) | **`1.016432`** (**`-0.685`** / **`5.32%`**) | **`1.016864`** (**`-0.253`** / **`4.23%`**) | **`1.016243`** (**`-0.874`** / **`5.32%, 4.23%`**) | `0.07279` / `0.01569` |
| **$L=8, 953$ st**: `d8_full953_C2b_isoparam_mb32` (L0 only) | `0.970690` | `0.971403 / 0.971362` | `0.973990` | `0.973791` (`-0.199` / `2.04%, —`) | `0.973562` (`-0.428` / `4.04%, —`) | `0.973408` (`-0.582` / `5.35%`) | — | `0.973408` (`-0.582` / `5.35%, —`) | `0.08977` / — |
| **$L=8, 953$ st**: **`d8_full953_NG7_isoparam_mb32`** (V8) | **`0.970154`** | **`0.970971 / 0.970910`** | `0.974179` | `0.973814` (`-0.365` / `2.05%, —`) | `0.973497` (`-0.682` / `4.32%, —`) | — | — | **`0.973259`** (**`-0.920`** / **`5.99%, —`**) | `0.08166` / — |
| **$L=8, 953$ st**: **`d8_full953_NG7_mb32_v9diag`** (`L0 + VE`) | **`0.970170`** | **`0.970990 / 0.970928`** | `0.974168` | `0.973832` (`-0.335` / `2.10%, 1.22%`) | `0.973525` (`-0.643` / `4.37%, 3.18%`) | **`0.973505`** (**`-0.663`** / **`6.04%`**) | **`0.973877`** (**`-0.291`** / **`4.83%`**) | **`0.973287`** (**`-0.880`** / **`6.04%, 4.83%`**) | `0.08143` / `0.01938` |

**Key Findings from Table 4 (`Figure 1d`):**
1. **Readout Pathway Isolation (`Full L0-Only` vs. `Full VE-Only` vs. `Full Combined`)**: On full-horizon $L=6$ `d6_full953_NG7_isoparam_v9diag` (`2-bit Std = 1.017116`), folding $(M_{\text{full}}^*)^\top$ into $W_{\text{proj}}$ alone (`Full L0-Only`) lowers validation BPB by **$-0.685 \pm 0.087\text{ mBPB}$** ($78.4\%$ of the combined gain; Layer-0 readout MSE $-5.32\%$, `VE` MSE $0\%$), folding $(M_{\text{full}}^*)^\top$ into $\{W_{\text{pve}}^{(l)}\}$ alone (`Full VE-Only`) lowers validation BPB by **$-0.253 \pm 0.035\text{ mBPB}$** (Layer-0 MSE $0\%$, `VE` readout MSE $-4.23\%$), and folding into both readouts (`Full Combined`) achieves **$-0.874 \pm 0.062\text{ mBPB}$** (95% CI $[-1.03, -0.72]\text{ mBPB}$, nearly additive vs. $-0.685 - 0.253 = -0.938\text{ mBPB}$). On full-horizon $L=8$ `d8_full953_NG7_isoparam_mb32_v9diag` (`2-bit Std = 0.974168`), `Full L0-Only` yields **$-0.663 \pm 0.101\text{ mBPB}$** ($75.3\%$ of combined gain; Layer-0 MSE $-6.04\%$), `Full VE-Only` yields **$-0.291 \pm 0.008\text{ mBPB}$** (`VE` MSE $-4.83\%$), and `Full Combined` yields **$-0.880 \pm 0.105\text{ mBPB}$** (CI $[-1.14, -0.62]\text{ mBPB}$).
2. **Structural Decomposition (`Diag` vs. `Block-Diag` vs. `Full`) and Feature vs. Error Cross-Slice $R^2$**: Across `d8_full953_NG7_isoparam_mb32_v9diag`, per-channel `Diag` rescaling accounts for **$-0.335\text{ mBPB}$** ($38.1\%$ of total gain), within-slice `Block-Diag` calibration adds **$-0.308\text{ mBPB}$** ($35.0\%$), and cross-slice `Full Combined` calibration adds **$-0.237\text{ mBPB}$** ($26.9\%$). Why does off-diagonal cross-slice calibration help when table quantization is independent per row? First, the shared unigram context gate $g_t^{\text{pre}}$ and local convolution correlate the full-precision pre-projection slice features ($R^2(\tilde{e}^{(0)}, \tilde{e}^{(1)}) = 0.07279$ at $L=6$ and $0.08143$ at $L=8$; between bigram and trigram slices $R^2(\tilde{e}^{(0)}, \tilde{e}^{(2)}) \approx 0.011$), allowing a linear combination of neighbor slices to acts as a shrinkage estimator for the target slice $\tilde{e}_t$. Second, because two hash slices of the same N-gram order index the same token sequence with moduli $m$ and $m+2$, their pre-projection quantization errors $d_t = \tilde{q}_t - \tilde{e}_t$ also exhibit non-zero within-order correlation ($R^2(d^{(0)}, d^{(1)}) = 0.01569$ at $L=6$, range $[0.0112, 0.0214]$; $0.01938$ at $L=8$, range $[0.0165, 0.0236]$).
3. **Comparison Against Quantization-Aware Training (QAT) and Uncompressed 16-Bit Baselines**: Straight-through 2-bit QAT baselines (`full_06_lowbit_oe_2bit_150k` `1.024996` and `full_08_core_qste_2bit_110mb` `1.026211`) require $331.5\text{ MiB}$ of FP32 master-weight training state yet trail `2-bit CoDeM-Ngram + Full Combined L2` (`1.016027` / `1.016243` at $L=6$) by **$-8.75\text{ to }-8.97\text{ mBPB}$**. Furthermore, simulated 2-bit `CoDeM-Ngram + Full Combined L2` ($16.42\text{ MiB}$ module / $66.68\text{ MiB}$ whole model at $L=6$; $22.61\text{ MiB}$ module / $118.61\text{ MiB}$ whole model at $L=8$) outperforms **uncompressed 16-bit `OE-Ember`** by **$-3.647\text{ to }-3.863\text{ mBPB}$** at $L=6$ and **$-3.139\text{ to }-3.168\text{ mBPB}$** at $L=8$ (CI $[-4.33, -1.95]\text{ mBPB}$).

---

## 5. Conclusion and Limitations
We have shown that replacing static narrow-slice N-gram table injection with **Context-Demodulated Multi-Pathway Phrase Memory (`CoDeM-Ngram` / `NG7-IsoParam`)** and **Multi-Readout Ridge-Regularized L2 Calibration** yields consistent, parameter-matched improvements across both $L=6$ ($d=384$) and full-budget $L=8$ ($d=512$) Transformers on ClimbMix-400B, outperforming `OE-Ember` at equal steps and equal training time, parameter-matched `FactorEngram` [6] basis-level gating (`FE_basis_gate_isoparam`), and strictly gate-detachment-matched `Unigram-VE` (`C2b_gate_det_isoparam`). Three limitations bound our claims: (1) our collision-rate intervention ($m=150,855 \to 32,767$) and common-modulus purity stratification ($m_{2,\text{ref}}=150,859$) support context-driven collision mitigation alongside unigram-context fallback when table fidelity drops, but do not imply that trained hash rows are exact linear superpositions of isolated phrase embeddings; (2) ridge-regularized L2 readout folding minimizes pre-projection reconstruction error rather than exact token cross-entropy; and (3) low-bit memory figures reflect exact dequantized Lloyd-Max simulation (`quantize_tables_lloyd_max`) rather than custom bit-packed integer CUDA kernels.

---

## References
1. H. Huang et al., *Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling*, arXiv:2501.16975, 2025.
2. D. Yu et al., *Scaling Embedding Layers in Language Models (SCONE)*, arXiv:2502.01637, 2025.
3. *Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models (Engram)*, arXiv:2601.07372, 2026.
4. *TN-gram: Tensorized N-gram Embeddings for Language Modeling*, arXiv:2606.08347, 2026.
5. K. Shah, *Ember: A Kilobyte-State Optimizer for Token Interfaces*, arXiv:2607.01455, 2026.
6. B. Yang et al., *FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models*, arXiv:2609.35578, September 2026.
7. S. Chung and W. Du, *Low-Bit N-Gram Embeddings for Language Models*, arXiv:2610.31646, 2026.
8. N. Shazeer and M. Stern, *Adafactor: Adaptive Learning Rates with Sublinear Memory Cost*, ICML / arXiv:1804.04235, 2018.
9. J. Li et al., *Frequency-aware SGD for Efficient Embedding Learning*, arXiv:2110.04844, 2021.
10. *Rows, bits, and exposure: where the capacity of hashed n-gram input tables goes, and what it costs in bytes*, rStation `general_pretraining_6/v4`, 2026.
11. *Hashed N-gram Table Rows Need Little Optimizer History, and the Tested Forward or Averaged Writes Recover Only a Fifth of a Gradient Write*, rStation `general_pretraining_11/v3`, 2026.
12. *Separating initialization, coefficient learning, and lookup cost in shared value embeddings*, rStation `general_pretraining_9/v1`, 2026.
13. *Cross-Family Composition and Checkpoint Diagnostics of Moment-Free Hashed N-Gram Input Tables Under Pre-Polar Equilibrated Muon Pretraining*, rStation `general_pretraining_16/v5`, 2026.

---

## Appendix A: Additional Architectural Progression & Literature Baselines ($L=6$, `240` & `953` Steps)

| Arm ID & Description (`ngram_authoritative_summary.json`) | Steps | Seeds | Mod / Whole BF16 (`MiB`) | Val BPB (`240 / 953`) | `heldout_dedup` (`240 / 953`) | 3-Bit `Std` Val (`240 / 953`) | 2-Bit `Std` Val (`240 / 953`) |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| `01_plain_d6` / `full_01_plain_d6` (No N-gram table) | `240 / 953` | `3 / 3` | `0.00 / 50.25` | `1.166445 / 1.047419` | `— / 1.048257*` | — | — |
| `02_oe_mfree_110mb` / `full_05_oe_mfree_110mb` (`OE-MFree`) | `240 / 953` | `3 / 3` | `110.49 / 160.74` | `1.140305 / 1.024745` | `— / 1.025559*` | — | — |
| `04_lowbit_oe_2bit_150k` / `full_06_lowbit_oe_2bit` (2-bit QAT) | `240 / 953` | `3 / 3` | `14.96 / 65.21` | `1.138417 / 1.024996` | `— / 1.025802*` | — | `1.138417 / 1.024996` |
| `L3_tngram_faithful_r640` / `full_17_tngram_r640` (`TN-gram` [4]) | `240 / 953` | `3 / 3` | `107.52 / 157.78` | `1.142076 / 1.023739` | `1.147590 / 1.024400` | — | — |
| `L4_engram_faithful_110mb` / `full_16_engram_110mb` (`Engram` [3]) | `240 / 953` | `3 / 3` | `111.62 / 161.87` | `1.141655 / 1.023127` | `1.147223 / 1.023835` | — | — |
| `44_eb1_sqrt_glob` / `full_12_eb1_sqrt_glob` (`EB1`, `lr=0.10`) | `240 / 953` | `3 / 3` | `110.49 / 160.74` | `1.135550 / 1.022183` | `1.141089 / 1.022870` | `1.136359 / 1.023004` | `1.139053 / 1.025861` |
| `41_oe_adam_lr02` / `full_10_oe_adam_lr02` (`OE-Adam`, `lr=0.20`) | `240 / 953` | `3 / 3` | `110.49 / 160.74` | `1.134879 / 1.020452` | `1.140367 / 1.021137` | `1.135614 / 1.021179` | `1.137994 / 1.023672` |
| `43_oe_ember_lr02` / `d6_full953_oe_ember_lr02` (`OE-Ember`, `lr=0.20`) | `240 / 953` | `3 / 3` | `110.49 / 160.74` | `1.132978 / 1.019890` | `1.138392 / 1.020597` | `1.133745 / 1.020667` | `1.136328 / 1.023309` |
| `B3_ember_cyclic04` / `full_14_b3_ember_cyclic04` (Fixed cyclic `VE`) | `240 / 953` | `3 / 3` | `110.49 / 160.74` | `1.131443 / 1.019126` | `1.136871 / 1.019848` | `1.132185 / 1.019849` | `1.134662 / 1.022411` |
| `P2_ember_learned_proj` / `full_19_ember_learned_proj` ($W_{\text{proj}}$) | `240 / 953` | `3 / 3` | `110.77 / 161.02` | `1.129370 / 1.017638` | `1.134834 / 1.018392` | `1.130191 / 1.018426` | `1.132990 / 1.021205` |
| `V9_240_FE_basis_gate` / `d6_full953_FE_basis_gate_isoparam` (`FactorEngram` [6] basis-gate) | `240 / 953` | `3 / 3` | `110.49 / 160.74` | `1.127364 / 1.015425` | `1.132844 / 1.016151` | `1.128376 / —` | `1.131897 / —` (`Full: 1.131150`) |
| `V9_240_NG6_isoparam` / `d6_full953_NG6_isoparam` (L0-only `CoDeM`, isoparam) | `240 / 953` | `3 / 3` | `110.49 / 160.74` | `1.126285 / 1.013828` | `1.131821 / 1.014593` | `1.127303 / —` | `1.130752 / —` (`Full: 1.130058`) |
| **`V9_240_NG7_isoparam` / `d6_full953_NG7_isoparam` (`CoDeM-Ngram`)** | `240 / 953` | `3 / 3` | `110.49 / 160.74` | **`1.125059 / 1.011987`** | **`1.130673 / 1.012689`** | **`1.126128 / 1.013015`** | **`1.129928 / 1.016880` (`Full: 1.129078 / 1.016027`)** |
