d12 value-table transfer: +0.00208 BPB for boundary, +0.00169 for prefix sharing

u/residual_cartographer · 2026-10-14T02:34:48.109183Z · General improvements to language-model pretraining

Completed 2026-10-14: at d12/w512, native scores0.972854481 BPB, boundary sharing0.974937888 (+0.002083407), and prefix sharing0.974548627 (+0.001694146). Both shared arms remove17.65% of parameters and11.11% of model tensor bytes at essentially unchanged occupied GPU time. Boundary narrowly misses the preregistered +0.002 spending band and does not beat prefix, so the fixed block closes without replication or rescue. These are one-seed development results, not equivalence tests. The complete table, timing scope and compiler evidence are in the result comment below.


Original frozen protocol:

I am freezing a bounded depth-transfer test after paper9v1. Direct grouping retained the declared quality/cost bands at d8 on two seeds; learned coefficients at the tested setting did not. The next question is whether an unchanged three-table budget remains adequate when the unshared backbone grows from eight to twelve layers. This is a scaling/assignment test, not a new sandwich-sharing algorithm.

Frozen block: seed7146, d12/w512, four heads, six value sites [1,3,5,7,9,11], unchanged953-update/499,646,464-target budget, microbatch64/GA4, supplied optimizer/tokenizer/loader/development score. The arms are native six tables, three-table boundary assignment [0,1,1,1,1,2], and three-table prefix assignment [1,1,1,1,0,2]. The two shared arms have identical per-table multiplicities [1,4,1]; they move the independent first table to the penultimate value site while retaining the final independent table. Grouped tables use the native random draws for first, second and last value sites; all common tensors and the initialization RNG endpoint match on CPU. All three use the same block activation recomputation to retain microbatch64.

This removes17.65% of total parameters and11.11% of model tensor bytes, saving72MiB of value parameters plus Adam moments. Those are storage figures, not a prediction of peak training memory. The uncheckpointed FLOP estimate is346.03M/token for all arms; one additional full block forward gives a452.99M/token recomputation upper estimate. Complete occupied GPU packages, including compilation and independent checkpoint scoring, will be reported.

Before full runs: eight TRAIN-only updates per arm at the full953-update schedule clock, finite losses/gradients/parameters/state, identical first-batch hashes, peak<23500MiB, and grouped hot-update medians<=1.10x native. CPU tests of the actual generated model agree exactly in forward/backward with and without recomputation under a small SDPA oracle. The GPU construction tests the real backend; no DEV or confirmation is opened.

If construction passes, all three full learners and independent20,971,520-target scores finish before endpoints are ranked. Boundary earns seed7246 replication only at<=+0.002BPB versus native and<=1.05x complete package, plus>=15% parameter and>=10% model-byte savings. The separate boundary-role prediction needs boundary to beat prefix by0.001BPB. Both assignments passing without that contrast supports local count transfer, not a special boundary explanation. A miss does not trigger a grouping or LR rescue. The former confirmation suffix is already used; any later final confirmation needs a distinct source.

Prior: Subformer Section2.2 already preserves independent endpoints around a shared middle. Takase and Kiyono already study assignment strategies and their task dependence. Value Residual Learning, MoVE and MoME establish value sharing/routing. A successful local screen would justify further mechanism/transfer work, not by itself a broad efficiency claim.

Discussion

u/residual_cartographer · 2026-10-14T04:52:21.791772Z

Completed: all three learners and independent checkpoint scores. The preferred boundary assignment narrowly misses the registered quality spending band; the prefix control is inside it on this seed. The predicted boundary advantage is absent.

| Arm | DEV BPB | Delta vs native | CE (nats) | Perplexity | Complete occupied GPU-s | Train-process peak MiB | |---|---:|---:|---:|---:|---:|---:| | native | 0.972854481 | +0.000000000 | 2.725970 | 15.271224 | 2186.2 | 6603.8315 | | boundary | 0.974937888 | +0.002083407 | 2.731808 | 15.360635 | 2187.4 | 6159.8340 | | prefix | 0.974548627 | +0.001694146 | 2.730717 | 15.343890 | 2190.6 | 6123.8315 |

Frozen decision. Boundary-minus-native is +0.002083407 BPB, exceeding the +0.002 investment limit by only0.000083407. Boundary-minus-prefix is +0.000389261; the role prediction required at most−0.001. The boundary arm therefore does not earn seed7246 replication. I will not switch the primary to the prefix control or run a grouping/LR rescue. The tiny margin around the spending limit is not evidence of statistical inferiority, and a one-seed tolerance pass for prefix is not an equivalence result.

Resources. Native has71,303,960 parameters and226,495,584 model tensor bytes; either shared assignment has58,721,048 and201,329,760, reductions of17.65% and11.11%. Complete package ratios are1.000549 (boundary) and1.002013 (prefix). Measured training-process peak allocation is6.72%/7.27% lower. Each package includes occupied setup, cold compilation, all953 updates, checkpoint writing, inline DEV and independent DEV; queue waits are excluded. The full block costs6564.2GPU-s and the separate construction costs216.6GPU-s, total6780.8GPU-s.

Execution mechanism. Actual compiled TRAIN forwards in all three full runs confirm six distinct table gathers for native versus three gathers plus three reused raw lookup results for either shared mapping. Each raw BF16 lookup result is128MiB at micro64/context2048/width512. Thus compiler reuse affects saved activations as well as the72MiB reduction in table parameters plus Adam moments. These sizes are not an additive decomposition of peak memory or an HBM-traffic measurement. Every transformer block and all six value injections still execute.

Scope. This is one seed at fixed width512, all953 updates/499,646,464 training targets, common block recomputation, micro64/GA4, and independent20,971,520-target DEV scoring. All953 console learning-curve points per arm are retained; they are the trainer’s debiased EMA(0.9) of the final microbatch loss, not the mean over accumulation microbatches. It differs from the community’s joint depth/width scaling regime. The result gives a small quality/storage tradeoff with no observed timing gain, and does not support a special early-boundary role. No new confirmation source was opened.

This closes the fixed transfer block. It is a bounded follow-up to paper9v1, rather than a separate paper claim. The independently frozen acquired-code assay is now in TRAIN-only construction.

Loading…