Delayed weight-change credit without a backward phase in language-model pretraining

Problem and contribution

Weight Maximization (WM) gives a stochastic unit a reward derived from the change of its outgoing weights [1]. Its ordinary implementation evaluates those rewards in reverse layer order. Chung's eligibility-trace extension instead permits every layer to learn from old neighboring updates, removing the same-example reverse dependency. That change introduces delayed credit, historical state and temporal interference. Practical transformers add shared parameters, attention across tokens, nonlinear normalization and residual paths.

We investigate a complete transformer adaptation in which neither global nor local activation derivatives are used. Forward sampling writes action-specific records; a clocked queue delivers old raw receiver updates to their matching source actions. Finite local observations supply the nonlinear interface features. A second graph assigns residual writers to the terminal reader through forward-computed residual features. This is a changed credit graph, not an accelerated implementation of the original graph.

The completed evidence identifies a useful intervention: value an old packet with the receiver's current weight instead of its origin-time snapshot. In the native 50.3 M-parameter model, this reduces development bits per byte (BPB) by 0.11961 and 0.12657 in two seeds at 134.2 M presented targets. It also improves the original 33-tick local graph by 0.11872 BPB in one seed. The effect cannot yet be attributed solely to receiver freshness: using current weights adds an exact displacement term that can include feedback from the packet's own earlier update. At 499.65 M presented targets, the live/snapshot difference narrows to 0.01930 BPB and full live learning improves over a live root/head control by 0.02861 BPB. The retained gain is modest; longer training does not preserve the larger early advantage. At fixed short exposure, the live advantage survives increased depth (16 layers) and width (768), while absolute quality changes little. These finite-size controls support useful hidden learning but do not establish bounded statistical sample requirements.

A subsequent terminal intervention always measures the target and one sampled non-target logit. At equal four-probe effort it improves full-model BPB by 0.11408/0.11228 and lexical-control BPB by 0.09600/0.09905 in seeds 46/47. The smaller 0.01808/0.01324 interactions support an additional hidden-learning benefit. Most of the measured gain is shared with the lexical learner. At the full presented-target budget the full/root sampler gains shrink to 0.03807/0.04430 BPB, and the interaction reverses to+0.00623. The revised full learner scores 2.06373 BPB versus 2.07003 for its lexical control: the short-run hidden interaction is not persistent. Standard stratification and its conditional covariance identity are ingredients, not a new theorem or causal explanation.

The method has a substantial quality and time gap to backpropagation (BP). The completed short runs do not demonstrate statistical scalability or a practical speed advantage. Their contribution is a concrete credit mechanism without a backward phase, useful learning relative to its matched controls, and an explicit account of what remains unresolved. A fixed number of forward lanes is a computation property, not evidence that the sample requirement stays bounded.

Relation to Chung's algorithms and perturbation learning

WM replaces a common scalar reward by an individual first-order outgoing norm reward, $v_i^\top\Delta v_i$ [1]. Original goal alignment requires a mixed derivative identity; its small-outgoing-weight argument does not establish alignment for normalized transformers. With $M$ layers, the trace kernel for unit $i$ at depth $d(i)$ is

$$ \lambda_i(k)= \begin{cases} 0,&k<M-d(i),\\ (1-\lambda)\lambda^{k-(M-d(i))},&k\geq M-d(i). \end{cases} $$

The shift handles the earliest possible arrival of credit; the tail represents lasting action effects. The paper's Appendix recurrence omits the common $(1-\lambda)$ factor, a scale convention that implementations must disclose. The paper reports poorer learning with traces in its reinforcement-learning experiments and identifies temporal interference as a difficulty.

Our current packet queues preserve action identity and old-update dependence but have no exponential tail. They deliver each action's packet once at its declared age. They are not a faithful reproduction of the full original trace algorithm. The residual graph further changes the receivers and utilities.

MAP propagation conditions hidden-state inference on an output before applying local score updates [2]. Its Gaussian stationary-point relation uses local derivative chains at settled states; we do not implement that inference or inherit its identity. Unbiased WM expresses a Bernoulli counterfactual gap through integration and downstream importance weighting [3]. The conditional distribution and importance-ratio requirements do not transfer to our sparse Gaussian transformer policies. Structured coordination motivates treating exploration dependencies as part of the policy [4]; our tested correlated writer variant did not improve learning. None of these original theorems establishes unbiasedness of the present reward-normalized learner.

Dust uses tokenwise node perturbations, direct loss rewards and local attention-internal perturbations [5]. Its nominal population excludes head and local-attention probes. Boost and Cut use intermediate readouts, structured directions and, for Cut, frozen attention routes [6]. The newer Noise identity-path record measures a full-width Hadamard cycle at the top stream and transports its current credit through residual scalars. Its source then computes current MLP credit, the attention projection contraction and attention-internal credit before the same optimizer step. In particular, value credit is a transposed attention-map contraction, and the readout cycle has one direction per stream coordinate. These are substantive local credit and optimizer innovations, but they do not satisfy the fixed-probe and no-local-backward criterion studied here. We do not claim residual feedback itself as new. Our route instead delivers old raw weight-update packets to forward-written action records, with fixed observation lanes and no current-credit-directed probes or current-origin reverse chain.

Our common-sequence-reward control is not Dust, and no native faithful Dust/Boost/Cut/identity-path comparison is claimed here. Published benchmark times also differ in hardware, corpus, vocabulary, batch, initialization and optimizer; the identity-path record uses ANVIL II on block matrices and Adam elsewhere. Those comparisons remain required before a competitive quality--compute claim. We inspected the pinned source, rather than treating absence of autograd alone as absence of backward computation.

The concurrent Co-WM-S study analyzes sibling decoupling through micro-horizon parameter-space contractions and reports promising reduced-setting language model results [7]. Its version 3 explicitly defines forward-only as excluding a global reverse depth/autograd sweep while retaining local elementwise/directional derivative computations. The source orders current readout credit, MLP credit, norm pullback and attention credit before the optimizer step. Finite forward-function differences evaluated in the downstream-credit direction retain that local adjoint dependency when the Jacobian is symmetric. It addresses a weaker absence-of-global-reverse-sweep criterion than the complete absence of backward computation considered here. Its conditional covariance analysis is relevant related work, not evidence that our adaptation has the same properties. We credit the earlier public discussion of residual credit routes [8]; no peer implementation was adopted.

Model, actions and terminal observations

Inference uses the task's unchanged decoder: eight layers, width 512, rotary embeddings, QK normalization, squared-ReLU MLPs, value embeddings, learned residual/root mixing, context 2048, vocabulary 8192 and 50, 332, 176 parameters. Training retains microbatch 64 and the supplied ClimbMix corpus/tokenizer. Model and optimizer provenance are the task's pinned autoresearch-derived supplement [9]. The first seed 46 dense BP reproduction is preserved as the strong task reference.

For each sequence choose $K=8$ stratified context positions. The same position list is used for all 64 sequences. At selected affine outputs, embeddings and scalar policies, sample Gaussian innovations with $\sigma=0.1$. The ordinary forward action is rounded to its normal dtype. An affine actor stores its input $x_e$, innovation $\epsilon_e$, and forward score $s_e=\epsilon_e/\sigma$. Its eventual raw parameter contribution is an outer product of its score-weighted utility and $x_e$. Embedding updates use a scatter; scalar policies use their own sampled scalar actions. No attention, normalization or activation derivative is evaluated to obtain those updates.

Rounding the sampled action does not by itself invalidate its latent Gaussian score: for a fixed rounding map and continuous mean, the density-score identity still applies to rewards of the rounded action. The native affine mean is itself computed in reduced precision, however, and shared parameters also have unsampled deterministic uses. The saved-input score outer product therefore does not automatically give the gradient of the complete quantized program. The action-dependent delayed utilities and their transformations add separate limitations to any objective-gradient claim.

At each selected label, sample vocabulary coordinate $J$ from the equal mixture of its target label and the current model distribution. With $a$ the raw readout logits and $\ell$ ordinary softcapped cross-entropy, write

$$ C_e=-\frac{\ell(a+\sigma e_J)-\ell(a-\sigma e_J)}{2\sigma q(J)}e_J. $$

The two losses are evaluated by a stable forward partition-function update at the one changed logit. There are exactly two scalar probes per selected label, independently of vocabulary size. Conditional on fixed logits, this samples the vector of coordinate central secants with the correct inverse probability; that vector is not exactly the analytic loss gradient at finite $\sigma$. No analytic softmax error is used. The head's actual raw contribution has form $C_e z_e^\top$, with $z_e$ its saved forward input.

Old receiver packets and the two credit graphs

For a linear receiver, contracting a raw packet $C_e x_e^\top$ with its weight $W$ yields column progress $x_e\odot(C_e^\top W)$. Our local-reset utility replaces $x_e$ by a declared finite forward feature $f_{i,e}$ belonging to source actor $i$:

$$ R_{i,e}=f_{i,e}\odot(C_e^\top W),\qquad Q_{i,e}=s_{i,e}\odot\operatorname{clip}_{[-5,5]} \left(\frac{R_{i,e}}{\max(\operatorname{RMS}(R_i),10^{-6})}\right). $$

This is a per-event contrast utility of an actual old raw update. It is not the full batch-aggregated post-Muon outgoing norm change. RMS normalization and clipping change its expectation; the raw-update identity must not be confused with an unbiased optimizer displacement.

Finite features are differences between ordinary local forward values after resetting one actor family to its pre-innovation value. MLP features evaluate squared-ReLU normally. Attention features evaluate fixed query/key/value/gate/ value-embedding reset lanes. Query utilities group by head; key/value/gate utilities include the declared selected-query aggregation. The joint-family resets are not separate counterfactuals for every neuron or every source token.

In the local graph, each MLP output writer credits the next block's readers; attention internals credit their output writer; MLP expansion credits its output writer. The longest delay is $4L+1$, hence33 ticks at $L=8$. Actors consume only previous-tick packets of their own origin. All ready sets are frozen before new contributions are published. There is no within-origin reverse depth traversal and no BP-trained critic.

In the residual graph, root embedding, residual writers and mixing scalars consume the old terminal packet at age 1; internal actors consume their old writer packet at age 2. Two forward carrier vectors accumulate realized innovations along the residual additions. Prefix log-magnitudes, signs and zero counts recover subsequent realized scalar transport without a reverse suffix pass. Downstream branch outputs remain fixed at observed values, so these carriers do not represent full network counterfactual reruns.

Direct output guidance is established prior. Direct Feedback Alignment [10] projects the current output error through fixed random matrices and multiplies local activation derivatives; it avoids a reverse hidden-layer sequence but retains computations excluded here. Its discussion also considers temporal activation differences. Our residual graph instead uses delayed raw update packets, live learned receiver weights and forward-written action features. This distinction does not make direct feedback, label-conditioned feature learning or parallel hidden updates new in themselves.

At the terminal residual $x$ and normalized value $z$, the observed scale $\rho=\langle z,x\rangle/\langle x,x\rangle$ gives a direct feature $D_{i,e}=\rho\alpha_i\Delta a_{i,e}$. Two ordinary reset normalizations also measure a shared nonlinear remainder. The direct rule uses only the owned feature; the shared control broadcasts the remainder to actors in that carrier. The shared control performs worse in every tested shape. The selected direct rule is consequently a deliberately incomplete forward effect, not an exact normalization pullback.

The residual graph has bounded age independent of depth, but changes credit routes and utility. Comparing it with the local graph does not identify the causal effect of delay alone.

The clock is part of the method. At tick $t$, execution is:

1. Run the new stochastic forward world with current weights; save its behavior scores, inputs and forward finite features. Form its terminal packet from scalar forward-loss observations.
2. Freeze the ready actors of every older origin. A receiver packet is eligible only if it was published strictly before $t$. Contract those old raw updates with the current receiver weights, construct utilities, and install the corresponding score-times-utility parameter fields.
3. Publish the newly constructed old-origin packets only after their ready actors have all read. Emit the new origin's terminal packet and install its head field. Newly published packets cannot be consumed at this tick.
4. Apply the declared per-matrix optimizer and clear its input fields. Advance the clock; after the last example, drain the remaining queues.

One origin's actual packet deliveries at ages 0, 1, 2. Counts 1, 33, 40 refer to parameter-family actor records, not individual neurons. New microbatches execute ordinary forward worlds while older origins wait; the diagram describes dependencies rather than hardware-parallel speed.

Thus a matrix contraction with an old packet is an outgoing weight-progress reward, not an activation adjoint passed through a nonlinear interface. Forward features are already fixed before the packet exists; no downstream credit chooses their probe direction. The source audit checks these dependencies in addition to disabling autograd. The repeated per-actor contractions and queue work are counted even though no reverse-depth phase is executed.

Live receiver utility changes the learning game

Chung's original trace formulation already values old updates with current receiver weights [1]. We do not claim to invent that convention. The following comparison tests restoring it after our origin-snapshot adaptation.

Snapshot mode saves $W_e$ at action generation. Live mode retains an alias and uses $W_t$ when the old packet arrives. The scores, origin inputs, finite features and packet coefficient are unchanged. In real arithmetic the exact utility difference is

$$ R_{i,e}^{\mathrm{live}}-R_{i,e}^{\mathrm{snapshot}} = f_{i,e}\odot\big(C_e^\top(W_t-W_e)\big). $$

This identity does not require a gradient interpretation. It shows why a live reader is more than a cheaper way to store a snapshot. Write $\Delta=W_t-W_e$. In a simplified one-step SGD receiver with $\Delta=\eta B^{-1}\sum_b C_bx_b^\top$, the projected change is

$$ C_e^\top\Delta=\frac{\eta}{B}\sum_b\langle C_e,C_b\rangle x_b^\top. $$

The self term contains $\|C_e\|^2x_e$. It survives reversing the signs of all receiver coefficients, so its existence cannot be interpreted simply as better tracking of the desired terminal direction. Intervening examples add other terms. This is an algebraic decomposition for the stated SGD simplification; Muon/AdamW, state-dependent coefficients, normalization, clipping and rounding prevent an automatic native expectation theorem. The actual clock narrows the interpretation: terminal head credit reads exactly after its own origin-associated head update, with no unrelated head update in between. The age 2 writer reads include one preceding-origin update and their own update, with no subsequent displacement. Therefore a pre-own-head control would duplicate the origin snapshot on a common trajectory. The native head displacement still depends on AdamW history, so it is not caused solely by the new packet. Attribution to beneficial label agreement, self-reinforcement or adaptive optimizer history remains unresolved.

Conditional variance motivation and its limits

Fix an exogenous receiver vector $g$, scale $\rho$ and transport $\alpha$, independently of a standard Gaussian vector $\epsilon$. A direct feature $\alpha\rho\sigma\epsilon_i$ gives coordinate field $Q_i=\alpha\rho g_i\epsilon_i^2$. Therefore

$$ \mathbb E[Q]=\alpha\rho g,\qquad \operatorname{tr}\operatorname{Cov}(Q)=2\alpha^2\rho^2\|g\|^2. $$

A broadcast scalar utility $\langle g,\epsilon\rangle$ instead gives $\mathbb E[\epsilon\epsilon^\top g]=g$ and covariance trace $(w+1)\|g\|^2$ at width $w$. These standard conditional identities motivate individual utility. They are not a new theorem and do not describe the native online learner: $g$, $\rho$ and receiver weights depend on actions, and the actual rule normalizes/clips utility. Restricted field-moment advantages have already failed to predict learning rankings in our prior controls.

Keeping action identity is an explicit construction choice: each utility is paired with the originating score and feature. That avoids accidental mixing of unmatched records; it does not establish that the resulting deep learning signal is informative. The native controls below test that separate question.

Terminal sampling intervention: conditional prediction

The following calculation motivated the completed seed 46 comparison below. Conditional on the entire sampled forward world and its labels, let $d_j$ be the coordinate central secant of the softcapped CE. Write $a=-d_y e_y$, draw $J\ne y$ from the target-excluded model distribution $q'_j=p_j/(1-p_y)$, and define $B=-d_J e_J/q'_J$, $b=\mathbb E[B]$. The stratified packet $X=a+B$ always measures the target and one non-target. It uses four scalar forward-loss observations per selected label. The code computes the rest mass through a target-excluded log-sum-exp rather than subtracting a possibly rounded unit target probability.

The equal-probe control $Y$ averages two independent balanced mixture draws, each equal to $2a$ or $2B$ with probability one half. Both packets have conditional mean $-d$, and total covariance gives

$$ \operatorname{Cov}(Y)-\operatorname{Cov}(X) =\tfrac12(a-b)(a-b)^\top\succeq0. $$

This ordering survives a fixed linear map conditional on the forward world, including the raw head outer product. It does not survive arbitrary clipping, normalization, adaptive optimization or endogenous live-reader feedback as a general guarantee. In particular, for $G=E^{-1}\sum_e C_e z_e^\top$ with independent terminal draws across events,

$$ \mathbb E[G_{Y,jh}^2]-\mathbb E[G_{X,jh}^2] =\frac{1}{2E^2}\sum_e d_{ej}^2 z_{eh}^2. $$

At a common AdamW state this changes the expected next raw second moment by $1-\beta_2$ times the displayed quantity. It does not order the expectation of the actual ratio of first and second moments. Thus even a head improvement would test the sampler inside the chosen optimizer, not an optimizer-independent covariance effect. Later live credit reads that optimizer's own displacement.

In the uncapped infinitesimal CE limit, $\mathbb E\|X\|^2=2(1-p_y)^2$ and $\mathbb E\|X\|^2/\|d\|^2\le2$. For finite radius $\sigma$ and the monotone 1-Lipschitz cap, an absolute bound remains $\mathbb E\|X\|^2\le2e^{2\sigma}(1-p_y)^2$. There is no uniform native relative bound: saturate the target's cap while keeping the non-target logits equal and unsaturated. Its target secant tends to zero; if each other secant tends to $c>0$, the second-moment ratio tends to $V-1$. This distinguishes useful absolute control from a vocabulary-independent relative-noise guarantee. Neither statement establishes hidden-neuron or depth scalability.

Finite radius also has a restricted mean guarantee in real arithmetic. For the actual cap $s(a)=15\tanh(a/15)$ and radius0.1, standard logarithmic derivative bounds and the symmetric Taylor remainder give $|d_j-g_j|\le0.003130|g_j|$, where $g_j$ is the mathematical per-event raw-logit derivative. The learner never evaluates that derivative. This componentwise bound is independent of vocabulary size but excludes floating-point saturation and rounding. It does not imply a relative bound after parameter contributions cancel, nor does it address hidden-action bias, normalization, adaptive updates or learning. Both terminal samplers share this finite-radius mean.

The frozen six-arm panel crosses full/root learning with the incumbent two-probe target/model proposal, balanced two-iid four-probe sampling, and the stratified four-probe packet. Only the latter pair matches proposal and scalar probe count. The full/root interaction addresses additional useful hidden learning; lexical gains alone cannot establish it. Every added categorical draw, vocabulary reduction and scatter remains charged in time and operation counts. These standard sampling identities are attributed ingredients, not the claimed new learning mechanism.

Completed terminal-stratification screen

The six-arm seed 46 panel used the native d 8/w 512 model, microbatch 64, context 2048, K 8, sigma 0.1, 1024 GA 1 origins and schedule 256. Every arm received 134, 217, 728 presented targets and 524, 288 selected labels. All six TRAIN and six fresh official DEV stages completed before comparative readout; the frozen source, stream, packet, checkpoint and cost checks passed.

| Scope / terminal sampler | DEV BPB | Ordinary CE | PPL | TRAIN s | Peak GiB | |---|---:|---:|---:|---:|---:| | Full / incumbent |2.285049665|6.402784343|603.523116|420.381|5.534| | Full / two iid |2.247411015|6.297319609|543.114199|406.576|5.534| | Full / stratified |2.133328132|5.977655617|394.514390|406.322|5.534| | Root/head / incumbent |2.371093366|6.643881625|768.070576|366.549|5.025| | Root/head / two iid |2.270370425|6.361652632|579.202775|366.424|5.025| | Root/head / stratified |2.174368982|6.092653429|442.594247|366.220|5.025|

At equal four-probe effort, stratified minus iid is−0.114082883 BPB for full learning and−0.096001443 BPB for root/head. Full minus root/head is−0.022959410 with iid sampling and−0.041040850 with stratification, giving a favorable interaction of−0.018081440 BPB. Most of the gain is shared with the lexical control; the interaction supports a smaller additional hidden benefit. This single seed cannot establish robustness, identify variance as the mediator, or distinguish the sampler's interaction with optimizer state from other effects.

Against the incumbent two-probe rule, stratification improves full/root BPB by 0.151721533/0.196724384. This changes both proposal and probe count. The full-minus-root margin is actually larger for the incumbent (0.086043701 BPB), so the total improvement must not be described as uniformly better hidden credit assignment. The separate short K 8 BP reference remains 0.178335637 BPB better than full stratification and has a shorter measured loop.

Each arm executes 1024 full worlds plus four startup checks and fixed local observation lanes. The iid and stratified arms each use 2, 097, 152 scalar CE probes, compared with 1, 048, 576 for the incumbent. Each balanced arm evaluates 524, 288 vocabulary vectors for each of full logsumexp, target-excluded logsumexp and target-excluded softmax. IID uses 1, 048, 576 categoricals and the same number of mixture coins; stratification uses 524, 288 categoricals and no coins. These operations, scatters and copies are included in observed time; equal scalar-probe counts do not imply identical total work. Full/root packet ages are 2/1, with zero backward computations or current-origin reverse sweeps. Age 1/2 actors receive 1/2 final zero-rate deliveries, respectively.

TRAIN and DEV occupy 2383.3 and 232.4 GPU seconds, totaling 2615.7 s. The separate construction cost 142.4 GPU seconds, including twelve forward numerical cases and six 32-origin native runs. Partial matrix-product arithmetic is 8.68123 PFLOPs per full arm and 8.64385 per root arm, excluding nonlinear and reduction work, RNG, scatter, optimizer, movement, compilation, checks and DEV. The small iid versus stratified timing differences do not establish a speed advantage. Order and compiler state were fixed; concurrent CPU-only lossless archiving and older-checkpoint hashing are recorded as possible timing interference.

The useful equal-probe gain earned an exact-source seed 47 replication of the four iid/stratified full/root arms. All four TRAIN and four fresh DEV stages completed before comparative readout; the frozen analyzers passed.

| Seed47 arm | DEV BPB | Ordinary CE | PPL | TRAIN s | Peak GiB | |---|---:|---:|---:|---:|---:| | Full / two iid |2.246001033|6.293368794|540.972688|404.274|5.534| | Full / stratified |2.133716964|5.978745137|394.944456|405.029|5.534| | Root/head / two iid |2.273515465|6.370465135|584.329558|365.709|5.025| | Root/head / stratified |2.174468595|6.092932547|442.717800|366.305|5.025|

Seed 47 stratified-minus-iid is−0.112284069 BPB for full learning/−0.099046870 root; full-minus-root is−0.027514431 iid/−0.040751631 stratified. Its interaction is−0.013237199 BPB. The descriptive two-seed means are−0.113183476 full,−0.097524157 root and −0.015659320 interaction. Two seeds do not give a precise population uncertainty interval. The common terminal improvement and smaller hidden benefit replicate; causal attribution to conditional variance alone remains unsupported.

The replication occupies 1574.4 TRAIN+154.4 DEV=1728.8 GPU seconds. Its identical source reuses the 142.4 s construction above; that preparation is not counted a second time. Probe, vector, packet, age and arithmetic counts match the four corresponding seed 46 arms. Two brief CPU-only comparator construction checks ran concurrently and are disclosed; no CPU compression occurred. Small timing differences are not evidence of a speed effect.

Full-budget terminal comparison

All four TRAIN and four fresh DEV stages completed before comparative readout. The source-identical seed 46 extension uses 3812 GA 1 origins, schedule 953, 499, 646, 464 presented targets and 1, 951, 744 selected labels per arm. Thus exposure and warmdown change together. The final shard remains untouched.

| Arm | DEV BPB | Ordinary CE | PPL | TRAIN s | Peak GiB | |---|---:|---:|---:|---:|---:| |full_iid2|2.101801205|5.889316131|361.158215|1497.414|5.534| |full_stratified2|2.063733349|5.782648746|324.617883|1500.362|5.534| |root_iid2|2.114334596|5.924435104|374.067067|1354.983|5.025| |root_stratified2|2.070033074|5.800300781|330.398923|1355.045|5.025|

Stratified-minus-iid is−0.038067856 BPB for full learning and−0.044301522 root. Full-minus-root is−0.012533391 iid and−0.006299724 stratified. The interaction reverses to +0.006233666, compared with−0.018081440/−0.013237199 in short seeds 46/47. Terminal improvement persists, but the favorable hidden interaction does not. Nearly all remaining benefit is shared with the lexical control. This one-seed length comparison does not isolate a variance, optimizer or timing mechanism.

Each arm executes 3812 full worlds+4 startup checks and 7, 806, 976 scalar CEprobes. IIDuses 3, 903, 488 categoricals+coins; strata 1, 951, 744 categoricals/no coins. Both perform 1, 951, 744 vocabulary-vector evaluations each for full logsumexp, excluded logsumexp and excluded softmax. Maxpacketage 2 full/1 root; zero backward/localVJP/ same-origin reverse sweeps; all deliveries drain with 1/2 zero-rate tail updates.

New occupancy 5742.2 TRAIN+154.8 DEV=5897.0 GPU-s. Prior 142.4 construction is reused. Partial matrix products 32.317231222 PFLOPs/full and 32.178063693/root carry the same exclusions as the short panel. Rule/optimizer/audit/hash-check/drain costs, copy/probe/trace ledgers and all four models are retained. SharedGPU, fixedorder and disclosed CPUpreparation preclude fine speed claims; no compression ran.

The separate longK 8 BP is 1.658700750 BPB/1358.724 TRAIN seconds, and strongdenseBP is 0.999626863/1121.552 s with full supervision at the same presented-token budget. The candidate still has a substantial quality gap and no speed result. Retain stratification as a terminal ingredient; develop additional old-branch credit rather than another terminal sampler grid. The completed three-arm branch comparison below tests one bounded extension.

Complete terminal comparisons at 134 Mtargets in two seeds and 500 Mtargets in one seed. The terminal gain persists but shrinks, while the favorable hidden interaction reverses. At the full budget both separate sparse BP and strong dense BP remain more accurate and faster in the measured training loops. The length comparison also changes warmdown.

A bounded old-MLP branch did not improve learning

The direct residual route holds later branch outputs fixed. We tested one additional consequence: an attention output writer receives its cached age 1 head utility together with its own block's old expansion-update utility atage 3. Its attention internals then learn atage 4. The two utilities are separately RMS-normalized/clipped and averaged without another bound. A matched delay control performs the same extra contraction but substitutes the head utility for the expansion utility. Direct credit retainsage 1/2. All use the stratified terminal, native model, seed 46, 1024 origins and the same schedule/optimizer.

This is a different local reward, with no inherited WM alignment guarantee. It addsLexpansion contractions perorigin, not an extra full world or forward reset lane. Ready sets remain frozen before new packets. Actual installed histories, live receiver identity, age 1 caching, all-family displacements, inference/RNG and drains passed native 16-origin construction in every arm. Clock instrumentation was disabled for learning. All 3 TRAIN+3 DEV completed before comparative readout; no loss-dependent horizon or arm was selected.

| Arm | DEV BPB | Ordinary CE | PPL | TRAIN s | Peak GiB | |---|---:|---:|---:|---:|---:| |direct|2.133429664|5.977940113|394.626644|404.955|5.534| |delayed_direct|2.132899650|5.976454997|394.041013|405.361|6.001| |delayed_mlp|2.133876386|5.979191841|395.120919|405.730|6.001|

The added utility is 0.000976735 BPB worse than matched delay and 0.000446722 worse than direct. The delay control is 0.000530014 better than direct. These small, unreplicated differences do not support useful information from the tested mixture or establish delay as the principal defect. The direct endpoint differs by 0.000101532 from the earlier same-seed run; CUDA embedding reductions are not bitwise reproducible. We do not promote or tune this mixture from these results.

Peak memory increases by 0.46713 GiB. Each run uses 2, 097, 152 scalar CEprobes, 1024 full worlds+4 startup checks; delayed arms add 8, 192 expansion contractions. Partialmatrixcost is 8.681229 PFLOPs/direct versus 8.690025/delayed with the same exclusions as above. Actual occupancy 1239.8 TRAIN+116.8 DEV=1356.6 GPU-s, plus 53.7 newconstruction; 142.4 terminalconstruction is reused. All three models, complete costs/clocks and sources are retained. This is evidence about one changed utility, not a general failure of delayed branch credit.

Completed native evidence

The first table uses 1024 GA 1 origins, 134, 217, 728 presented targets, 524, 288 selected labels, micro 64/context 2048 and the same task tokenizer/development scorer. All stages in each admitted comparison finished before comparative readout. Development evaluation uses 20, 971, 520 target tokens. BPB, CE and perplexity exclude special tokens consistently; complete values and TRAIN curves are in the numerical supplement.

| Graph and seed | Snapshot BPB | Live BPB | Live minus snapshot | |---|---:|---:|---:| | Residual,46 |2.404746058|2.285132475|-0.119613583| | Residual,47 |2.408387662|2.281815713|-0.126571949| | Local,47 |2.608172785|2.489451793|-0.118720993|

The two residual seeds give descriptive replication, not a precise uncertainty interval. The local result is one seed at a different graph. No endpoint was chosen from an intermediate validation sweep.

Completed within-graph comparisons: current receiver weights lower development BPB in two age-2 residual seeds and one age-33 local-graph seed. Peak allocation falls, but training-time differences do not establish a speed advantage.

At seed 47, root/head-only snapshot/live score 2.376730926/2.372542637, a 0.004188290 BPB difference. Full live beats live root/head by 0.090726924 BPB, whereas full snapshot loses to snapshot root/head by 0.031656736. The reader-by-scope contrast is-0.122383660 BPB. Zero-initialized frozen output writers make the root/head control a current-token lexical model. This control rules out attributing the full gain solely to that lexical learner, but does not show that every hidden layer receives correct contextual credit.

An inference intervention replaces prior-token attention by self-only attention. At seed 46 it raises live BPB from 2.285132475 to 2.342606177 and snapshot BPB from 2.404746058 to 2.854002603; the lexical checkpoint is invariant. Both full models therefore use prior context under this intervention. Greater snapshot damage is not evidence that live credit has superior context attribution: the intervention changes the inference distribution and does not locate the source of the training gain.

The earlier snapshot residual rule beats common-sequence reward at all five tested shapes, including d 16/w 512 and d 8/w 768. At those larger shapes its BPB is 2.421006375 and 2.422281986, respectively, worse in absolute value than native seed 46, while the relative advantages narrow to 0.122907203 and 0.110505717 BPB. This finite-size transfer is not evidence of increasing returns to capacity. Those earlier snapshot controls used a different observation implementation and are historical references, not controls for the live panel below.

Several natural repairs were negative: adding the shared normalization remainder, the tested correlated writer exploration, source-owned attention features with nearby paired exposure, and freezing residual mixing. These results constrain the tested formulations; they do not rule out coordination, better attention credit or other parameterizations in general. The integrated numerical record preserves their complete matched comparisons.

Costs and strong comparisons

Native residual snapshot/live at seed 47 takes 404.082/404.612 training seconds and peaks at 5.843/5.534 GiB. The corresponding local graph takes 571.613/581.747 seconds and peaks at 10.577/8.854 GiB. Live reading reduces record-copy traffic from 177, 324, 687, 360 to 57, 063, 505, 920 bytes in these studies, a 67.82% reduction. Its lower copied bytes and allocation do not establish a speed advantage. Run order and compilation cache state were not randomized for a small timing claim.

The residual implementation executes one full model world per origin, fixed local finite observation lanes and two scalar terminal probes per selected label. The historical local implementation executes an additional unused clean diagnostic world, charged in its measured time. Removing that world has passed CPU replay and a native 48-origin paired construction with the complete 33-tick drain, for both snapshot and live readers under deterministic algorithms. All recorded raw fields, parameters, RNG states and deliveries match exactly. Default CUDA raw-field bitwise reproducibility fails even for an identical- setting replay (first mismatch at tick 11), while model/RNG/delivery records match. Thus the deterministic result does not establish default bitwise parity. The complete construction costs 887.8 occupiedGPU seconds including all failed attempts; extensive hashing and sequential warm-cache order preclude a speed claim. This is numerical construction, not additional learning evidence. There are zero backward calls, local VJPs or same-origin reverse sweeps in either candidate. Trace updates, utility contractions, copying, optimizer work, synchronization, construction checks and end-of-run drain remain actual costs.

Separate native K 8 BP scores 1.954992495 BPB in 369.97 training seconds with 20.122 GiB peak memory at the short budget. It is better and faster than the candidate. The strong dense BP control scores 0.999626863 BPB at the full 499, 646, 464-target budget in 1121.55 training seconds with 22.27 GiB peak memory. That dense control uses 953 GA 4 updates and full labels; it must not be replaced by the weaker K 8 comparator in a parity claim.

The completed full-budget panel extends the fixed seed 46 rule to 3812 GA 1 origins, 499, 646, 464 presented targets and 1, 951, 744 selected labels, with schedule 953. All four TRAIN and four fresh DEV stages completed before comparative readout; the frozen source, stream, event, operation and checkpoint checks passed.

| Arm | DEV BPB | Ordinary-token CE | PPL | TRAIN seconds | Peak GiB | |---|---:|---:|---:|---:|---:| | Full live |2.149924500|6.024159186|413.293992|1503.603|5.534| | Full snapshot |2.169226898|6.078245139|436.262942|1510.701|5.843| | Root/head live |2.178535925|6.104329341|447.792225|1378.348|5.025| | Separate K8 BP |1.658700750|4.647734077|104.348272|1358.724|20.122|

The live advantage over snapshot narrows from 0.119613583 BPB at the short seed 46 budget to 0.019302398 BPB. Full live still beats live lexical root/head by 0.028611425 BPB. Exposure and warmdown schedule change together, so this does not identify a cause for the smaller gain. Separate sparse BP is 0.491223749 BPB better and has a shorter measured loop; the dense BP reference remains much stronger. Candidate peak allocation is lower. Neither the small within-reader timing difference nor partial arithmetic establishes a speed advantage.

Total newly occupied GPU time is 5946.9 s, including 5787.3 TRAIN and 159.6 DEV; allocator waiting is excluded. Partial affine/attention/local-contraction arithmetic is 32.317/32.317/32.178/95.944 PFLOPs for the four rows, excluding nonlinearities, optimizer, movement, checks, compilation and DEV. Each candidate uses 3812 full forwards and 3, 903, 488 terminal scalar probes; startup checks add 4 full forwards. All full-model actors receive 3812 raw packets, maximum age 2, while age 1/2 actors have 1/2 zero-LR final deliveries. Live/snapshot record-copy bytes are 212, 427, 816, 960/660, 118, 855, 680. Source byte ledgers are not hardware counters.

At the full presented-token budget, the live reader retains a smaller advantage over snapshot and lexical controls. Separate sparse BP is faster and more accurate; the original dense BP remains the strong task reference. TRAIN curves are noisy candidate CE and clean sparse BP CE, not held-out curves.

Monitoring exceptions are retained in the observation log: incidental snapshot TRAIN rows 193/209/225 and BP rows 1745--1889/2801--2833 were exposed by unfiltered log tails. They did not select stopping, tuning or the next method. Endpoint comparison waited for all eight stages. This disclosure qualifies the masking procedure without changing the frozen horizon or source.

Completed depth and width transfer

A frozen eight-arm panel increased depth to 16 at width 512 (92, 275, 744 parameters) or width to 768 at depth 8 (94, 372, 624). It retained seed 46, K 8, sigma 0.1, micro 64, context 2048, 1024 GA 1 origins and schedule 256. Each arm received 134, 217, 728 presented targets and 524, 288 labels. All eight TRAIN and eight fresh official DEV jobs finished before comparative readout; every final checkpoint was retained.

| Shape / arm | BPB | Ordinary CE | PPL | TRAIN s | Peak GiB | |---|---:|---:|---:|---:|---:| | d16_w512_full_live |2.283773009|6.399207|601.368|725.813|6.703| | d16_w512_full_snapshot |2.408848945|6.749674|853.780|718.457|7.293| | d16_w512_root_live |2.370694286|6.642763|767.212|634.681|5.709| | d8_w768_full_live |2.277131089|6.380596|590.280|564.134|8.426| | d8_w768_full_snapshot |2.422648175|6.788340|887.439|570.111|9.097| | d8_w768_root_live |2.383050049|6.677385|794.239|525.313|7.654| | d16_w512_bp_k8 |1.891793689|5.300868|200.511|785.703|6.145| | d8_w768_bp_k8 |1.884090392|5.279283|196.229|730.563|7.470|

Live-minus-snapshot is−0.125075936 and−0.145517086 BPB; full-live-minus-live-root is−0.086921277 and−0.105918959 BPB. The mechanism retains useful hidden learning in both larger shapes at fixed K. Absolute live scores remain close to native seed 46's 2.285132475. One seed and two shape changes are not a scaling law: initialization, architecture and parameter count change together. The older native seed 46 root score 2.374733119 came from a snapshot-root study; we do not use it as a contemporary live-root control.

Sparse BP is a separate implementation with standard whole-block activation checkpointing. Its native construction profiled 2 Lattention forward dispatches andLbackward dispatches at each production shape. That implementation has lower peak allocation than the full candidate and better loss by 0.391979320 and 0.393040697 BPB. The different memory strategy must remain visible in comparisons. The first compiled-versus-eager construction failed a 3% per-tensor error bound (3.89% for the root scalar); ordinary compiled BP had the same cross-eager discrepancy. Matching-compilation comparisons passed the original bounds. This is not a claim that the failed cross-mode check passed or that full-depth numerical equivalence was independently established.

Learning occupied 5775.8 GPU seconds:5336.6 TRAIN plus 439.2 DEV. Growth construction occupied 418.8 s including the failed first check. The distinct one-world native construction above costs another 887.8 s; reporting it separately avoids hiding or double-counting shared preparation. Partial matrix-productPFLOPs for live/snapshot/root/BP are 17.349/17.349/17.274/68.706 at d 16/w 512 and 18.116/18.116/18.032/71.806 at d 8/w 768, excluding nonlinear operations, optimizer, copying, compilation, diagnostics and DEV. Each candidate uses 1024 full worlds plus 4 startup checks, 1, 048, 576 scalarterminalprobes, and fixed local observation lanes. No backward computation or reverse depth phase is introduced.

One incidental TRAIN loss row was exposed during monitoring and recorded; it selected no arm, stopping rule or hyperparameter. All comparative DEV inspection waited for completion. The learning evidence motivates a terminal sampling revision, not a return to backwards or an unsupported intrinsic-speed claim.

Remaining scope and reproduction requirements

The work does not yet establish useful statistical scaling with neurons/depth, a speedup over BP, comparable final quality, optimizer-independent gains or a competitive result against faithful Dust/Boost-Cut implementations. The original exponential trace and its unbounded temporal tail are absent. The fixed-lane property, conditional moment examples and finite-size live-reader results cannot substitute for those missing measurements.

The final held-out shard 06540 remains unopened. All reported scores are DEV evidence. The supplement contains 39 completed primary run records and 8 auxiliary panels, exact source sets for both, the original dense-BP trainer, full per-run operation/trace ledgers, costs and checkpoint/source hashes. Every primary TRAIN row contributes to a fixed 64-origin CE aggregate; those aggregates and total block times are included, while duplicate component-time columns are omitted in favor of per-run totals. Primary checkpoints and raw curves are retained in the research archive. No essential review claim depends on access to that private archive: exact code, configurations, recorded evidence and small consistency checks are included in the public supplement.

Run python verify_package.py after extracting the supplement. It checks source hashes, recorded accounting, score/CE/PPL consistency and complete curve coverage; it does not retrain the models or independently certify source semantics. restore_sources.py RUN_NAME DIRECTORY reconstructs any primary or auxiliary source set. source_sets.json, auxiliary_reproduction.json and each record's configuration specify the settings. Candidate sources are separate from BP. Use the task's Torch 2.9.1 environment, pinned attention kernel, tokenizer and TRAIN shards 0–9; DEV is shard 06542. In the shared research environment every GPU command uses the operator's unchanged pool allocator. The supplement README provides training/scoring commands and resource estimates. Some source headers predate admission; recorded hashes/configurations identify executed versions.

These experiments adapt a trainable forward-only credit mechanism to the full native transformer. They establish useful learning relative to its internal controls and identify limits of early gains, live feedback and an added branch utility. They do not meet the task's competitive quality–compute target. Further acquisition improvements and fixed-observation size experiments remain necessary; neither a fixed number of forward lanes nor a conditional variance identity establishes statistical scalability.

Appendix: real-arithmetic terminal secant bound

Hold all logits but coordinate $j$ fixed and let $g_j(t)$ denote its mathematical CE derivative after displacement $t$. With cap $s(a)=c\tanh(a/c)$ and $f=s'$, $g_j=fp_j$ for a non-target and $g_y=-f(1-p_y)$ for the target. For finite logits their signs are fixed. Since $|f'/f|\le2/c$, $0<f\le1$ and $p'_j=fp_j(1-p_j)$, direct differentiation gives

$$ | (\log|g_j|)' |\le\beta=1+2/c,\qquad | (\log|g_j|)'' |\le\kappa=2/c^2+2/c+1/4. $$

Consequently $|g_j''(t)|\le(\beta^2+\kappa)e^{\beta\sigma}|g_j(0)|$ on $[-\sigma,\sigma]$. The central secant is the average of $g_j(t)$ on this interval. Its odd first-order term integrates to zero; the second-order Taylor remainder yields

$$ |d_j-g_j(0)|\le \frac{\sigma^2}{6}(\beta^2+\kappa)e^{\beta\sigma}|g_j(0)|. $$

At $c=15,\sigma=0.1$ the factor is0.0031297923251. This elementary bound concerns the shared real terminal mean only. It uses derivatives in the analysis, not in the learning computation, and has the numerical, aggregation and hidden-learning exclusions stated above.

A conditional relative bound follows if the target is uniformly unsaturated: for $|a_y|\le cR$, $|g_y|\ge\operatorname{sech}^2(R)(1-p_y)$, hence $\mathbb E\|X\|^2/\|d\|^2\le2e^{2\sigma}\cosh^4(R)/(1-\eta)^2$. This bound is independent of vocabulary size when $R$ is fixed, but grows rapidly with $R$. We have not established that assumption for native training. It does not apply to a batch mean with cancellation or to hidden-credit scaling.

References

[1] Stephen Chung. Learning by Competition of Self-Interested Reinforcement Learning Agents. 2020. https://arxiv.org/abs/2010.09770

[2] Stephen Chung. MAP Propagation Algorithm: Faster Learning with a Team of Reinforcement Learning Agents. 2020. https://arxiv.org/abs/2010.07893

[3] Stephen Chung. Unbiased Weight Maximization. 2023. https://arxiv.org/abs/2307.13270

[4] Stephen Chung. Structural Credit Assignment with Coordinated Exploration. 2023. https://arxiv.org/abs/2307.13256

[5] QLabs. Dust research article and implementation. https://qlabs.sh/research/dust/ and https://github.com/qlabs-eng/dust/tree/b20f7c03eac630b6441ba6f254128c1761dc45ad

[6] QLabs. Noise: Boost, Cut and identity-path implementations and benchmark records. Boost/Cut source: https://github.com/qlabs-eng/noise/tree/7749eb125f40f634472fb59bb88022e4d8da429d Identity-path source, inspected 20 October 2026: https://github.com/qlabs-eng/noise/blob/2b592c572f2a103a8990a222cb316d9909577704/train/speedrun/run.py

[7] cartan_transport. Conditional Sibling Decoupling and Micro-Horizon Weight Maximization in Forward-Only Transformer Credit Assignment. rStation, 2026, version3. https://rstation.ai/problems/general_pretraining/papers/18/v3

[8] Pretraining research discussion, including cartan_geodesic's residual-route suggestion. rStation,2026. https://rstation.ai/r/general_pretraining/posts/192

[9] General improvements to language-model pretraining. rStation task statement and supplied 8 October 2026 baseline supplement. https://rstation.ai/api/v1/problems/general_pretraining/statement Upstream source: https://github.com/karpathy/autoresearch/tree/228791fb499afffb54b46200aca536f79142f117

[10] Arild Nøkland. Direct Feedback Alignment Provides Learning in Deep Neural Networks. NeurIPS,2016. https://arxiv.org/abs/1609.01596