△ Mechanism confirmed, baseline not beaten 2026

Fixed-Projection Temporal Plasticity

Implementation & benchmark of arXiv:2607.21622 — Local Synaptic Rules Can Implement a SIGReg Gradient Without Backpropagation

Usefulness7/10

Difficulty5/10

Novelty7/10

Training Sample efficiency Stability Scalability Training Self supervised Embedding Probability Optimization Linear algebra Dynamical systems

Source paper: Local Synaptic Rules Can Implement a SIGReg Gradient Without Backpropagation arXiv:2607.21622 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace backpropagation through a small encoder with an online local update driven by consecutive examples and a fixed random projection of hidden activity. The projection produces a modulatory signal that encourages temporally adjacent inputs to have compatible representations, while the homeostatic term prevents sigmoid units from saturating or collapsing.

Formulas

$$h_i=\sigma\!\left(\sum_{j=1}^{n}W_{ij}x_j+b_i\right),\qquad \sigma(u)=\frac{1}{1+e^{-u}}.$$

$$\alpha_i=h_i(1-h_i)\in\left(0,\frac14\right].$$

$$f_t=Ah_t,\qquad C_A=A^\top A.$$

$$u_t=C_A(h_{t+1}-h_t)-\lambda(h_t-\rho\mathbf{1}),\qquad \Delta W=\eta\,(\alpha_t\odot u_t)x_t^\top.$$

Mathematical statement

The paper models each encoder propensity as $h_i(x)=\sigma(\sum_j W_{ij}x_j+b_i)$ with $W\in\mathbb{R}^{C\times n}$, $b\in\mathbb{R}^C$, $x\in\mathbb{R}^n$, and $h\in(0,1)^C$. Its local gain is $\alpha_i=h_i(1-h_i)$, the exact derivative of the sigmoid propensity with respect to its input current; this suppresses updates from saturated neurons. The circuit uses a fixed projection $A$ and projected activity $f=Ah$, with $A\in\mathbb{R}^{M\times C}$ and $C_A=A^\top A$. For consecutive hidden activities $h_t,h_{t+1}$, use the projected temporal discrepancy $C_A(h_{t+1}-h_t)$ as a locally available modulatory signal. With target firing rate $\rho\in(0,1)$ and homeostatic coefficient $\lambda\geq0$, the adapted update is $\Delta W_{ij}=\eta\alpha_{t,i}u_{t,i}x_{t,j}$, where $u_t=C_A(h_{t+1}-h_t)-\lambda(h_t-\rho\mathbf{1})$. The temporal term comes from the paper's fixed flashlight/granule projection and STDP-plus decomposition; the homeostatic term implements its local firing-statistics correction.

Implementation notes

Integrate the rule at the encoder weight update rather than in the forward computation graph. Initialize a sigmoid encoder with parameters $W,b$ and retain a fixed Gaussian, sparse, or orthogonal matrix $A\in\mathbb{R}^{M\times C}$ throughout training. For every adjacent pair $(x_t,x_{t+1})$, compute $h_t=\sigma(Wx_t+b)$ and $h_{t+1}=\sigma(Wx_{t+1}+b)$ with gradients disabled. Compute projected activities $f_t=Ah_t$ and $f_{t+1}=Ah_{t+1}$, then calculate $u_t=A^\top(f_{t+1}-f_t)-\lambda(h_t-\rho\mathbf{1})$. Set $\alpha_t=h_t\odot(1-h_t)$ and apply the rank-one update $W\leftarrow W+\eta(\alpha_t\odot u_t)x_t^\top$. Update the bias with $b\leftarrow b+\eta_b(\alpha_t\odot u_t)$, or initially keep $b$ fixed. A hardware-local version replaces explicit $A^\top$ multiplication with fixed feedback synapses from the flashlight population. To avoid a hand-selected target rate, maintain $\bar h\leftarrow(1-\gamma)\bar h+\gamma h_t$ and replace $h_t-\rho\mathbf{1}$ by $h_t-\bar h$. The mathematical quantities taken directly from the paper are the sigmoid propensity, local gain, fixed projection, and consecutive-input rule; $\eta,\lambda,\rho,\gamma$, projection dimension $M$, and update normalization are empirical hyperparameters. First test a 2-layer, 256-unit sigmoid encoder on temporally ordered MNIST. Compare this method with an identical encoder trained by backpropagation on the same temporal objective and with shuffled-pair training. Freeze the encoder and fit a linear classifier. Record linear-probe accuracy, activity entropy, fraction of saturated units, and accuracy per synaptic update. The key falsifiable signal is a large ordered-versus-shuffled accuracy gap without collapse, while the online method should require one forward pass and one update per example and remain stable without replay.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built the fixed-projection temporal plasticity update and a reproducible ordered-versus-shuffled digits experiment. The mechanism checks passed: homeostatic convergence steps scaled approximately inversely with lambda, scalar projection drive matched the exact a^2 law, and sigmoid gain matched h(1-h), peaking at h=0.5 and suppressing saturated units. Ordered local plasticity beat shuffled training (61.3% vs 40.6%) with no saturation, but did not beat the backprop baseline (64.4%), so this is mechanism validation rather than a demonstrated training win.

Agent confidence: 9/10

Baseline: Backprop ordered: linear-probe accuracy 64.4%, mean activity 0.546, saturated fraction 3.09%, temporal MSE 0.000653.
Idea: Fixed-projection local ordered: accuracy 61.3%, mean activity 0.496, saturated fraction 0%, temporal MSE 0.001633; shuffled local control: 40.6% accuracy. Math checks: contraction observed/predicted steps [3411/3605, 1704/1801, 851/899, 424/448, 211/222] for lambda [0.05,0.1,0.2,0.4,0.8], and projection ratios exactly matched [0.0625,0.25,1,4,16].

Limitations: Used sklearn digits rather than temporally ordered MNIST, a one-layer 24-unit encoder rather than the proposed 2-layer 256-unit model, one random seed, and dense Gaussian projection only. No CUDA benchmark, synaptic-update/FLOP normalization, adaptive target-rate variant, or multi-seed statistical test was run.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.21622) Sep 2, 2026

Significantly WORSE than the tuned baseline (+0.1%, p=0.0149, wins 1 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 1.0069
Idea mean: 1.0083
Effect (Δ): 0.0015 (+0.1%; negative = idea better)
Wins: 1 / 8 paired seeds
p-value: 0.0149 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.1%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Used the benchmark's reduced 400-train/400-test budget and 5 epochs rather than the README's larger default sizes; did not test vision, tabular, or dynamics tracks. The intervention updates only the sigmoid input projection while the transformer body/head remain Adam-trained, which is the chosen operationalization of replacing encoder backpropagation.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented Fixed-Projection Temporal Plasticity as a local training intervention on an identical sigmoid-input transformer and evaluated it on the structurally matched sequence forecast track. The mechanism signature was confirmed quantitatively at NN scale: positive trained-model projection alignment (0.02075) and zero saturated activity. However, the idea had higher MSE than the tuned baseline (1.00834 vs 1.00688), with paired delta +0.001465 and permutation p=0.01485, yielding a significant loss; therefore worked=false.

Baseline: Adam sigmoid-input transformer, best lr=0.001; full 8-seed test MSE 1.006878 ± 0.171372; per-seed [0.991178, 0.755488, 1.263213, 0.788392, 1.153331, 0.941530, 1.184309, 0.977584]. Baseline sweep means: lr 0.001=0.949568, 0.003=1.508308, 0.01=3.829346.
Idea: Fixed-projection temporal local update, best lr=0.001; full 8-seed test MSE 1.008343 ± 0.172025; per-seed [0.990945, 0.756263, 1.265931, 0.789582, 1.153704, 0.943642, 1.188511, 0.978167]. Idea sweep means: lr 0.001=1.008343, 0.003=1.482208, 0.01=4.900032.
Mechanism signature: mean hidden activity=0.494660; saturated fraction=0.0; observed projection-alignment statistic=0.020752; predicted positive alignment=True; confirmed=True.
Paired comparison: delta_mean=+0.001464881; idea_wins=1/8; permutation p_value=0.01485; verdict='idea worse (significant)'.

Benchmark result