Input-Subspace Perturbation Learning
Implementation & benchmark of arXiv:2607.06079 — Scalable Perturbation Learning for Online Self-Supervised Learning in Echo State Networks
Source paper: Scalable Perturbation Learning for Online Self-Supervised Learning in Echo State Networks arXiv:2607.06079 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace full-dimensional node or weight perturbation with perturbations in an input-conditioned d-dimensional tangent subspace, where d is the input or feature dimension and is much smaller than the reservoir width or parameter count. Estimate the update using only scalar self-supervised losses from positive and negative perturbations, then map the low-dimensional update back to the trainable parameters.
Formulas
Mathematical statement
The paper defines weighted seminorms by ||x||_G^2 := x^T G x and ||X||_{G,H}^2 := tr(G X H X^T), for symmetric positive-semidefinite matrices G and H. Its central construction orthogonally separates an online self-supervised loss into an input-dependent component and a redundant component determined only by fixed ESN parameters; only the former is perturbed. The operational adaptation is as follows. Let theta in R^p be trainable parameters, u_t in R^d the current input, and A_t in R^(p x d) an input-conditioned tangent map. For a scalar loss L_t(theta), perturb theta along A_t epsilon, with epsilon in R^d, rather than along a p-dimensional random vector. The two-sided estimator g_t = (1/(2 m sigma)) sum from j=1 to m of [L_t(theta + sigma A_t epsilon_j) - L_t(theta - sigma A_t epsilon_j)] epsilon_j estimates A_t^T grad_theta L_t when epsilon_j are independent isotropic vectors with E[epsilon_j epsilon_j^T] = I_d. The update theta <- theta - eta A_t g_t changes only the dynamically necessary subspace. Under bounded directional derivatives, the estimator's perturbation variance is controlled by d and the Frobenius geometry of A_t, instead of by p; this is the transferable mechanism behind the paper's reservoir-to-input dimensionality reduction.
Implementation notes
Integrate this at the recurrent-state update or readout update of an echo state network, lightweight SSM, or recurrent neural network that must learn online without backpropagation through time. Let theta be the trainable readout vector or a small adapter attached to the recurrent cell, let u_t be the current input, and let d be its feature dimension. Construct A_t in R^(p x d) as an input-conditioned tangent map. The simplest MVP uses a fixed sparse Gaussian matrix A followed by input gating: A_t = A diag(g_t), where g_t = sigmoid(B u_t); a more ESN-like option uses the current reservoir state h_t and sets g_t = tanh(C[u_t;h_t]). Normalize A_t so ||A_t||_F is approximately constant across timesteps. For each online sample, compute a self-supervised target such as u_(t+1), form the scalar prediction loss L_t(theta), draw m antithetic epsilon_j ~ N(0,I_d), evaluate the same loss with theta_plus = theta + sigma A_t epsilon_j and theta_minus = theta - sigma A_t epsilon_j, and accumulate g_hat using the displayed two-sided formula. Apply theta <- theta - eta A_t g_hat, optionally with Adam moments maintained only in R^d. Pseudocode: h_t = reservoir(u_t,h_prev); A_t = normalize(A * diag(sigmoid(B*u_t))); g=0; for j in 1..m: e=randn(d); lp=L(theta+sigma*A_t@e); lm=L(theta-sigma*A_t@e); g += (lp-lm)*e/(2*m*sigma); theta -= eta*A_t@g. The paper-derived quantities are the low-dimensional perturbation, antithetic scalar estimator, and input-subspace update; estimate A, sigma, and eta empirically. First test on sequential MNIST or permuted sequential MNIST using a width-1,000 ESN with a linear next-symbol predictor. Compare full p-dimensional node perturbation, ordinary finite-difference perturbation, and the proposed d-dimensional method at equal scalar loss evaluations. Success means lower update variance, faster online loss descent, and substantially better performance as reservoir width increases, while peak memory remains independent of the number of reservoir units apart from the forward state.
Verification
Stage 1 · Toy mechanism gate: Passed ✓
Stage 2 · Mechanism transferred to benchmark: Confirmed ✓
Stage 2 · Practical benchmark result: Significantly worse ✗
Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built an input-subspace antithetic scalar-loss estimator in input_subspace.py and verified its mechanism on a linear loss. Observed estimator MSE followed the Gaussian prediction E||g_hat-q||²=(d+1)||q||²/m: dimension slope 1.006 versus predicted 1, inverse-m slope 17.04 versus predicted 17, and ambient-p coefficient of variation 0.76%, confirming independence from p when the projected gradient is fixed. The subspace learner reached 1.68e-12 final-window loss, but the full-dimensional baseline reached 2.95e-13, so the mechanism was validated while a practical loss win was not observed.
- Agent confidence
- 9/10
- Baseline
- Full-dimensional antithetic perturbation: final 100-step loss 2.95e-13; all-step mean loss 0.0304; parameter error 3.59; 16 scalar evaluations per step.
- Idea
- Input-subspace antithetic perturbation: final 100-step loss 1.68e-12; all-step mean loss 0.0781; parameter error 7.22e-7; 16 scalar evaluations per step. Math sweeps: dimension slope 1.006 (predicted 1), inverse-m slope 17.04 (predicted 17), ambient-p CV 0.0076.
Limitations: Only a deterministic linear online regression toy was tested; no ESN, recurrent dynamics, sequential MNIST, input gating experiment, wall-clock benchmark, memory benchmark, or broader hyperparameter/seed sweep was performed. The exact Gaussian MSE includes the additive +1 fourth-moment term, while the idea's d-scaling statement is an order-of-growth claim rather than an exact equality.
How to run: python3 experiment.py
Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.06079)
Significantly WORSE than the tuned baseline (+13118.5%, p=0.0081, wins 0 / 8 paired seeds).
- Benchmark
- Dynamics forecasting
- Model
- rnn_small
- Paired seeds
- 8
- Baseline mean
- 0.0042
- Idea mean
- 0.5571
- Effect (Δ)
- 0.5528 (+13118.5%; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 0.0081 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- n/a
- Mechanism
- Confirmed ✓
- Practical effect
- Hurts
- Baseline tuning
- swept over 3 configs
Only the built-in dynamics track with rnn_small, 400 training samples, 200 test samples, 8 epochs, m=2, and three sigma settings was tested. No larger reservoirs, longer horizons, alternative tangent maps, low-dimensional Adam moments, wall-clock or memory measurements, or other tracks were tested.
How to run: python3 bench_run.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Verification history
Stage 2 — Benchmark Worked ✓
Agent confidence: 8/10
The tabular track was selected because this idea modifies the optimizer/training procedure. The practical result is negative: the input-subspace optimizer reached MSE 233.314 versus the tuned baseline's 15.059, with 0/8 wins. The mechanism criterion transferred quantitatively at NN-scale ambient dimension: observed variance slope 0.4878 versus predicted 0.5, but this did not translate into better training.
- Baseline
- Tuned bench.train_model baseline, best lr=0.01 and weight_decay=0.0; mean test MSE 15.0593 over 8 seeds.
- Idea
- Input-gated d=10 tangent optimizer, eta=0.01, sigma=0.03, m=2; mean test MSE 233.3140 over 8 seeds; paired delta +218.2548, 0/8 wins, permutation p=0.0081 unfavorable to the idea.
Only tabular was tested; vision, sequence, and dynamics were not tested. The variance signature is an exact Gaussian linear-loss check at NN-scale ambient dimension, not nonlinear MLP-loss variance. Idea hyperparameters were fixed rather than swept, and wall-clock/memory were not measured.
How to run: python3 run_bench.py
Stage 2 — Benchmark latest Failed ✗
Agent confidence: 10/10
Implemented faithful input-subspace antithetic scalar-loss training for dynamics/rnn_small with A_t=A·diag(sigmoid(Bu)), normalized columns, scalar positive/negative losses, and low-dimensional updates. The estimator variance matched the predicted (d+1)/m law within 1%, but practical transfer failed: idea mean MSE 0.557051 versus baseline 0.004214, paired delta +0.552837, p=0.0081, with 0 of 8 wins.
- Baseline
- Adam rnn_small on dynamics, lr=0.01 selected by sweep, mean test MSE 0.004214 over 8 paired seeds.
- Idea
- Input-subspace antithetic update, best setting lr=0.003, sigma=0.03, m=2, mean test MSE 0.557051 over 8 seeds; delta +0.552837; p=0.0081.
- Mechanism signature
- At NN scale p=13313,d=3, observed/predicted estimator-MSE ratios for m=1,2,4,8 were 1.0037, 0.9936, 1.0090, 0.9975; confirmed=true.
Benchmark result
Significantly WORSE than the tuned baseline (+13118.5%, p=0.0081, wins 0 / 8 paired seeds).
- Benchmark
- Dynamics forecasting
- Model
- rnn_small
- Paired seeds
- 8
- Baseline mean
- 0.0042
- Idea mean
- 0.5571
- Effect (Δ)
- 0.5528 (+13118.5%; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 0.0081 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- n/a
- Mechanism
- Confirmed ✓
- Practical effect
- Hurts
- Baseline tuning
- swept over 3 configs
Only the built-in dynamics track with rnn_small, 400 training samples, 200 test samples, 8 epochs, m=2, and three sigma settings was tested. No larger reservoirs, longer horizons, alternative tangent maps, low-dimensional Adam moments, wall-clock or memory measurements, or other tracks were tested.
How to run: python3 bench_run.py
Artifacts
- bench_report.json 3.0 KB View Raw
- bench_run.py 5.4 KB View Raw
- experiment.py 4.3 KB View Raw
- input_subspace.py 1.2 KB View Raw
- report.md 1.7 KB View
- report_bench_2026-08-30T121431.md 3.8 KB View
- report_bench_2026-08-30T130252.md 3.6 KB View
- results.json 4.8 KB View Raw
- run_bench.py 5.8 KB View Raw