# Input-Subspace Perturbation Learning

- ID: 462
- Canonical URL: https://synthcore.org/idea/462/input-subspace-perturbation-learning
- API JSON: https://synthcore.org/api/idea/462.json
- API Markdown: https://synthcore.org/api/idea/462.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.06079](https://arxiv.org/abs/2607.06079)
- Category: training
- Solves: speedup, stability, memory
- ML areas: rnn, ssm, optimizer, training
- Math tags: linear-algebra, optimization, stochastic-processes, functional-analysis
- Ratings: usefulness 8/10; difficulty 5/10; novelty 7/10

## Idea description

Replace full-dimensional node or weight perturbation with perturbations in an input-conditioned d-dimensional tangent subspace, where d is the input or feature dimension and is much smaller than the reservoir width or parameter count. Estimate the update using only scalar self-supervised losses from positive and negative perturbations, then map the low-dimensional update back to the trainable parameters.

## Mathematical statement

The paper defines weighted seminorms by ||x||_G^2 := x^T G x and ||X||_{G,H}^2 := tr(G X H X^T), for symmetric positive-semidefinite matrices G and H. Its central construction orthogonally separates an online self-supervised loss into an input-dependent component and a redundant component determined only by fixed ESN parameters; only the former is perturbed. The operational adaptation is as follows. Let theta in R^p be trainable parameters, u_t in R^d the current input, and A_t in R^(p x d) an input-conditioned tangent map. For a scalar loss L_t(theta), perturb theta along A_t epsilon, with epsilon in R^d, rather than along a p-dimensional random vector. The two-sided estimator g_t = (1/(2 m sigma)) sum from j=1 to m of [L_t(theta + sigma A_t epsilon_j) - L_t(theta - sigma A_t epsilon_j)] epsilon_j estimates A_t^T grad_theta L_t when epsilon_j are independent isotropic vectors with E[epsilon_j epsilon_j^T] = I_d. The update theta <- theta - eta A_t g_t changes only the dynamically necessary subspace. Under bounded directional derivatives, the estimator's perturbation variance is controlled by d and the Frobenius geometry of A_t, instead of by p; this is the transferable mechanism behind the paper's reservoir-to-input dimensionality reduction.

## Key formulas

- $$\|x\|_{G}^{2}:=x^{\top}Gx,\qquad \|X\|_{G,H}^{2}:=\operatorname{tr}(G X H X^{\top}).$$
- $$\widehat g_t=\frac{1}{2m\sigma}\sum_{j=1}^{m}\left[L_t(\theta+\sigma A_t\varepsilon_j)-L_t(\theta-\sigma A_t\varepsilon_j)\right]\varepsilon_j,\qquad \mathbb E[\varepsilon_j\varepsilon_j^{\top}]=I_d.$$$$
- $$\theta_{t+1}=\theta_t-\eta A_t\widehat g_t.$$$$
- $$\operatorname{Var}(\widehat g_t)\ \text{is controlled by the perturbation dimension }d\text{ and }\|A_t\|_F^2,\text{ rather than by the full parameter dimension }p.$$$$

## Implementation notes

Integrate this at the recurrent-state update or readout update of an echo state network, lightweight SSM, or recurrent neural network that must learn online without backpropagation through time. Let theta be the trainable readout vector or a small adapter attached to the recurrent cell, let u_t be the current input, and let d be its feature dimension. Construct A_t in R^(p x d) as an input-conditioned tangent map. The simplest MVP uses a fixed sparse Gaussian matrix A followed by input gating: A_t = A diag(g_t), where g_t = sigmoid(B u_t); a more ESN-like option uses the current reservoir state h_t and sets g_t = tanh(C[u_t;h_t]). Normalize A_t so ||A_t||_F is approximately constant across timesteps. For each online sample, compute a self-supervised target such as u_(t+1), form the scalar prediction loss L_t(theta), draw m antithetic epsilon_j ~ N(0,I_d), evaluate the same loss with theta_plus = theta + sigma A_t epsilon_j and theta_minus = theta - sigma A_t epsilon_j, and accumulate g_hat using the displayed two-sided formula. Apply theta <- theta - eta A_t g_hat, optionally with Adam moments maintained only in R^d. Pseudocode: `h_t = reservoir(u_t,h_prev); A_t = normalize(A * diag(sigmoid(B*u_t))); g=0; for j in 1..m: e=randn(d); lp=L(theta+sigma*A_t@e); lm=L(theta-sigma*A_t@e); g += (lp-lm)*e/(2*m*sigma); theta -= eta*A_t@g`. The paper-derived quantities are the low-dimensional perturbation, antithetic scalar estimator, and input-subspace update; estimate A, sigma, and eta empirically. First test on sequential MNIST or permuted sequential MNIST using a width-1,000 ESN with a linear next-symbol predictor. Compare full p-dimensional node perturbation, ordinary finite-difference perturbation, and the proposed d-dimensional method at equal scalar loss evaluations. Success means lower update variance, faster online loss descent, and substantially better performance as reservoir width increases, while peak memory remains independent of the number of reservoir units apart from the forward state.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built an input-subspace antithetic scalar-loss estimator in input_subspace.py and verified its mechanism on a linear loss. Observed estimator MSE followed the Gaussian prediction E||g_hat-q||²=(d+1)||q||²/m: dimension slope 1.006 versus predicted 1, inverse-m slope 17.04 versus predicted 17, and ambient-p coefficient of variation 0.76%, confirming independence from p when the projected gradient is fixed. The subspace learner reached 1.68e-12 final-window loss, but the full-dimensional baseline reached 2.95e-13, so the mechanism was validated while a practical loss win was not observed.

### Mechanism check

- Verdict: Built an input-subspace antithetic scalar-loss estimator in input_subspace.py and verified its mechanism on a linear loss. Observed estimator MSE followed the Gaussian prediction E||g_hat-q||²=(d+1)||q||²/m: dimension slope 1.006 versus predicted 1, inverse-m slope 17.04 versus predicted 17, and ambient-p coefficient of variation 0.76%, confirming independence from p when the projected gradient is fixed. The subspace learner reached 1.68e-12 final-window loss, but the full-dimensional baseline reached 2.95e-13, so the mechanism was validated while a practical loss win was not observed.
- Confidence: 9/10
- Limitations: Only a deterministic linear online regression toy was tested; no ESN, recurrent dynamics, sequential MNIST, input gating experiment, wall-clock benchmark, memory benchmark, or broader hyperparameter/seed sweep was performed. The exact Gaussian MSE includes the additive +1 fourth-moment term, while the idea's d-scaling statement is an order-of-growth claim rather than an exact equality.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/95/bench_report.json)
- [bench_run.py](https://synthcore.org/code/95/bench_run.py)
- [experiment.py](https://synthcore.org/code/95/experiment.py)
- [input_subspace.py](https://synthcore.org/code/95/input_subspace.py)
- [report.md](https://synthcore.org/code/95/report.md)
- [report_bench_2026-08-30T121431.md](https://synthcore.org/code/95/report_bench_2026-08-30T121431.md)
- [report_bench_2026-08-30T130252.md](https://synthcore.org/code/95/report_bench_2026-08-30T130252.md)
- [results.json](https://synthcore.org/code/95/results.json)
- [run_bench.py](https://synthcore.org/code/95/run_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/95)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.