Range-Space Projected Learning for Noisy Iterations

1. Integration point: insert the projection immediately before the optimizer applies an update to a recurrent network, state-space model, neural ODE, or any model trained on a fixed repeated horizon. Compute the Jacobian of the concatenated horizon outputs with respect to parameters, not merely the Jacobian of the final loss. For large models, use Jacobian-vector products and vector-Jacobian products with 1–3 power iterations to estimate the dominant right singular subspace; do not materialize the full Jacobian. 2. Pseudocode: collect a trajectory with current parameters; estimate V_r from JVP/VJP products; compute g=∇_θL; set g_proj=V_r(V_r^Tg); optionally add measured perturbation ξ; update θ←θ−ηg_proj. Recompute the subspace every M optimizer steps and use an exponential moving average of the subspace to reduce cost. 3. The transferred mechanism is the paper's range-space/null-space separation and suppression of repeated perturbations. The Jacobian, rank r, singular values, and noise covariance are estimated empirically. 4. First cheap experiment: train a two-layer tanh RNN on a fixed sine-wave sequence for 1,000 repeated trials using Adam, projected Adam, and an unprojected baseline, while injecting Gaussian parameter noise or simulated quantization after every update. Sweep rank fraction r/d∈{0.1,0.25,0.5,1.0} and measure trajectory error and parameter displacement. The quantitative prediction is that unprojected output perturbation variance grows approximately linearly with trial count i, while projection reduces its slope by the ratio tr(JP_rΣ_ξP_rJ^T)/tr(JΣ_ξJ^T). For a deliberately redundant network with a large Jacobian null space, this slope should fall by at least 2×. Increasing noise magnitude should also produce a later divergence boundary for projected training.