# Norm-aware feedback learning-rate preconditioner

- ID: 771
- Canonical URL: https://synthcore.org/idea/771/norm-aware-feedback-learning-rate-preconditioner
- API JSON: https://synthcore.org/api/idea/771.json
- API Markdown: https://synthcore.org/api/idea/771.md
- Verification status: unverified
- Source: [arXiv:2607.11835](https://arxiv.org/abs/2607.11835)
- Category: optimization
- Solves: speedup, stability
- ML areas: rnn, ssm, optimizer, training-dynamics
- Math tags: control-theory, linear-algebra, dynamical-systems, optimization, functional-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Use the feedbacked control-to-state norm as a conditioning diagnostic to adapt the optimizer step applied to recurrent residual outputs. When the estimated horizon amplification is large, reduce or precondition the residual-control update; when feedback makes it small, permit larger updates.

## Mathematical statement

Let T_N map a length-N residual-control sequence v=(v_0,...,v_{N-1}) to the resulting hidden-state sequence under h_{t+1}=A_cl h_t+Bv_t. Its largest singular value sigma_N=||T_N||_2 measures perturbation amplification. The block at output time i and control time j is A_cl^{i-j}B for i>=j and zero otherwise. Since lambda_max(T_N^*T_N)=sigma_N^2, the reduced quadratic curvature induced by the state pathway is bounded by sigma_N^2. Therefore scale the residual-control gradient g_v by 1/(epsilon+sigma_hat_N^2), where epsilon>0 prevents division by zero and sigma_hat_N is a power-iteration estimate. This is a conditioning heuristic inspired by the paper's reduced-gradient and preconditioned-system analysis; it does not require replacing AdamW globally.

## Key formulas

- $$\sigma_N=\|T_N\|_2=\sup_{v\ne0}\frac{\|T_Nv\|_2}{\|v\|_2}$$
- $$\lambda_{\max}(T_N^*T_N)=\sigma_N^2$$
- $$g_v^{\mathrm{fb}}=\frac{1}{\varepsilon+\widehat\sigma_N^2}g_v,\qquad \eta_v=\frac{\eta_0}{\varepsilon+\widehat\sigma_N^2}$$
- $$\mathcal T_N(A_{\mathrm{cl}},B)_{ij}=\begin{cases}A_{\mathrm{cl}}^{\,i-j}B,&i\ge j,\\0,&i<j.\end{cases}$$

## Implementation notes

Integrate this at the optimizer step for parameters that generate recurrent residual controls, such as the input projection and recurrent residual matrix in an RNN or SSM. Do not initially rescale every model gradient; the paper's conditioning quantity concerns the control-to-state pathway. Every M optimizer steps, estimate sigma_hat with one or two power iterations without constructing the dense T_N matrix. Draw a normalized sequence q with shape [N,d_u], compute z=T_Nq using `z[0]=0; z[t+1]=A_cl@z[t]+B@q[t]`, then compute r=T_N^Tz by reverse accumulation. With p initialized to zero, iterate t from N-1 down to 0, accumulate `r[t]=B.T@p`, add the appropriate output-state contribution `z[t+1]` to p, and propagate `p=A_cl.T@p`; normalize r and estimate sigma_hat=sqrt(max(<q,r>,0)). Validate indexing against an explicitly materialized small T_N. Apply `g <- g/(eps+sigma_hat**2)` only to control-producing parameters, or set their learning rate to `eta0/(eps+sigma_hat**2)`. Smooth sigma_hat with an exponential moving average and clip the multiplier to [s_min,s_max]. The mathematical inputs are T_N and its spectral bound; empirical inputs are sigma_hat, gradient norms, and validation loss. First test a 2-layer tanh RNN on copy-memory and sequential MNIST against AdamW, global gradient clipping, and fixed learning-rate scaling at equal batch size and FLOPs. Log sigma_hat, effective learning rate, gradient norm, and loss. Success requires faster time-to-accuracy or stable training at longer horizons, rather than merely smaller updates.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.