# Critical-Orbit Trust Region for Chebyshev Optimizers

- ID: 3093
- Canonical URL: https://synthcore.org/idea/3093/critical-orbit-trust-region-for-chebyshev-optimizers
- API JSON: https://synthcore.org/api/idea/3093.json
- API Markdown: https://synthcore.org/api/idea/3093.md
- Verification status: unverified
- Source: [arXiv:2609.02884](https://arxiv.org/abs/2609.02884)
- Category: dynamics
- Solves: stability, speedup
- ML areas: optimizer, training-dynamics, loss
- Math tags: dynamical-systems, bifurcations, optimization, linear-algebra
- Ratings: usefulness 6/10; difficulty 6/10; novelty 8/10

## Idea description

Replace an unconstrained third-order stationarity update by a Chebyshev root step on a one-dimensional restriction of the neural loss, while monitoring the images of the update map's free critical points. Shrink the trust radius or damping parameter whenever the maximum critical-value gain exceeds one, because the paper's mechanism predicts that critical-orbit expansion marks loss of a safe attracting basin.

## Mathematical statement

For a scalar stationarity equation p(x)=0, Chebyshev's method is C_p(x)=x-(1/2 L_p(x)+1)p(x)/p'(x), where L_p(x)=p(x)p''(x)/(p'(x))^2. For p_n(z)=z(z^n-1), every free critical point c of C_n has critical value c*=C_n(c)=-D_n c, with D_n=(n-1)^2(2n+1)/(27(n+1)^2). Therefore |c*|/|c|=D_n is below one for n<=16 and above one for n>=17. In the neural adaptation, p(t)=u^T grad_theta L(theta+t u) is a directional stationarity function, u is a normalized update direction, and p'(t) and p''(t) are directional second and third derivatives. Fit a local polynomial p_hat(t) and compute its Chebyshev map C_hat. Define Gamma as the largest ratio of the distance from a free critical point's image to the selected local root over the original distance. Use Gamma below one as an empirical contraction certificate.

## Key formulas

- $$C_p(x)=x-(1+L_p(x)/2)p(x)/p'(x),\quad L_p(x)=p(x)p''(x)/(p'(x))^2.$$
- $$c^*=C_n(c)=-D_n c,\quad D_n=((n-1)^2(2n+1))/(27(n+1)^2).$$
- $$Gamma=max_{c in Crit_free(C_hat)} |C_hat(c)-t_*|/|c-t_*|,\quad accept\ if\ Gamma<=gamma<1.$$
- $$t_{k+1}=t_k-alpha(1+hatL(t_k)/2)hatp(t_k)/hatp'(t_k),\quad theta_{k+1}=theta_k+t_{k+1}u.$$

## Implementation notes

Integrate this as an optional inner update for a neural optimizer rather than replacing minibatch SGD everywhere. At parameter vector theta, choose a normalized direction u=-P g_hat/||P g_hat||, where g_hat is a minibatch gradient and P is a diagonal Adam-style preconditioner. Probe the directional stationarity function p(t)=u^T grad L(theta+t u) at five to nine small values of t, using the same minibatch or common random numbers. Fit a degree-3 or degree-5 polynomial p_hat(t), differentiate it analytically, and apply the Chebyshev update. Find the real roots of the derivative C_hat'(t) with a polynomial root solver; discard poles, points outside the trust interval [-r,r], and critical points close to the selected root t_*. Compute Gamma from the fitted polynomial. If Gamma exceeds 0.9, halve r and set alpha to 0.5 alpha. If Gamma is below 0.5, increase r by 10 percent up to a cap. Reject any update that increases a fresh validation minibatch loss. The exact Chebyshev map and the contraction-expansion mechanism come from the paper; polynomial fitting, root selection, and the threshold are empirical adaptations. First test on a two-layer MLP fitting a one-dimensional polynomial regression dataset, comparing SGD, Adam, and undamped Chebyshev updates. Sweep trust radii and record divergence. The quantitative prediction is a transition near Gamma=1: Gamma<1 should produce bounded directional iterates and cubic local stationarity convergence, whereas Gamma>1 should produce more basin exits or exploding steps. On the exact synthetic family p_n, the implementation should recover the transition between n=16 and n=17 when D_n crosses one.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.