Universal Trust-Region Neural Optimizer

1. Integration point: replace AdamW or SGD's parameter update during full-batch or large-batch training. Represent the parameter vector as $\theta$, compute $g=\nabla F(\theta)$, and implement products $Bv$ without materializing $B$. For an MVP, let $B$ be a damped Gauss-Newton operator, empirical-Fisher operator, or diagonal curvature estimate $B=\operatorname{diag}(b_i)+\lambda I$; for a stronger experiment, use autodiff Hessian-vector products with truncated conjugate gradient. 2. Pseudocode: initialize $\theta$ and $\Delta$. At iteration $k$, compute batch loss $F(\theta)$ and gradient $g$. Approximately solve $\min_{\|s\|\leq\Delta}g^Ts+\frac12s^TBs$ using 5--20 truncated-CG iterations. Stop CG at the boundary or when it detects negative curvature. If the resulting predicted decrease is below $\kappa_C$ times the Cauchy decrease, replace it with the Cauchy step. Evaluate $F(\theta+s)$ on the same batch, compute $\rho=[F(\theta)-F(\theta+s)]/[m(0)-m(s)]$, and reject if $\rho<\eta_1$. For an accepted step, update $\theta\leftarrow\theta+s$. Set $\Delta\leftarrow\gamma_{\rm dec}\Delta$ after rejection; if $\rho>\eta_2$ and $\|s\|$ is near $\Delta$, set $\Delta\leftarrow\min(\gamma_{\rm inc}\Delta,\Delta_{\max})$; otherwise leave it unchanged. Use $\eta_1=0.1$, $\eta_2=0.75$, $\gamma_{\rm dec}=0.25$, and $\gamma_{\rm inc}=2$ as initial values. 3. The paper supplies the trust-region adaptation principle and the unknown-$\nu$ rate targets. Estimate $Bv$, the Cauchy decrease, and the Hölder behavior empirically; do not estimate $\nu$ for the controller. With minibatch noise, either compute actual decrease on the model batch or average the ratio over several batches to prevent noise-induced radius collapse. 4. First cheap experiment: train a full-batch 2-layer MLP on MNIST and a small ViT on a fixed CIFAR-10 subset. Compare AdamW, damped Newton, and the trust-region optimizer at matched gradient and Hessian-vector-product budgets. Track loss versus wall-clock time and curvature products, rejected-step fraction, radius trajectory, final accuracy, and local convergence after the loss becomes small. Also test an intentionally aggressive initialization or increased curvature to induce instability in fixed-step baselines. Success means fewer catastrophic loss spikes than AdamW, faster loss descent per curvature product, and an approximately superlinear final loss decrease; failure is persistent radius collapse or inferior descent at equal compute.