Bidirectional Saturation-Aware Trust Region

Integrate the mechanism after gradient computation and before the optimizer update, initially using SGD or Adam with the proposed Adam-preconditioned step treated as \(p_t\). Maintain one scalar radius per parameter group, or one radius per layer. Pseudocode: compute minibatch gradient \(g_t\); form the optimizer proposal \(p_t\); compute \(q_t=\min(1,\|p_t\|/(r_t+\varepsilon))\); apply \(\Delta\theta_t=p_t\min(1,r_t/(\|p_t\|+\varepsilon))\); update parameters; then set \(r_{t+1}=\operatorname{clip}(r_t\exp(\kappa_+q_t-\kappa_-(1-q_t)),r_{\min},r_{\max})\). Use an exponential moving average of \(q_t\) with time constant 20--100 steps to suppress minibatch noise. The paper supplies the bidirectional logic: relax during active saturation, recover after saturation, and tighten during sustained inactivity. The curvature estimate \(\widehat L_t\) and the smooth-loss inequality are estimated or tested empirically. First test on a 3-layer MLP and ResNet-18 using MNIST or CIFAR-10, comparing fixed clipping, AdamW, and ordinary trust-region clipping. Initialize \(\kappa_+=0.02\), \(\kappa_-=0.01\), \(r_{\min}=0.1r_0\), and \(r_{\max}=10r_0\). Log \(q_t\), radius, loss, gradient norm, and effective step size. The quantitative prediction is that clipping frequency falls after an expansion transient, followed by radius contraction when the moving-average saturation falls below approximately \(\kappa_-/(\kappa_++\kappa_-)\); with these gains the threshold is about 1/3. In locally quadratic regions, divergence should begin near effective step size \(2/\widehat L_t\), while setting \(c=1\) should keep the cap below that boundary and reduce exploding-loss runs relative to uncapped Adam.