Barrier-Projected Neural Updates

1. Integration point: replace the raw optimizer velocity in AdamW, SGD, or a custom continuous-time optimizer by a barrier-filtered velocity. Use inexpensive minibatch-estimated barriers: \(h_1=\tau_\theta^2-\|\theta\|_2^2\) for a parameter-norm trust region, \(h_2=\tau_a^2-\operatorname{mean}(a^2)\) for hidden activation energy, and, for attention, \(h_3=\tau_z^2-\operatorname{mean}(z^2)\) for logits. 2. Pseudocode: compute the nominal Adam direction \(v_{nom}\); evaluate each \(h_j\) and its parameter gradient \(g_j=\nabla_\theta h_j\) using autodiff; solve the QP \(\min_v \frac12\|v-v_{nom}\|^2\) subject to \(g_j^Tv+\alpha_jh_j\ge0\); update \(\theta\leftarrow\theta+\eta v\). For an MVP, solve the QP with an active-set loop: start with violated constraints, solve the equality-constrained least-squares system, remove constraints with negative multipliers, and add constraints violated by the candidate. 3. Computed from the mechanism: the barrier inequality, KKT multipliers, active set, and projected update. Estimated empirically: barrier gradients from minibatches, curvature or Taylor remainder, and suitable thresholds \(\tau_j\). 4. First experiment: train a 2-layer MLP on MNIST and a small Transformer on a synthetic long-context copy task, comparing AdamW with and without the QP filter. Sweep step size \(\eta\) and barrier rate \(\alpha\). The quantitative prediction is that barrier violations begin near the explicit-Euler boundary \(\eta\alpha=1\), while for \(\eta\alpha\le0.5\), initially safe barriers should remain nonnegative apart from minibatch and second-order errors. The fraction of projected steps should sharply increase when the nominal update points outside the safe set. Log this transition, KKT residual, loss, gradient norm, and activation overflow rate. A useful acceptance criterion is a measured safety boundary within 20% of the predicted \(\eta_c\approx1/\alpha\), with substantially fewer activation or parameter-norm excursions at matched training loss.