# Fast Feedback-Constrained Optimizer

- ID: 1394
- Canonical URL: https://synthcore.org/idea/1394/fast-feedback-constrained-optimizer
- API JSON: https://synthcore.org/api/idea/1394.json
- API Markdown: https://synthcore.org/api/idea/1394.md
- Verification status: unverified
- Source: [arXiv:2607.25193](https://arxiv.org/abs/2607.25193)
- Category: optimization
- Solves: stability, accuracy, generalization
- ML areas: optimizer, training, regularization
- Math tags: optimization, control-theory, dynamical-systems, linear-algebra
- Ratings: usefulness 5/10; difficulty 5/10; novelty 5/10

## Idea description

Replace penalty-based equality-constrained training with a two-timescale optimizer. A fast variable tracks the normal correction that drives constraint residuals toward zero, while the slow parameter update follows the task gradient projected onto the local constraint tangent space. This should reduce sensitivity to very large penalty weights and preserve feasibility more accurately during training.

## Mathematical statement

The paper studies distributed minimization subject to equality constraints and constructs a feedback-linearized normal correction together with a fast residual-tracking realization. For a neural-network parameter vector $\theta\in\mathbb{R}^p$, let $F(\theta)$ be the task loss, let $h(\theta)\in\mathbb{R}^r$ collect differentiable equality constraints, and let $J(\theta)=\nabla_\theta h(\theta)\in\mathbb{R}^{r\times p}$ be their Jacobian. Assuming $J$ has full row rank locally, define the regularized right pseudoinverse $J_\varepsilon^\dagger=J^\top(JJ^\top+\varepsilon I_r)^{-1}$ and tangent projector $P_\varepsilon=I_p-J_\varepsilon^\dagger J$. Here $I_p$ and $I_r$ are identity matrices, and $\varepsilon>0$ prevents ill-conditioned solves. The feedback-linearized normal correction is $u^*(\theta)=-\alpha J_\varepsilon^\dagger h(\theta)$, where $\alpha>0$ sets the desired residual-correction rate. The ideal constrained flow is $\dot\theta=u^*(\theta)-P_\varepsilon\nabla F(\theta)$: the first term reduces constraint residuals, while the second approximately lies in the tangent space because $JP_\varepsilon\approx0$. The nonlocal algebraic correction is replaced by a fast tracker $z$ satisfying $\dot z=\beta(u^*(\theta)-z)$, with $\beta\gg1$ the fast timescale, and the parameter dynamics become $\dot\theta=z-P_\varepsilon\nabla F(\theta)$.

## Key formulas

- $$\min_{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\in\mathbb{R}^{d}}\sum_{i=1}^{N}f_i(\mathbf{x}_i)\quad\mathrm{s.t.}\quad \mathbf{x}_i=\mathbf{x}_j,\ \forall i,j,\qquad \mathbf{h}_i^{\mathrm d}(\mathbf{x}_i)=\mathbf{0}.$$
- $$J_\varepsilon^\dagger(\theta)=J(\theta)^\top\bigl(J(\theta)J(\theta)^\top+\varepsilon I_r\bigr)^{-1},\qquad P_\varepsilon(\theta)=I_p-J_\varepsilon^\dagger(\theta)J(\theta).$$
- $$u^*(\theta)=-\alpha J_\varepsilon^\dagger(\theta)h(\theta),\qquad \dot z=\beta\bigl(u^*(\theta)-z\bigr),\qquad \dot\theta=z-P_\varepsilon(\theta)\nabla F(\theta).$$
- $$z_{k+1}=z_k+\eta_z\bigl(u^*(\theta_k)-z_k\bigr),\qquad \theta_{k+1}=\theta_k+\eta_\theta\bigl[z_k-P_\varepsilon(\theta_k)\widehat{\nabla F}(\theta_k)\bigr],\qquad \eta_z\gg\eta_\theta.$$

## Implementation notes

(1) Integrate this at the optimizer-update stage, replacing AdamW's parameter step for selected constrained parameter blocks. Suitable first constraints are one scalar constraint per neuron or channel, such as $h_j(\theta_j)=\|\theta_j\|_2^2-c_j$, or a few differentiable conservation or normalization constraints. Do not construct a full Jacobian for unconstrained parameters. For each constrained block, compute the task gradient $g$, residual vector $h$, and Jacobian operators $Jg$ and $J^\top v$. (2) Pseudocode: initialize $z=0$. For each minibatch, compute the task loss, $g=\nabla_\theta F$, $h(\theta)$, and the constraint Jacobian $J$. Solve $(JJ^\top+\varepsilon I_r)v=h$, set $u^*=-\alpha J^\top v$, solve $(JJ^\top+\varepsilon I_r)q=Jg$, and set the projected gradient $Pg=g-J^\top q$. Then update $z\leftarrow z+\eta_z(u^*-z)$ and $\theta\leftarrow\theta+\eta_\theta(z-Pg)$. Apply weight decay only to the tangent component, or disable it for constrained blocks. (3) The paper's mathematics determines the pseudoinverse correction, tangent projection, and two-timescale update. The implementation should estimate the useful ratio $\eta_z/\eta_\theta$ through a small sweep. Monitor the smallest eigenvalue and condition number of $JJ^\top+\varepsilon I_r$; increase $\varepsilon$ when the solve is unstable. Automatic differentiation can provide Jacobian-transpose-vector products, while $Jg$ can be obtained by differentiating the scalar $h(\theta)^\top g$ or by forward-mode autodiff. (4) First experiment: train an MLP and ResNet-18 on CIFAR-10 with per-neuron norm constraints, comparing AdamW, AdamW plus a fixed penalty $\lambda\|h(\theta)\|_2^2$, and this optimizer at matched forward/backward FLOPs. Track training loss, test accuracy, mean absolute constraint violation, condition numbers, and unstable runs across several seeds. The primary success signal is lower constraint violation at equal accuracy without the penalty-induced optimization slowdown; a stronger result is faster loss descent and equal or better test accuracy at the same constraint tolerance.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.