# Conservative-field gradient envelope

- ID: 888
- Canonical URL: https://synthcore.org/idea/888/conservative-field-gradient-envelope
- API JSON: https://synthcore.org/api/idea/888.json
- API Markdown: https://synthcore.org/api/idea/888.md
- Verification status: unverified
- Source: [arXiv:2607.13973](https://arxiv.org/abs/2607.13973)
- Category: optimization
- Solves: stability, accuracy, generalization
- ML areas: optimizer, moe-routing, training-dynamics, loss
- Math tags: optimization, dynamical-systems, geometry, analysis
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Replace the single arbitrary autodiff derivative at a piecewise-smooth interface with a sampled conservative-field gradient envelope. For each minibatch and parameter point, collect gradients from locally reachable branches, average them as a convex combination, and use the resulting direction in a stochastic update. This is intended for architectures with routing, clipping, hard masks, or custom continuous branching where ordinary autodiff can select an unstable branch.

## Mathematical statement

The paper uses a finite selection representation $f(x)=f_{s(x)}(x)=\sum_{i=1}^{N}\mathbbm{1}_{\mathcal A_i}(x)f_i(x)$, where $\{\mathcal A_i\}_{i=1}^N$ are branch regions and every branch $f_i$ is $C^p$. Continuity requires branch values to agree on shared interfaces. The associated-gradient set is $D_f(x)=\{\nabla f_i(x):x\in\overline{\mathcal A_i}\}$, where $\overline{\mathcal A_i}$ is the closure of the region on which branch $i$ is active. The conservative-field chain rule states that for a Lipschitz curve $\gamma$, $(f\circ\gamma)'(t)=\langle v,\dot\gamma(t)\rangle$ for almost every $t$ and every $v\in D_f(\gamma(t))$. The Clarke subdifferential is obtained from the convex envelope, $\partial^C f(x)=\operatorname{conv}D_f(x)$. For neural-network parameters $\theta$, estimate several elements $g_{k,j}\in D_L(\theta_k)$ by replaying nearby branch configurations, choose $\bar g_k=\sum_jw_{k,j}g_{k,j}$ with $w_{k,j}\ge0$ and $\sum_jw_{k,j}=1$, and update $\theta_{k+1}=\theta_k-\eta_k\bar g_k$. A Robbins–Monro-style schedule uses $\sum_k\eta_k=\infty$ and $\sum_k\eta_k^2<\infty$, for example $\eta_k=\eta_0/(k+1)^{0.6}$.

## Key formulas

- $$f(x)=f_{s(x)}(x)=\sum_{i=1}^{N}\mathbbm{1}_{\mathcal A_i}(x)f_i(x),\qquad \mathbbm{1}_{\mathcal A_i}(x)=\begin{cases}1,&x\in\mathcal A_i,\\0,&x\notin\mathcal A_i.\end{cases}$$
- $$D_f(x)=\{\nabla f_i(x):x\in\overline{\mathcal A_i}\},\qquad \partial^C f(x)=\operatorname{conv}D_f(x)$$
- $$(f\circ\gamma)'(t)=\langle v,\dot\gamma(t)\rangle\quad\text{for a.e. }t\text{ and every }v\in D_f(\gamma(t))$$
- $$\theta_{k+1}=\theta_k-\eta_k\bar g_k,\qquad \bar g_k=\sum_{j=1}^{r_k}w_{k,j}g_{k,j},\quad g_{k,j}\in D_L(\theta_k),\quad w_{k,j}\ge0,\quad\sum_jw_{k,j}=1$$

## Implementation notes

Integrate this at the parameter-gradient stage of a network whose forward pass contains continuous piecewise-smooth operations such as ReLU, clamp, top-k routing, mixture-of-experts gates, or custom branches. The scalar minibatch loss is $L(\theta)$. Run the normal forward pass and record branch predicates and near-interface tensors. Define a near-interface event using a margin threshold $\tau$: for ReLU use $|z|<\tau$, and for routing use the top-two logit gap $|a_{(1)}-a_{(2)}|<\tau$. Pseudocode: `loss = forward(theta, mode='base'); G = [backward(loss)]`; for `m=1,...,M`, perturb only near-interface preactivations with $\epsilon\sim\mathrm{Uniform}[-\tau,\tau]$, replay the branch decisions, backpropagate the perturbed computation, and append the resulting gradient `g_m` to `G`; clip each gradient to a common norm; compute `gbar = sum(w_m*g_m)` with uniform weights or replay-frequency weights; update `theta = theta - eta_k*gbar`. Use $\eta_k=\eta_0/(k+1)^\alpha$ with $0.5<\alpha\le1$ for the conservative-field experiment, and compare against a constant-step AdamW control. The paper supplies $D_L$, its convex hull, and the pathwise chain rule; the implementation estimates $D_L$ through finite branch replay and estimates interface frequency empirically. Log gradient radius $R_k=\max_m\|g_m-\bar g\|$, pairwise cosine similarity, near-interface fraction, loss spikes, and validation accuracy. First test a 2-layer ReLU MLP on MNIST or CIFAR-10, then a small continuous top-2 MoE. Compare ordinary autodiff, standard subgradient SGD, and the envelope update at equal total backward FLOPs. Success is lower interface-associated gradient variance, fewer loss spikes, and improved validation accuracy or stability at equal compute; failure is no gain after accounting for the extra backward passes.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.