# Pointwise Clarke-Tangent Training

- ID: 389
- Canonical URL: https://synthcore.org/idea/389/pointwise-clarke-tangent-training
- API JSON: https://synthcore.org/api/idea/389.json
- API Markdown: https://synthcore.org/api/idea/389.md
- Verification status: unverified
- Source: [arXiv:2607.03195](https://arxiv.org/abs/2607.03195)
- Category: geometry
- Solves: stability, accuracy, generalization
- ML areas: optimizer, training, world-model
- Math tags: geometry, functional-analysis, optimization, measure-theory
- Ratings: usefulness 5/10; difficulty 6/10; novelty 6/10

## Idea description

Train a neural function under hard pointwise constraints by projecting its desired output-space update into the Clarke tangent cone of the admissible set at every sampled input. Fit the resulting feasible measurable direction with a parameter update instead of repeatedly allowing the network to violate constraints and repairing it with a penalty.

## Mathematical statement

The paper defines the decomposable selection set $\mathsf{Sel}_{p}(S)=\{x\in L^{p}(T,Z):x(t)\in S(t)\ \text{a.e.}\}$, where $(T,\Sigma,\mu)$ is a complete $\sigma$-finite measure space, $Z$ is a separable Banach space, $S:T\rightrightarrows Z$ is measurable with nonempty closed values, and $p\in[1,\infty)$. Its central result is $\widehat{\mathbf T}_{\mathsf{Sel}_{p}(S)}(x)=\{v\in L^{p}(T,Z):v(t)\in\widehat{\mathbf T}_{S(t)}(x(t))\ \text{a.e.}\}$. Here $x$ is the current feasible function, $v$ is a global function-space direction, $\widehat{\mathbf T}_{C}(u)$ is the regular/Clarke tangent cone to a closed set $C$ at $u$, and $\widehat{\mathbf T}_{S(t)}(x(t))$ is the pointwise cone. A usable distance characterization is $v\in\widehat{\mathbf T}_{C}(u)$ when $\lim_{h\downarrow0}\operatorname{dist}(u+h v,C)/h=0$. Thus, for finite $p$, local pointwise tangent membership is exactly the first-order feasibility condition for the entire selection-valued neural function. In a minibatch, replace the almost-everywhere condition by tangent membership at every sampled point and approximate the $L^p$ norm by a batch average.

## Key formulas

- $$\mathsf{Sel}_{p}(S):=\{x\in L^{p}(T,Z):x(t)\in S(t)\ \text{a.e.}\}.$$
- $$\widehat{\mathbf T}_{\mathsf{Sel}_{p}(S)}(x)=\{v\in L^{p}(T,Z):v(t)\in\widehat{\mathbf T}_{S(t)}(x(t))\ \text{a.e.}\}.$$
- $$\widehat{\mathbf T}_{C}(u)\approx\left\{v:\lim_{h\downarrow0}\frac{\operatorname{dist}(u+h v,C)}{h}=0\right\},\qquad v_i^{\mathrm{tan}}=\operatorname*{argmin}_{v\in\widehat{\mathbf T}_{S(t_i)}(y_i)}\|v-g_i\|_2^2.$$
- $$\Delta\theta=\operatorname*{argmin}_{\delta\theta}\sum_{i=1}^{B}\left\|J_{\theta}(t_i)\delta\theta-v_i^{\mathrm{tan}}\right\|_2^2+\lambda\|\delta\theta\|_2^2,\qquad \theta\leftarrow\theta+\eta\Delta\theta.$$

## Implementation notes

(1) Integration point: apply the method at the output of a neural operator, dynamics model, or constrained prediction head. For each minibatch input $t_i$, compute $y_i=f_\theta(t_i)$ and define a closed admissible set $S(t_i)$. Start with boxes, simplices, nonnegative physical quantities, or norm-bounded controls. Compute the ordinary output-space descent direction $g_i=-\nabla_{y_i}\ell_i$ while holding the network parameters fixed. (2) Pseudocode: `y=f(theta,t); loss=task_loss(y,target); g=-grad_y(loss); for i in batch: C=S(t[i]); v[i]=project_to_tangent_cone(g[i], C, y[i]); J[i]=jvp(f,theta,t[i]); delta_theta=solve_ridge(J_stack,v_stack,lambda); theta += eta*delta_theta`. For a box $C=[a,b]^d$, use $v_j=\min(g_j,0)$ at $y_j=a_j$, $v_j=\max(g_j,0)$ at $y_j=b_j$, and $v_j=g_j$ in the interior. For a simplex, impose $\mathbf 1^Tv=0$ and require $v_j\ge 0$ on active zero coordinates, solving a small quadratic program. For a general set, approximate tangent projection by minimizing $\|v-g\|^2+\rho\,\operatorname{dist}(y+\varepsilon v,C)^2$ for small decreasing $\varepsilon$. (3) The theorem supplies the pointwise-to-global tangent principle; Jacobian-vector products, cone projections, ridge regularization, and step-size selection are empirical approximations. Use JVPs or implicit least-squares iterations rather than materializing the Jacobian, and line-search or clip $\eta$ because tangent feasibility is only first order. (4) First experiment: train a small MLP on synthetic regression with simplex-valued or box-constrained outputs, comparing Adam with a violation penalty, ordinary projected outputs, and tangent training at equal FLOPs. Record task loss, fraction of violated coordinates, maximum violation, and loss decrease per optimizer step. Success means lower violation at equal loss or faster loss descent without penalty-induced oscillations.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.