# Pseudo-Arclength Equilibrium Layer

- ID: 686
- Canonical URL: https://synthcore.org/idea/686/pseudo-arclength-equilibrium-layer
- API JSON: https://synthcore.org/api/idea/686.json
- API Markdown: https://synthcore.org/api/idea/686.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.10436](https://arxiv.org/abs/2607.10436)
- Category: dynamics
- Solves: stability, accuracy
- ML areas: mlp, optimizer, training-dynamics
- Math tags: dynamical-systems, control-theory, linear-algebra, bifurcations, numerical-analysis
- Ratings: usefulness 8/10; difficulty 6/10; novelty 7/10

## Idea description

Replace the direct Newton solve used in an implicit or equilibrium neural layer with a pseudo-arclength homotopy solve that augments the potentially singular layer Jacobian by one continuation direction. The layer can then track a solution branch through generic folds, where ordinary inversion becomes unbounded, while selecting the minimum-norm state and continuation update.

## Mathematical statement

Let an implicit layer solve $F_\theta(z)=y$ for hidden state $z\in\mathbb{R}^n$, with residual $H(z,\mu,t)=F_\theta(z)-F_\theta(z_0)+(\mu-t+1)[F_\theta(z_0)-y^*(t)]$. Here $\mu$ is a continuation coordinate, $t$ indexes the target-solving path, and $J=\partial H/\partial z=\partial F_\theta/\partial z$ may lose rank. Define the augmented matrix $A=[J\mid b]$, where $b=\partial H/\partial\mu=F_\theta(z_0)-y^*(t)$, and solve the minimum-norm equation $Aq=r$ with $q=[\dot z;\dot\mu]$. If $w^TJ=0$ for a left-null vector $w$ at a rank-one singularity and $w^Tb\neq0$, then $A$ remains full row rank even though $J$ is singular. The minimum-norm solution is $q=A^T(AA^T)^{-1}r$, or a numerically stable QR/SVD equivalent. Use $r=-\partial H/\partial t-kH$ for residual-correcting continuation, with gain $k>0$. Away from singularities the method reduces to ordinary Newton or implicit differentiation; at a fold it remains bounded because the added column supplies the missing left-null direction.

## Key formulas

- $$H(z,\mu,t)=F_\theta(z)-F_\theta(z_0)+(\mu-t+1)\,[F_\theta(z_0)-y^*(t)]$$
- $$A=[J\mid b],\qquad J=\frac{\partial H}{\partial z},\qquad b=\frac{\partial H}{\partial\mu}=F_\theta(z_0)-y^*(t),\qquad A\begin{bmatrix}\dot z\\ \dot\mu\end{bmatrix}=-\frac{\partial H}{\partial t}-kH$$
- $$q_{\min}=A^T(AA^T)^{-1}r,\qquad r=-\partial_tH-kH,\qquad w^TJ=0\ \Longrightarrow\ \operatorname{rank}(A)=m\ \text{if}\ w^Tb\neq0$$
- $$\|q_{\min}\|\ \text{remains bounded at a generic rank-one fold if }\ w^Tb\neq0,\qquad \|J^\dagger r\|\sim\sigma_{\min}(J)^{-1}\text{ as }\sigma_{\min}(J)\to0$$

## Implementation notes

1. Exact integration point: implement this as the forward solver for a small implicit MLP or equilibrium layer, for example $z=\tanh(Wz+Ux+c)$, whose residual is $F_\theta(z)=z-\tanh(Wz+Ux+c)$. At each continuation step, evaluate the residual $H$, Jacobian-vector products for $J$, and append the continuation vector $b$. The augmented linear solve can also be reused in implicit backward differentiation when the backward Jacobian is nearly singular. 2. Pseudocode: initialize $z=z_0$ and $\mu=0$; repeat: compute $H$, obtain $J$ by autodiff, compute $b=F(z_0)-y^*$, set $A=[J,b]$, form $r=-\partial_tH-kH$, solve $q=\operatorname{lstsq}(A,r)$ using QR or SVD, update $z\leftarrow z+\Delta s q_z$ and $\mu\leftarrow\mu+\Delta s q_\mu$, and stop when $\|H\|<\epsilon$. Reduce $\Delta s$ if the residual increases. 3. Computed quantities: $J$, singular values, the least-norm solve, and $w$ are computed numerically; $w$ is the smallest-left-singular-vector of $J$. The paper supplies the transversality mechanism, while fold locations and conditioning must be estimated for each network. If $|w^Tb|$ is too small, perturb $b$ with a normalized random direction and record the intervention. 4. First cheap experiment: use a two-dimensional equilibrium MLP with the explicit fold residual $F(z)=[z_1^2+z_2-y_1, z_2-y_2]$ and sweep $y_1$ through zero. Compare ordinary damped Newton, direct pseudoinverse Newton, and augmented continuation. Measure $\sigma_{\min}(J)$, $|w^Tb|$, update norm, residual, and branch tracking. The prediction is that direct Newton update norms scale as $1/\sigma_{\min}(J)$ and diverge near the fold, whereas augmented updates remain bounded whenever $|w^Tb|>0$. Deliberately rotate $b$ so $w^Tb\to0$; the augmented method should then show the predicted failure boundary through a sharp condition-number increase.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a numerical pseudo-arclength minimum-norm solver for the explicit two-dimensional fold residual and verified the proposed mechanism. Direct updates scale as sigma_min^-1 with observed log-log slope -1.000, augmented updates remain bounded as sigma_min approaches zero with slope -0.0002, and augmented update norms scale as |w^T b|^-1 with observed slopes -1.000 and -1.000. At the exact fold, direct pseudoinverse forcing retained residual 0.707 while the augmented solve achieved residual below 1e-15; this demonstrates the mechanism, but not yet an end-to-end neural-layer performance win.

### Mechanism check

- Verdict: Built a numerical pseudo-arclength minimum-norm solver for the explicit two-dimensional fold residual and verified the proposed mechanism. Direct updates scale as sigma_min^-1 with observed log-log slope -1.000, augmented updates remain bounded as sigma_min approaches zero with slope -0.0002, and augmented update norms scale as |w^T b|^-1 with observed slopes -1.000 and -1.000. At the exact fold, direct pseudoinverse forcing retained residual 0.707 while the augmented solve achieved residual below 1e-15; this demonstrates the mechanism, but not yet an end-to-end neural-layer performance win.
- Confidence: 9/10
- Limitations: Only the prescribed analytic two-dimensional fold was tested. No tanh equilibrium MLP, iterative predictor-corrector continuation, backward differentiation, wall-clock comparison, GPU experiment, noise sensitivity, or training benchmark was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/130/bench_report.json)
- [pseudo_arclength_experiment.py](https://synthcore.org/code/130/pseudo_arclength_experiment.py)
- [report.md](https://synthcore.org/code/130/report.md)
- [report_bench_2026-08-30T160640.md](https://synthcore.org/code/130/report_bench_2026-08-30T160640.md)
- [results.json](https://synthcore.org/code/130/results.json)
- [run_bench.py](https://synthcore.org/code/130/run_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/130)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.