# Passivity-Certified Softmax Optimizer

- ID: 2338
- Canonical URL: https://synthcore.org/idea/2338/passivity-certified-softmax-optimizer
- API JSON: https://synthcore.org/api/idea/2338.json
- API Markdown: https://synthcore.org/api/idea/2338.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.15308](https://arxiv.org/abs/2608.15308)
- Category: dynamics
- Solves: stability, accuracy
- ML areas: optimizer, attention, moe-routing, training-dynamics
- Math tags: control-theory, dynamical-systems, optimization, linear-algebra, geometry
- Ratings: usefulness 8/10; difficulty 5/10; novelty 7/10

## Idea description

Replace direct logit gradient updates for a simplex-valued neural module with a cascade consisting of a passive LTI filter followed by softmax. The filter can provide useful memory or momentum, but its transfer function is constrained to remain strictly passive, preventing the destabilization mechanism identified for nonpassive higher-order replicator dynamics.

## Mathematical statement

Let v be logits in R^n, x=sigma(v) the softmax vector, and g(x) the payoff signal, chosen for optimization as g(x)=-nabla_x L(x) or as a payoff vector. Use an auxiliary stable LTI system dot(z)=Az+Bg and y=Cz+Dg, followed by dot(v)=y. Its transfer function is H(s)=C(sI-A)^(-1)B+D. The relevant passivity condition is strict positive realness: H(j omega)+H(j omega)^* is at least 2 epsilon I for every real frequency omega and some epsilon>0, together with Hurwitz A. A continuous-time KYP certificate seeks P=P^T>0, Q>0, and epsilon>0 satisfying the displayed matrix inequality. The softmax differential is dot(x)=J(x)y, where J(x)=diag(x)-xx^T and J(x) is positive semidefinite on the tangent space Z={q:1^Tq=0}. For a strictly contractive game or locally strongly convex loss, the passive cascade supplies a storage function whose derivative is negative up to equilibrium error; violation of positive realness predicts that a contractive problem can become unstable.

## Key formulas

- $$\dot z=Az+Bg,\qquad y=Cz+Dg,\qquad \dot v=y,\qquad x=\sigma(v),\quad \sigma(v)_i=\frac{e^{v_i}}{\sum_j e^{v_j}}.$$
- $$H(s)=C(sI-A)^{-1}B+D,\qquad H(j\omega)+H(j\omega)^{\ast}\succeq 2\epsilon I\quad\forall\omega\in\mathbb{R},\ \epsilon>0.$$
- $$\begin{bmatrix}A^{\top}P+PA+Q&PB-C^{\top}\\B^{\top}P-C&-(D+D^{\top})+2\epsilon I\end{bmatrix}\preceq0,\qquad P\succ0,\ Q\succ0.$$
- $$\dot x=\bigl(\operatorname{diag}(x)-xx^{\top}\bigr)y=J(x)y,\qquad J(x)\succeq0\ \text{on}\ \mathcal Z=\{q:\mathbf{1}^{\top}q=0\}.$$

## Implementation notes

1. Integration point: use this module wherever a network currently maintains simplex weights, especially MoE router probabilities, attention mixtures, differentiable architecture choices, or learned convex combinations of experts. Maintain one filter state z per routed token and route; the forward pass uses x=softmax(v), while the backward signal produces g=-gradient with respect to x. 2. Pseudocode: initialize v and z; at each minibatch compute x=softmax(v), loss L, and g=-stop_gradient(grad_x(L)); update z <- z + dt*(A z + B g), then update v <- v + dt*(C z + D g), while optimizing ordinary network weights with Adam. Parameterize A as A=-(R R^T+alpha I) to make it Hurwitz, then solve or check the KYP LMI for P,Q,epsilon. Reject filter parameters if the LMI is infeasible. For a cheap scalar or diagonal implementation use H_i(s)=d_i+k_i/(tau_i s+1), with tau_i>0, k_i>=0, and d_i>=epsilon. 3. Computed from the paper: the cascade, softmax Jacobian, positive-real/passivity condition, and KYP certificate. Estimated empirically: local Jacobians of the loss or game, the finite-frequency minimum eigenvalue of H(j omega)+H(j omega)^*, and the observed growth rate of perturbations. 4. First experiment: train a two-expert MoE MLP on MNIST or CIFAR-10 with direct SGD logits, unconstrained momentum filtering, and the passive filter. Sweep filter gain and timestep. The predicted signature is a sharp stability boundary when m=min_omega lambda_min((H(j omega)+H(j omega)^*)/2) crosses zero: for m>0, router-logit perturbations should decay and routing entropy should settle; for m<0, a suitably coupled contractive quadratic toy loss should show growing oscillations or divergence. In the scalar first-order case H(s)=d+k/(tau s+1), the boundary is k=0 when d=0, and the measured decay rate should decrease continuously as m approaches zero. Compare the empirical boundary with the KYP or frequency estimate within approximately 20 percent.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a scalar first-order passive-filter softmax optimizer toy with frequency-domain passivity, analytic characteristic roots, nonlinear simulation, softmax-Jacobian checks, and an analytic KYP certificate. The predicted boundary k=-d=-0.2 matched the refined linear boundary (-0.200000000001), with instability for k<-d and decay for k>-d; measured growth closely tracked linear predictions for Lambda up to 3.0. The mechanism manifested clearly, although the passive filter was slower than direct H=1 gradient flow in this setup, so this is not evidence of an optimization-speed win.

### Mechanism check

- Verdict: Built a scalar first-order passive-filter softmax optimizer toy with frequency-domain passivity, analytic characteristic roots, nonlinear simulation, softmax-Jacobian checks, and an analytic KYP certificate. The predicted boundary k=-d=-0.2 matched the refined linear boundary (-0.200000000001), with instability for k<-d and decay for k>-d; measured growth closely tracked linear predictions for Lambda up to 3.0. The mechanism manifested clearly, although the passive filter was slower than direct H=1 gradient flow in this setup, so this is not evidence of an optimization-speed win.
- Confidence: 9/10
- Limitations: Only a low-dimensional continuous-time quadratic toy was tested. No MNIST/CIFAR MoE, neural-network training, minibatch noise, discrete-time implementation, multidimensional coupled filters, numerical SDP KYP solver, wall-clock/FLOP comparison, or robustness study was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_passivity.py](https://synthcore.org/code/807/bench_passivity.py)
- [bench_report.json](https://synthcore.org/code/807/bench_report.json)
- [passivity_experiment.py](https://synthcore.org/code/807/passivity_experiment.py)
- [report.md](https://synthcore.org/code/807/report.md)
- [report_bench_2026-09-01T110457.md](https://synthcore.org/code/807/report_bench_2026-09-01T110457.md)
- [results.json](https://synthcore.org/code/807/results.json)
- [Download all files as ZIP](https://synthcore.org/download/807)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.