# Delayed hysteretic residual mixer

- ID: 330
- Canonical URL: https://synthcore.org/idea/330/delayed-hysteretic-residual-mixer
- API JSON: https://synthcore.org/api/idea/330.json
- API Markdown: https://synthcore.org/api/idea/330.md
- Verification status: unverified
- Source: [arXiv:2607.01386](https://arxiv.org/abs/2607.01386)
- Category: dynamics
- Solves: stability, accuracy, speedup
- ML areas: training-dynamics, rnn, ssm, moe-routing
- Math tags: dynamical-systems, control-theory, bifurcations, stability
- Ratings: usefulness 5/10; difficulty 5/10; novelty 8/10

## Idea description

Insert a two-mode residual mixer whose mode is selected by a delayed sign variable rather than an instantaneous sign or sigmoid. The delayed mode creates a hysteresis-like effect that prevents high-frequency switching when the latent state is close to the decision surface, while the paper's reduced equations provide a constraint for choosing the delay and mixing strength so the latent energy contracts.

## Mathematical statement

The nonholonomic controller uses the state q=(x,y)^T and binary mode s=sign(z), with J=[[0,1],[-1,0]] and delayed feedback qdot=-q_tau+alpha J q_tau sign(z_tau). Here tau>0 is the delay, alpha is the mode-mixing strength, and q_tau=q(t-tau). The paper's first-order delay approximation x(t-tau) approximately x(t)-tau xdot(t) yields the quadratic-energy law Vdot=2((alpha^2 tau+tau-1)/((1-tau)^2+alpha^2 tau^2))V, where V=||q||^2. Thus the approximate contraction condition is alpha^2 tau+tau-1<0, equivalently tau(1+alpha^2)<1. The associated scalar dynamics are zdot=(-2 alpha/((1-tau)^2+alpha^2 tau^2))V, and the predicted attraction region is -1/2 alpha/(alpha^2 tau+tau-1)||q||^2<|z|. The neural adaptation uses q as a two-channel latent residual, s as a hard mode, and z as a learned scalar mode margin; the engineering goal is to exploit delayed sign changes as hysteresis, not to reproduce the nonholonomic plant exactly.

## Key formulas

- $$\dot{x}=-x_{\tau}+\alpha y_{\tau}\operatorname{sign}(z_{\tau}),\qquad \dot{y}=-y_{\tau}-\alpha x_{\tau}\operatorname{sign}(z_{\tau})$$
- $$\dot{V}=2\left(\frac{\alpha^{2}\tau+\tau-1}{(1-\tau)^{2}+\alpha^{2}\tau^{2}}\right)V,\qquad V=x^{2}+y^{2}$$
- $$\dot{z}=\frac{-2\alpha}{(1-\tau)^{2}+\alpha^{2}\tau^{2}}V,\qquad -\frac{1}{2}\frac{\alpha}{\alpha^{2}\tau+\tau-1}(x^{2}+y^{2})<|z|$$
- $$q_{k+1}=q_k+h\left[-q_{k-d}+\alpha Jq_{k-d}s_{k-d}\right],\quad s_k=\operatorname{sign}(z_k),\quad J=\begin{bmatrix}0&1\\-1&0\end{bmatrix},\quad d=\lfloor\tau/h\rfloor$$

## Implementation notes

Integrate this at the residual update of a small recurrent or state-space network, not inside the feed-forward MLP. For each token or time step, maintain a two-channel residual state q_k in R^2 per feature group, a scalar margin z_k, and a FIFO buffer containing the previous d states and signs. Let r_k be the ordinary block output projected to two channels. Compute q_del=q_{k-d} and s_del=s_{k-d}, then update q with q_next=q_k+h*(-q_del+alpha*J*q_del*s_del), exactly matching the adapted delayed equation. Add r_k to the remaining hidden state through a learned projection of q_next. Define z_k as a learned scalar readout of the current residual mismatch, for example z_k=w^T(r_k-q_k)+b. During training, use s=sign(z) in the forward pass and a straight-through gradient s_grad=tanh(z/epsilon) in the backward pass. Initialize tau and alpha so tau*(1+alpha^2)<0.8, then optionally learn them as tau=sigmoid(rho)*tau_max and alpha=softplus(a), rejecting or projecting parameter updates that violate the contraction constraint. The paper's formulas provide the delay discretization and energy-rate prediction; estimate the actual latent-energy slope and switching rate empirically. First test on a recurrent sequential-MNIST or row-wise CIFAR-10 model, comparing an ordinary residual RNN, an instantaneous sign gate, and this delayed mixer at equal hidden size and FLOPs. Measure validation accuracy, gradient-norm variance, mode switches per sequence, and robustness to activation noise. Success means fewer switches and lower gradient variance without slower loss descent, or equal accuracy with a longer stable recurrent rollout.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.