# Dry-Friction Active Optimizer

- ID: 2725
- Canonical URL: https://synthcore.org/idea/2725/dry-friction-active-optimizer
- API JSON: https://synthcore.org/api/idea/2725.json
- API Markdown: https://synthcore.org/api/idea/2725.md
- Verification status: unverified
- Source: [arXiv:2608.24689](https://arxiv.org/abs/2608.24689)
- Category: optimization
- Solves: stability, generalization, speedup
- ML areas: optimizer, training-dynamics, regularization
- Math tags: dynamical-systems, stochastic-processes, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Replace the usual momentum state in an optimizer with a persistent Ornstein-Uhlenbeck-driven velocity subject to a dry-friction threshold. Correlated forcing can help traverse shallow noisy regions, while the friction term suppresses parameter motion when the effective force is small, potentially reducing update noise and improving late-stage stability.

## Mathematical statement

The paper's jerk equation is \(\lambda\dddot{\mathbf r}=\mathbf F(\mathbf r,\dot{\mathbf r},\ddot{\mathbf r})\), where \(\mathbf r\) is position, \(\lambda\) is the jerk coefficient, and \(\mathbf F\) is a generalized force; under constant force and rest initial conditions it gives \(v(t)\sim t^2\) and displacement \(\delta r(t)\sim t^3\). Its implementable stochastic model uses an active Ornstein-Uhlenbeck process \(\dot{\mathbf n}(t)=-\mathbf n(t)/\tau+\sqrt{2/\tau}\,\boldsymbol\eta(t)\), where \(\mathbf n\) is a dimensionless correlated forcing state, \(\tau>0\) is its persistence time, and \(\boldsymbol\eta\) is delta-correlated Gaussian white noise. The velocity dynamics are \(m\dot{\mathbf v}=-\Delta_F\hat{\mathbf v}+\mathbf f_A+\sqrt{2K}\boldsymbol\xi\), where \(m\) is inertial mass, \(\Delta_F\ge0\) is the dry-friction threshold, \(\hat{\mathbf v}=\mathbf v/\|\mathbf v\|\), \(\mathbf f_A=f_A\mathbf n\) is the active force with amplitude \(f_A\), and \(K\) controls additional white noise. For optimization, set position \(\mathbf r=\theta\), force \(\mathbf F=-g\) with gradient \(g=\nabla_\theta L\), and use the smooth zero-safe friction direction \(\mathbf v/(\|\mathbf v\|+\epsilon)\). The friction threshold creates a dead-zone in which small effective forces do not produce large parameter motion.

## Key formulas

- $$\lambda\dddot{\mathbf r}=\mathbf F(\mathbf r,\dot{\mathbf r},\ddot{\mathbf r}),\qquad v(t)\sim t^2,\qquad \delta r(t)\sim t^3.$$
- $$\dot{\mathbf n}(t)=-\frac{\mathbf n(t)}{\tau}+\sqrt{\frac{2}{\tau}}\,\boldsymbol\eta(t).$$
- $$m\dot{\mathbf v}(t)=-\Delta_F\hat{\mathbf v}+f_A\mathbf n(t)+\sqrt{2K}\,\boldsymbol\xi(t),\qquad \hat{\mathbf v}=\frac{\mathbf v}{\|\mathbf v\|}.$$
- $$n_{k+1}=\left(1-\frac{h}{\tau}\right)n_k+\sqrt{\frac{2h}{\tau}}\,\zeta_k,\quad v_{k+1}=v_k+\frac{h}{m}\left[-g_k-\Delta_F\frac{v_k}{\|v_k\|+\epsilon}+f_A n_{k+1}\right],\quad \theta_{k+1}=\theta_k+h v_{k+1}.$$

## Implementation notes

(1) Integrate this into the optimizer update after each minibatch gradient is computed and before parameters are changed. Maintain one velocity tensor \(v\) and one Ornstein-Uhlenbeck tensor \(n\) for every trainable parameter tensor \(\theta\). Use a global or per-layer mass \(m\), persistence time \(\tau\), active amplitude \(f_A\), friction threshold \(\Delta_F\), and integration step \(h\). Initially omit the paper's optional white-noise term because minibatch sampling already supplies noise. (2) Pseudocode: `g = grad(loss, theta)`; `n = (1-h/tau)*n + sqrt(2*h/tau)*NormalLike(n)`; `direction = v/(sqrt(sum(v*v))+eps)`; `v = v + h/m*(-g - DeltaF*direction + fA*n)`; `theta = theta + h*v`. For a scalar-per-coordinate dead-zone variant, replace the friction term with the proximal rule `v = sign(v)*max(abs(v)-h*DeltaF/m, 0)`. Apply weight decay separately and decoupled, so friction is tested independently. (3) The OU recursion and friction update come directly from the paper; \(m,\tau,f_A,\Delta_F\) are tuned hyperparameters, while gradients and minibatch noise are measured empirically. Log gradient norm, velocity norm, update variance, and the fraction of coordinates whose velocity is effectively zero. (4) First run a 2D quadratic sanity test with known optimum, then train a 2-layer MLP on MNIST or CIFAR-10 against tuned SGD with momentum and AdamW at equal optimizer steps and matched learning-rate search budgets. Sweep \(\Delta_F\) over \(10^{-4},10^{-3},10^{-2}\) times the median gradient norm and \(\tau\) over 1, 10, and 100 steps. Success means lower validation loss or higher accuracy at equal steps, at least 10% lower late-stage update variance, fewer unnecessary parameter updates, and no increase in gradient-norm spikes.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.