Clipped-Difference Stochastic DEQ Solver

Integrate the method into the forward fixed-point solve of a stochastic deep-equilibrium layer. Let the implicit map be $T_\theta(x;\xi)=f_\theta(x,u;\xi)$, where $u$ is the layer input and $\xi$ is a dropout mask, sampled minibatch element, augmentation, or Monte Carlo random variable. Modify only the tensor used by the solver initially; keep the DEQ loss and backward pass unchanged. Initialize $x=x_{anchor}$ and compute $v$ as a minibatch mean of $T_\theta(x;\xi)$. For each iteration, set $x_{old}=x$, propose $x_{new}=(1-\alpha)x_{anchor}+\alpha v$, then draw one shared minibatch and compute $d=B^{-1}\sum_b[T_\theta(x_{new};\xi_b)-T_\theta(x_{old};\xi_b)]$. Apply $d\leftarrow d\min(1,\gamma\|x_{new}-x_{old}\|/(\|d\|+10^{-12}))$, update $v\leftarrow v+d$, set $x\leftarrow x_{new}$, and stop when $\|v-x\|\leq\epsilon$. Every $m$ iterations, refresh $v$ using a larger independent anchor minibatch to limit accumulated estimator drift. Shared randomness between the two evaluations is essential for cancellation. Estimate $\gamma$ on calibration batches using $\|T(x+\eta)-T(x)\|/\|\eta\|$ or a Jacobian power iteration, and sweep conservative values such as $0.8,0.9,0.99$. The paper provides the clipping and concentration mechanism; batch size, damping, refresh period, and local $\gamma$ are empirical choices. First test a small DEQ MLP on CIFAR-10 against standard damped fixed-point iteration with independent minibatch evaluations, using equal oracle calls. Add Student-t or lognormal perturbations with finite variance to expose the proposed advantage. Record residual versus oracle calls, solver divergence rate, final task loss, and wall-clock time. Success is lower residual and fewer failed solves at equal compute, especially under heavy-tailed noise, without degrading downstream accuracy.