Unverified 2026

Projected Memory Optimizer

Implementation & benchmark of arXiv:2607.20134 — Derivation of the Boltzmann equation with no "molecular chaos"-type approximation

Usefulness6/10

Difficulty6/10

Novelty6/10

Dynamics Stability Speed Generalization Optimizer Training dynamics Scheduler Stochastic processes Dynamical systems Control theory Statistical mechanics

Source paper: Derivation of the Boltzmann equation with no "molecular chaos"-type approximation arXiv:2607.20134 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the assumption of independent gradient noise with a projected generalized Langevin update containing a short finite-memory correction. The correction models correlations caused by data reuse, augmentation pipelines, momentum, or distributed-worker synchronization, and is switched off only after the measured correlation time is negligible compared with the parameter-relaxation time.

Formulas

$$\frac{d}{dt}P x(t)=PLP x(t)+\int_{0}^{t}PL\,e^{QL\tau}\,QLP x(t-\tau)\,d\tau+PL\,e^{QLt}Qx(0),\qquad Q=I-P.$$

$$K(\tau)=PL\,e^{QL\tau}QLP,\qquad \|K(\tau)\|\approx K_0e^{-\tau/t_{cor}},$$

$$\theta_{k+1}=\theta_k-\eta g_k-\eta\sum_{j=1}^{M}\widehat K_j g_{k-j},\qquad \widehat K_j\approx K(j\Delta t)\Delta t.$$

Mathematical statement

Let $\theta_t$ be neural-network parameters, $g_t=\nabla_\theta \ell(\theta_t;B_t)$ the mini-batch gradient, $P$ the projection onto retained variables $(\theta,g)$, $Q=I-P$ the unresolved subspace, and $L$ the generator of the full parameter-plus-data sampling dynamics. The Nakajima–Zwanzig construction gives a reduced equation with instantaneous drift $PLP$, memory kernel $K(\tau)=PL\exp(QL\tau)QLP$, and an initial-correlation contribution $PL\exp(QLt)Qx_0$, where $x_0$ is the full initial state. The paper's special projection absorbs the last term into a homogeneous equation. In the optimizer, estimate a short kernel $K(\tau)$ from gradient autocorrelations and use a truncated convolution over $M$ steps. The mechanism predicts that the kernel norm decays on $t_{cor}$, while the parameter trajectory changes on $t_{rel}$; memory correction should matter primarily when $t_{cor}/t_{rel}$ is not small and should become irrelevant when $t_{cor}\ll t_{rel}$.

Implementation notes

1. Integration point: implement this as a wrapper around SGD or Adam, operating on the raw mini-batch gradient before the usual parameter update. Maintain a ring buffer of the last $M$ gradients, preferably after flattening each layer and applying a diagonal, block-diagonal, or low-rank approximation to avoid storing full matrices. Use $u_k=g_k+\sum_{j=1}^{M}\widehat K_jg_{k-j}$, then pass $u_k$ to the base optimizer. 2. Pseudocode: initialize a gradient buffer $G$, kernel coefficients $K_1,\ldots,K_M$, and EMA estimates of gradient autocorrelations. At each step compute $g_k$, update $\widehat C_j\leftarrow(1-\alpha)\widehat C_j+\alpha\langle g_k,g_{k-j}\rangle/(\|g_k\|\|g_{k-j}\|+\epsilon)$, fit $\widehat C_j$ to $c_0e^{-j/t_{cor}}$, set $\widehat K_j=\gamma\widehat C_j$ with a small swept gain $\gamma$, form $u_k=g_k+\sum_j\widehat K_jg_{k-j}$, clip the convolution norm if it exceeds $r\|g_k\|$, and update parameters. 3. Computed from the paper: the projected memory structure, homogeneous treatment of initial correlations, and prediction of decay on $t_{cor}$. Estimated empirically: kernel amplitude $\gamma$, decay time, relaxation time, and any low-rank representation. 4. First cheap experiment: train a 2-layer MLP on MNIST and a small ResNet on CIFAR-10 using SGD, momentum SGD, and the memory optimizer at matched learning rates and compute cost. Deliberately induce correlations using a fixed-order sampler, a small replay buffer, or repeated augmentations. Measure gradient autocorrelation, loss smoothness, update variance, and divergence frequency. The quantitative prediction is that the correction helps most when the measured ratio $t_{cor}/t_{rel}\gtrsim0.1$, while its ablation becomes indistinguishable from SGD when $t_{cor}/t_{rel}\lesssim0.01$; fitting an exponential should yield a kernel whose norm decays approximately as $e^{-t/t_{cor}}$. Sweep $\gamma$ and estimate the spectral radius of the empirical finite-memory update operator: instability should begin near spectral radius one, while norm clipping should move the observed divergence boundary upward.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.