# Dissipative Completely-Monotone Memory Layer

- ID: 830
- Canonical URL: https://synthcore.org/idea/830/dissipative-completely-monotone-memory-layer
- API JSON: https://synthcore.org/api/idea/830.json
- API Markdown: https://synthcore.org/api/idea/830.md
- Verification status: beats_baseline
- Source: [arXiv:2607.12871](https://arxiv.org/abs/2607.12871)
- Category: dynamics
- Solves: stability, accuracy, scalability
- ML areas: ssm, rnn, training-dynamics, initialization
- Math tags: measure-theory, functional-analysis, dynamical-systems, convex-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Replace an unconstrained recurrent or state-space transition with a finite quadrature of completely monotone memory modes. Couple the visible state and memory states as adjoint operators, so their cross terms cancel in the energy derivative and the layer is contractive even when visible-state damping is zero.

## Mathematical statement

The paper studies the memory equation $\partial_tu(t)+\mathsf{A}_0u(t)+\mathsf{A}_1(k*u)(t)=f(t)$, where $u$ is the physical state, $\mathsf{A}_0$ and $\mathsf{A}_1$ are operators induced by symmetric bilinear forms, and $k$ is completely monotone. A completely monotone kernel has the Bernstein representation $k(t)=\int_{[0,\infty)}e^{-\lambda t}\,d\nu(\lambda)$, with nonnegative measure $\nu$, total mass $M_0=\nu([0,\infty))$, and first moment $M_1=\int\lambda\,d\nu(\lambda)$. Introduce one memory state $z_\lambda$ per decay rate, satisfying $\dot z_\lambda(t)=-\lambda z_\lambda(t)+Bx(t)$; then the memory convolution is represented by an integral of $z_\lambda$. The neural adaptation uses a finite positive quadrature $\nu\approx\sum_{j=1}^m w_j\delta_{\lambda_j}$, with $w_j\ge0$ and $\lambda_j\ge0$. The coupled dynamics are $\dot x=-Qx-B^\top\sum_jw_jz_j+Wr$ and $\dot z_j=Bx-\lambda_jz_j$, where $Q\succeq0$ is optional visible-state damping, $x\in\mathbb{R}^d$ is the layer state, $z_j\in\mathbb{R}^p$ are memory states, and $r$ is the input. For $E=\frac12\|x\|^2+\frac12\sum_jw_j\|z_j\|^2$, the homogeneous system obeys $\dot E=-x^\top Qx-\sum_jw_j\lambda_j\|z_j\|^2\le0$. The terms $-x^\top B^\top\sum_jw_jz_j$ and $\sum_jw_jz_j^\top Bx$ cancel because the aggregation and embedding maps are adjoint.

## Key formulas

- $$\partial_tu(t)+\mathsf{A}_0u(t)+\mathsf{A}_1(k*u)(t)=f(t),\qquad u(0)=u_0.$$
- $$k(t)=\int_{[0,\infty)}e^{-\lambda t}\,d\nu(\lambda),\qquad M_0=\nu([0,\infty)),\qquad M_1=\int_{[0,\infty)}\lambda\,d\nu(\lambda).$$
- $$\dot x=-Qx-B^\top\sum_{j=1}^m w_jz_j+Wr,\qquad \dot z_j=Bx-\lambda_jz_j.$$
- $$E(x,z)=\frac12\|x\|^2+\frac12\sum_{j=1}^mw_j\|z_j\|^2,\qquad \dot E=-x^\top Qx-\sum_{j=1}^mw_j\lambda_j\|z_j\|^2\le0.$$

## Implementation notes

Integrate this as a recurrent or state-space block between Transformer or MLP layers, operating on a sequence tensor with shape [batch, time, d]. Choose $m$ memory modes with logarithmically spaced rates $\lambda_j$ covering the task horizon, and positive weights $w_j$ normalized so $\sum_jw_j=M_0$. The trainable parameters are $B\in\mathbb{R}^{p\times d}$, input projection $W$, and optionally a positive semidefinite damping matrix $Q=LL^\top$; do not train the return coupling independently, because it must remain $-B^\top$ to preserve cancellation. For each time step, use implicit Euler on the coupled system: solve $x_{t+1}=x_t+h[-Qx_{t+1}-B^\top\sum_jw_jz_{j,t+1}+Wr_t]$ and $z_{j,t+1}=z_{j,t}+h[Bx_{t+1}-\lambda_jz_{j,t+1}]$. A cheaper semi-implicit variant first computes $z_{j,t+1}=e^{-h\lambda_j}z_{j,t}+((1-e^{-h\lambda_j})/\lambda_j)Bx_t$, using the limit $h$ when $\lambda_j=0$, and then updates $x$. Add a nonlinear gated readout after the stable state rather than placing an unconstrained nonlinearity inside the certified homogeneous dynamics. The exact structural checks are $w_j\ge0$, $\lambda_j\ge0$, $Q\succeq0$, and tied adjoint couplings. Estimate the useful rate range from sequence lengths, for example $\lambda_j\in[1/T,1]$, and optionally learn positive rates using $\lambda_j=\operatorname{softplus}(a_j)$. First test on sequential MNIST and a small WikiText character model against a vanilla GRU and an unconstrained diagonal SSM with equal parameter count. Measure hidden-state norm growth, exploding-gradient frequency, long-context validation loss, and robustness when the integration step size is increased. Success means bounded energy in homogeneous tests, fewer NaNs and gradient spikes, and lower perplexity or classification error at sequence lengths beyond training horizons. Include an ablation with $Q=0$ to verify that stability comes from memory damping and adjoint cancellation rather than ordinary visible-state regularization.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built a finite positive-quadrature dissipative memory layer with tied adjoint couplings and implicit Euler integration in dissipative_memory.py. The toy verification confirmed exact cross-term cancellation (maximum error 0), the explicit-Euler boundary h*=2λ/(λ²+b²) across five b values (maximum error 1.1e-16), linear continuous dissipation in λ including zero dissipation at λ=0, and implicit-Euler energy contraction for steps 0.01 through 100. On the small length-40 sum task, the memory model reached MSE 8.50e-6 versus 39.19 for the scalar unconstrained RNN, but this comparison is not a parameter-matched or difficult sequence benchmark.

### Mechanism check

- Verdict: Built a finite positive-quadrature dissipative memory layer with tied adjoint couplings and implicit Euler integration in dissipative_memory.py. The toy verification confirmed exact cross-term cancellation (maximum error 0), the explicit-Euler boundary h*=2λ/(λ²+b²) across five b values (maximum error 1.1e-16), linear continuous dissipation in λ including zero dissipation at λ=0, and implicit-Euler energy contraction for steps 0.01 through 100. On the small length-40 sum task, the memory model reached MSE 8.50e-6 versus 39.19 for the scalar unconstrained RNN, but this comparison is not a parameter-matched or difficult sequence benchmark.
- Confidence: 9/10
- Limitations: No sequential MNIST, WikiText, long-context extrapolation, FLOP-matched GRU/SSM benchmark, repeated-seed statistics, or trained multi-dimensional task was tested. The sequence comparison uses a deliberately tiny scalar RNN and the layer's simple semi/implicit setup; the certified homogeneous dynamics exclude input forcing and nonlinear readout effects.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/338/bench_report.json)
- [bench_stage2.py](https://synthcore.org/code/338/bench_stage2.py)
- [dissipative_memory.py](https://synthcore.org/code/338/dissipative_memory.py)
- [experiment.py](https://synthcore.org/code/338/experiment.py)
- [report.md](https://synthcore.org/code/338/report.md)
- [report_bench_2026-08-31T014511.md](https://synthcore.org/code/338/report_bench_2026-08-31T014511.md)
- [results.json](https://synthcore.org/code/338/results.json)
- [smoke_test.py](https://synthcore.org/code/338/smoke_test.py)
- [Download all files as ZIP](https://synthcore.org/download/338)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.