# Passivity-Regularized Sequence Layer

- ID: 587
- Canonical URL: https://synthcore.org/idea/587/passivity-regularized-sequence-layer
- API JSON: https://synthcore.org/api/idea/587.json
- API Markdown: https://synthcore.org/api/idea/587.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.08637](https://arxiv.org/abs/2607.08637)
- Category: regularization
- Solves: stability, generalization
- ML areas: rnn, ssm, regularization, training-dynamics
- Math tags: control-theory, dynamical-systems, optimization, functional-analysis
- Ratings: usefulness 6/10; difficulty 3/10; novelty 6/10

## Idea description

Use the paper's scattering energy balance as a measurable regularizer for an existing recurrent or state-space model instead of replacing its architecture. Penalize positive violations of the per-step energy inequality and, for paired examples, penalize violations of incremental passivity so that the model learns not to amplify perturbations over long sequences.

## Mathematical statement

The paper's defining inequality compares stored state energy and accumulated output energy with initial state energy and accumulated input energy: \(\|x(\tau)\|^2+\int_0^\tau\|y(t)\|^2dt\leq\|x(0)\|^2+\int_0^\tau\|u(t)\|^2dt\). For a discrete neural sequence layer with hidden state \(h_t\), input \(u_t\), and output \(y_t\), define the per-step residual \(r_t=\|h_{t+1}\|_2^2+\|y_t\|_2^2-\|h_t\|_2^2-\|u_t\|_2^2\). A passive trajectory should have nonpositive cumulative residual \(\sum_t r_t\). Incremental passivity applies the same inequality to differences between two trajectories: \(\delta h_t=h_t^{(a)}-h_t^{(b)}\), \(\delta u_t=u_t^{(a)}-u_t^{(b)}\), and \(\delta y_t=y_t^{(a)}-y_t^{(b)}\). The positive-part function \([z]_+=\max(z,0)\) turns violations into a differentiable-almost-everywhere penalty. This is a soft empirical version of the paper's energy guarantee and does not require modifying the original recurrent transition.

## Key formulas

- $$\|x(\tau)\|^{2}+\int_{0}^{\tau}\|y(t)\|^{2}\,dt\leq\|x(0)\|^{2}+\int_{0}^{\tau}\|u(t)\|^{2}\,dt.$$
- $$r_t=\|h_{t+1}\|_2^2+\|y_t\|_2^2-\|h_t\|_2^2-\|u_t\|_2^2,$$
- $$\mathcal L_{\mathrm{pass}}=\frac{1}{T}\sum_{t=0}^{T-1}[r_t]_+ + \gamma\left[\sum_{t=0}^{T-1}r_t\right]_+,$$
- $$r_t^{\mathrm{inc}}=\|\delta h_{t+1}\|_2^2+\|\delta y_t\|_2^2-\|\delta h_t\|_2^2-\|\delta u_t\|_2^2.$$

## Implementation notes

(1) Integrate this into the training loss of an existing GRU, LSTM, diagonal SSM, or custom RNN. Record \(h_t\) before each transition, \(h_{t+1}\) after the transition, the layer output \(y_t\), and the normalized input \(u_t\). Ensure the same feature scaling is used for all energy terms; otherwise the penalty is arbitrary. (2) Pseudocode: initialize \(L_p=0\) and \(R=0\); run the sequence model; for each timestep compute \(r=\|h_{t+1}\|_2^2+\|y_t\|_2^2-\|h_t\|_2^2-\|u_t\|_2^2\), update \(L_p\leftarrow L_p+\operatorname{relu}(r)\) and \(R\leftarrow R+r\); after the sequence set \(L_p\leftarrow L_p/T+\gamma\operatorname{relu}(R)\). For incremental training, create a second trajectory using a nearby input perturbation or an augmented view, compute \(\delta h,\delta u,\delta y\), and add \(\eta T^{-1}\sum_t\operatorname{relu}(r_t^{inc})\). Optimize \(L_{task}+\lambda L_p\). (3) The paper determines the target inequality and the residual definitions; \(\lambda\), \(\gamma\), \(\eta\), perturbation size, and normalization are empirical. Log maximum positive residual, cumulative residual, hidden-state norm, gradient norm, and finite-difference amplification \(\|\delta h_T\|/\|\delta h_0\|\). (4) First cheap experiment: train 2-layer GRUs with hidden size 128 on sequential MNIST, adding, and copy tasks. Compare no regularizer, spectral normalization, and this penalty for \(\lambda\in\{10^{-3},10^{-2},10^{-1}\}\), with and without paired trajectories. Evaluate ordinary sequences and sequences 5 times longer than training. Success is reduced hidden-state explosion and gradient variance, improved long-context accuracy, and less than 10% additional training FLOPs.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a passivity regularizer for a GRU, including per-step, cumulative, and optional incremental trajectory penalties, plus a direct telescoping-identity check. The identity held numerically with maximum error 2.6e-6, and regularization reduced hidden-state energy violations and perturbation amplification. However, long-context accuracy worsened from 0.846 for the baseline to 0.666 with regularization, so the stability effect is real but did not improve task generalization in this small test.

### Mechanism check

- Verdict: Built a passivity regularizer for a GRU, including per-step, cumulative, and optional incremental trajectory penalties, plus a direct telescoping-identity check. The identity held numerically with maximum error 2.6e-6, and regularization reduced hidden-state energy violations and perturbation amplification. However, long-context accuracy worsened from 0.846 for the baseline to 0.666 with regularization, so the stability effect is real but did not improve task generalization in this small test.
- Confidence: 8/10
- Limitations: Only a small GRU and synthetic sign-sum task were tested with one seed and one regularization weight. No spectral-normalization baseline, LSTM/SSM, FLOP accounting, gradient-variance measurement, or repeated-seed significance analysis was performed; CUDA failed due to shared cuDNN allocation and the run fell back to CPU.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.72575
- Paired wins: 4/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_passivity.py](https://synthcore.org/code/1399/bench_passivity.py)
- [bench_report.json](https://synthcore.org/code/1399/bench_report.json)
- [passivity_experiment.py](https://synthcore.org/code/1399/passivity_experiment.py)
- [report.md](https://synthcore.org/code/1399/report.md)
- [report_bench_2026-09-04T200930.md](https://synthcore.org/code/1399/report_bench_2026-09-04T200930.md)
- [results.json](https://synthcore.org/code/1399/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1399)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.