✓✓ Beats tuned baseline 2026

Contractive projected residual dynamics

Implementation & benchmark of arXiv:2607.08301 — Projected incrementally scattering passive systems on closed convex sets

Usefulness6/10

Difficulty5/10

Novelty5/10

Dynamics Stability Generalization Accuracy Rnn Ssm Training dynamics Regularization Dynamical systems Convex analysis Geometry Functional analysis

Source paper: Projected incrementally scattering passive systems on closed convex sets arXiv:2607.08301 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Build a recurrent or continuous-depth block from a dissipative vector field and project every state derivative onto the tangent cone of a closed convex hidden-state set. Unlike ordinary clipping, tangent-cone projection removes only the outward component at the boundary and preserves admissible motion. Under the paper's maximal-dissipativity result, the continuous flow is nonexpansive in its initial state.

Formulas

$$N_K(x)=\{n\in X:\langle n,z-x\rangle\le 0\ \forall z\in K\},\qquad T_K(x)=\overline{\{\alpha(z-x):\alpha\ge 0,\ z\in K\}}$$

$$\frac{d^+x(t)}{dt}\in A_\theta(x(t),v(t))-N_K(x(t)),\qquad \frac{dx(t)}{dt}=\Pi_{T_K(x(t))}A_\theta(x(t),v(t))$$

$$\langle A_\theta(x,v)-A_\theta(z,v),x-z\rangle\le 0\quad\Longrightarrow\quad \|x_1(\tau)-x_2(\tau)\|^2\le \|x_1(0)-x_2(0)\|^2$$

Mathematical statement

Let X be a Hilbert space, K a closed convex subset, and A_theta the learned drift. The tangent cone at x is T_K(x)=closure of {alpha(z-x): alpha>=0,z in K}, and the normal cone is N_K(x)={n: inner_product(n,z-x)<=0 for every z in K}. The constrained dynamics are dx/dt in A_theta(x,v)-N_K(x), equivalently dx/dt=Pi_{T_K(x)} A_theta(x,v) when the projection is single-valued. If A_theta is maximal dissipative, meaning inner_product(A_theta(x,v)-A_theta(z,v),x-z)<=0 for all x,z in K, the paper proves that the constrained operator remains maximal dissipative and generates a contraction semigroup. Thus two states satisfy ||x_1(t)-x_2(t)||<=||x_1(0)-x_2(0)||. The paper's equation (31) gives ||[y_tau1,x_1(tau),u_tau1]^T-[y_tau2,x_2(tau),u_tau2]^T|| <= ||[y_01,x_01,u_01]^T-[y_02,x_02,u_02]^T||, while the expanded energy inequality bounds output discrepancy, terminal state discrepancy, and future input discrepancy by initial state and total input discrepancy. In the neural adaptation, x is the hidden state, v is the input, K is a box or Euclidean ball, and A_theta is a parameterized dissipative drift.

Implementation notes

Integrate this into the hidden-state update of a recurrent model or an explicit Euler continuous-depth block. Let x_t in R^d be the hidden state and v_t the input embedding. Use K=[-1,1]^d initially. Parameterize the drift as A_theta(x_t,v_t)=-(L_theta^T L_theta+lambda I)x_t+B_theta v_t+b_theta, where L_theta and B_theta are learned matrices and lambda>=0; the negative semidefinite term enforces dissipativity in x. Compute q=A_theta(x_t,v_t). For each coordinate i, set p_i=q_i when -1<x_{t,i}<1; set p_i=max(q_i,0) when x_{t,i}=-1; and set p_i=min(q_i,0) when x_{t,i}=1. Update x_{t+1}=clip(x_t+h p,-1,1). The projection p is the discrete version of Pi_{T_K(x)}A_theta. For a more stable variant, perform 3 to 5 iterations of projected fixed-point updates for the implicit equation x_{t+1}=argmin_{z in K}||z-x_t-h A_theta(z,v_t)||^2. The paper guarantees contraction for the continuous maximal-dissipative system, not automatically for finite-step Euler updates, so estimate the implemented block's Jacobian norm by power iteration and log pairwise distance ratios. The first cheap experiment should use sequential MNIST or the adding problem with sequence lengths 500 and 1000. Compare a vanilla GRU, a spectrally normalized residual RNN, and this projected dissipative RNN at matched parameter count and FLOPs. Measure validation accuracy, gradient norm, hidden-state norm, perturbation amplification, and loss over sequence length. Success means bounded hidden states, no exploding gradients, final-state perturbation ratio near or below one, and equal-or-better accuracy than the stable baseline at long sequence lengths.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a projected dissipative residual RNN with box tangent-cone projection and a fixed-seed long-sequence toy experiment. The numerical check showed no distance-expansion violations across 1,000 trials (maximum final/initial distance ratio 0.0368) and states remained within [-1,1]. In training, the idea strongly improved stability versus the vanilla residual baseline (perturbation ratio 0.067 vs 9.67; bounded state magnitude 1.0 vs 14.56), but accuracy was lower (0.805 vs 0.906), so the stability effect is real but an overall accuracy win was not demonstrated.

Agent confidence: 9/10

Baseline: accuracy=0.90625, perturbation_ratio=9.6700, mean_final_norm=36.4342, max_abs_final=14.5563, final_loss=0.23464, parameters=305
Idea: accuracy=0.80469, perturbation_ratio=0.06728, mean_final_norm=2.2066, max_abs_final=1.0, final_loss=0.31358, parameters=321; math check max ratio=0.03677 with 0 violations over 1000 trials

Limitations: Only one toy sign-of-sum task, one random seed, hidden size 16, and sequence length 200 were tested. The experiment used explicit Euler rather than the optional implicit solver, did not compare against GRU or spectrally normalized RNN, and did not measure FLOPs, Jacobian power-iteration norms, or gradient-vs-sequence-length curves.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.08301) Sep 4, 2026

Beats the tuned baseline by 85.8% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Dynamics forecasting
Model: rnn_small_matched_residual
Paired seeds: 8
Baseline mean: 0.0024
Idea mean: 0.0003
Effect (Δ): -0.0021 (−85.8%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±25.6%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in dynamics track was tested; tabular, vision, and sequence tracks were not run. The matched baseline was a local unconstrained residual RNN rather than the bench GRU implementation, because the idea changes the recurrence and requires the same residual base architecture on both sides. The experiment used explicit Euler projection, one hidden width (64), one epoch budget (18), and three shared learning rates; implicit projected fixed-point updates, longer horizons, larger datasets, FLOPs, and external control benchmarks were not tested.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Worked ✓

Agent confidence: 9/10

Built a matched projected dissipative residual RNN and compared it with an unconstrained residual RNN on the structurally appropriate built-in dynamics track (controlled pendulum rollout). Across 8 paired seeds, the idea reduced test MSE from 0.00239047 to 0.000338893, with all 8 paired wins and permutation p=0.0081. The trained-model signature confirmed lower perturbation amplification (0.8873 vs 1.0010) and bounded hidden states (maximum absolute value 1.0).

Baseline: test MSE mean=0.0023904674, std=0.0008984586, per-seed=[0.0016786052,0.0020929552,0.0026723652,0.0044336999,0.0019304326,0.0020707706,0.0013639050,0.0028810054], best lr=0.006, epochs=18
Comparison: delta_mean=-0.0020515742, idea_wins=8/8, permutation_p=0.0081, verdict=idea better (significant)
Idea: test MSE mean=0.0003388932, std=0.0002533149, per-seed=[0.0002524245,0.0001881546,0.0002420832,0.0010012880,0.0002840845,0.0002892777,0.0001796607,0.0002741727], best lr=0.006, epochs=18
Mechanism signature: baseline perturbation ratio=1.0009678, idea perturbation ratio=0.8872583, idea/baseline=0.8864004, idea max abs hidden=1.0, confirmed=true

Benchmark result