Certified contraction implicit layer
Implementation & benchmark of arXiv:2607.11122 — Implicit Neural Networks as Static Controllers: Certificates and Performance Separation
Source paper: Implicit Neural Networks as Static Controllers: Certificates and Performance Separation arXiv:2607.11122 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace a deep feed-forward block by the fixed point z=phi(Wz+Vx+b), with the recurrent weight W constrained so that the fixed point is unique for every input. The same condition makes forward fixed-point iteration stable and makes implicit differentiation well-conditioned, allowing depth-independent memory usage while providing a measurable spectral failure boundary.
Formulas
Mathematical statement
Let x be the layer input, z be its d-dimensional equilibrium state, W be the d by d recurrent matrix, V be the input matrix, b be a bias, and phi be componentwise L-Lipschitz: ||phi(a)-phi(c)||_2 <= L||a-c||_2. The fixed point is z*=phi(Wz*+Vx+b). If L||W||_2<1, the map is a contraction, so z* is unique and fixed-point iteration z_{k+1}=phi(Wz_k+Vx+b) converges geometrically with factor at most q=L||W||_2. A less conservative positive-system condition is rho(L|W|)<1, where |W| is the entrywise absolute-value matrix and rho is spectral radius. For a scalar loss J, define A_phi=diag(phi'(Wz*+Vx+b)); implicit differentiation solves (I-A_phi W) dz = A_phi(V dx+dW z*+db), so gradients require one linear solve rather than backpropagation through many iterations.
Implementation notes
1. Integration point: implement an implicit replacement for an MLP residual block, transformer feed-forward block, or SSM transition. Given x, solve z=phi(Wz+Vx+b) by damped fixed-point iteration or Anderson acceleration, then return y=Uz+Rx. Parameterize W so its certified gain remains below one, for example W=(q/L)W_bar/(||W_bar||_2+epsilon), with q=sigmoid(alpha), or use spectral normalization after each optimizer step. 2. Pseudocode: compute W and estimate s=||W||_2; set q=Ls; reject or project parameters if q>=0.95; iterate z <- (1-gamma)z+gamma phi(Wz+Vx+b) until the residual is below tolerance; in backward mode form A_phi and solve (I-A_phi W)^T lambda=dJ/dz, then use the transpose implicit-gradient equations. 3. Computed from the paper: the fixed-point representation, contraction inequality, and implicit Jacobian. Estimated empirically: spectral norm, local convergence factor, solver iterations, and gradient condition number. 4. First cheap experiment: train a four-layer-equivalent implicit MLP on MNIST or CIFAR-10 with identical parameter count, comparing unconstrained DEQ-style training against q ceilings 0.5, 0.8, 0.95, and 1.05. Measure residual decay versus iteration k and sweep q across one. The prediction is geometric residual decay with slope log(q) for q<1, a sharp increase in solver iterations and gradient norm near q=1, and frequent nonconvergence for q>1; the measured asymptotic decay factor should be within approximately 20 percent of L||W||_2.
Verification
Stage 1 · Toy mechanism gate: Passed ✓
Stage 2 · Mechanism transferred to benchmark: Confirmed ✓
Stage 2 · Practical benchmark result: Inconclusive
Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built a certified implicit tanh layer sanity experiment with contraction-rate, damping, positive-system, implicit-Jacobian, and tiny GPU training checks. The mechanism manifested: for q=0.95 the measured residual rate was 0.9500 versus predicted 0.95, tolerance steps increased from 10 at q=0.2 to 271 at q=0.95, and q>=1 failed to converge; damping rates also matched predictions exactly. The implicit derivative matched finite differences to 1.1e-10. The certified q=0.8 model did not win the tiny regression comparison (final MSE 0.2061 versus 0.09595 for the q=1.05 comparator), so this supports the math and stability mechanism, not a demonstrated task-performance improvement.
- Agent confidence
- 9/10
- Baseline
- q=1.05 comparator: final MSE 0.09595, final fixed-point residual 3.31e-3, 60 iterations/step
- Idea
- certified q=0.8: final MSE 0.20612, final fixed-point residual 6.78e-9, 60 iterations/step; q=0.95 rate 0.9500 observed versus 0.95 predicted and 271 steps to 1e-6
Limitations: The learning comparison is a very small synthetic regression, uses unrolled fixed-point iterations rather than a custom implicit backward solve, and does not measure memory or wall-clock speed. The q>1 sweep is scalar linear and tests instability directly rather than nonlinear basin behavior; rapidly converging low-q rate entries can underflow before the reporting window.
How to run: python3 experiment.py
Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.11122)
No significant difference vs. the tuned baseline (+17.8%, p=0.3157, wins 4 / 8 paired seeds).
- Benchmark
- Dynamics forecasting
- Model
- rnn_small
- Paired seeds
- 8
- Baseline mean
- 0.0176
- Idea mean
- 0.0208
- Effect (Δ)
- 0.0031 (+17.8%; negative = idea better)
- Wins
- 4 / 8 paired seeds
- p-value
- 0.3157 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- ±40.1%
- Mechanism
- Confirmed ✓
- Practical effect
- Inconclusive
- Baseline tuning
- swept over 3 configs
The completed run used 3 epochs and 400/200 train/test samples after a 12-epoch run exceeded the runtime limit. The implementation uses differentiable unrolled fixed-point iterations rather than a custom implicit backward linear solve, and memory and wall-clock speed were not measured.
How to run: python3 bench_run.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Stage 2 — Benchmark latest Failed ✗
Agent confidence: 9/10
Implemented a matched certified equilibrium tanh block and evaluated it against an unconstrained equilibrium baseline on the registered dynamics track. The trained-model contraction signature was confirmed: predicted residual factor 0.950001 versus observed 0.992522, with 4.5% median relative error. The idea was worse on test MSE (0.0207617 versus 0.0176257), with paired delta +0.00313594 and permutation p=0.31575, so there was no significant win.
- Baseline
- Best unconstrained baseline configuration lr=0.005; mean test MSE 0.0176257133; std 0.0067040358
- Idea
- Best certified configuration q=0.95, lr=0.005; mean test MSE 0.0207616551; std 0.0095054335; paired delta +0.0031359418; permutation p=0.31575
Benchmark result
No significant difference vs. the tuned baseline (+17.8%, p=0.3157, wins 4 / 8 paired seeds).
- Benchmark
- Dynamics forecasting
- Model
- rnn_small
- Paired seeds
- 8
- Baseline mean
- 0.0176
- Idea mean
- 0.0208
- Effect (Δ)
- 0.0031 (+17.8%; negative = idea better)
- Wins
- 4 / 8 paired seeds
- p-value
- 0.3157 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- ±40.1%
- Mechanism
- Confirmed ✓
- Practical effect
- Inconclusive
- Baseline tuning
- swept over 3 configs
The completed run used 3 epochs and 400/200 train/test samples after a 12-epoch run exceeded the runtime limit. The implementation uses differentiable unrolled fixed-point iterations rather than a custom implicit backward linear solve, and memory and wall-clock speed were not measured.
How to run: python3 bench_run.py
Artifacts
- bench_report.json 5.9 KB View Raw
- bench_run.py 5.8 KB View Raw
- experiment.py 4.9 KB View Raw
- report.md 1.6 KB View
- report_bench_2026-08-30T161634.md 5.6 KB View
- results.json 2.3 KB View Raw