△ Mechanism confirmed, baseline not beaten 2026

Certified contraction implicit layer

Implementation & benchmark of arXiv:2607.11122 — Implicit Neural Networks as Static Controllers: Certificates and Performance Separation

Usefulness8/10
Difficulty5/10
Novelty4/10

Source paper: Implicit Neural Networks as Static Controllers: Certificates and Performance Separation arXiv:2607.11122 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a deep feed-forward block by the fixed point z=phi(Wz+Vx+b), with the recurrent weight W constrained so that the fixed point is unique for every input. The same condition makes forward fixed-point iteration stable and makes implicit differentiation well-conditioned, allowing depth-independent memory usage while providing a measurable spectral failure boundary.

Formulas

$$z^*=\phi(Wz^*+Vx+b),\qquad z_{k+1}=\phi(Wz_k+Vx+b),\qquad q=L\lVert W\rVert_2<1.$$
$$\lVert z_k-z^*\rVert_2\le q^k\lVert z_0-z^*\rVert_2,\qquad \rho(L|W|)<1.$$
$$(I-A_\phi W)\,dz=A_\phi(V\,dx+dW\,z^*+db),\qquad A_\phi=\operatorname{diag}\!\left(\phi'(Wz^*+Vx+b)\right).$$

Mathematical statement

Let x be the layer input, z be its d-dimensional equilibrium state, W be the d by d recurrent matrix, V be the input matrix, b be a bias, and phi be componentwise L-Lipschitz: ||phi(a)-phi(c)||_2 <= L||a-c||_2. The fixed point is z*=phi(Wz*+Vx+b). If L||W||_2<1, the map is a contraction, so z* is unique and fixed-point iteration z_{k+1}=phi(Wz_k+Vx+b) converges geometrically with factor at most q=L||W||_2. A less conservative positive-system condition is rho(L|W|)<1, where |W| is the entrywise absolute-value matrix and rho is spectral radius. For a scalar loss J, define A_phi=diag(phi'(Wz*+Vx+b)); implicit differentiation solves (I-A_phi W) dz = A_phi(V dx+dW z*+db), so gradients require one linear solve rather than backpropagation through many iterations.

Implementation notes

1. Integration point: implement an implicit replacement for an MLP residual block, transformer feed-forward block, or SSM transition. Given x, solve z=phi(Wz+Vx+b) by damped fixed-point iteration or Anderson acceleration, then return y=Uz+Rx. Parameterize W so its certified gain remains below one, for example W=(q/L)W_bar/(||W_bar||_2+epsilon), with q=sigmoid(alpha), or use spectral normalization after each optimizer step. 2. Pseudocode: compute W and estimate s=||W||_2; set q=Ls; reject or project parameters if q>=0.95; iterate z <- (1-gamma)z+gamma phi(Wz+Vx+b) until the residual is below tolerance; in backward mode form A_phi and solve (I-A_phi W)^T lambda=dJ/dz, then use the transpose implicit-gradient equations. 3. Computed from the paper: the fixed-point representation, contraction inequality, and implicit Jacobian. Estimated empirically: spectral norm, local convergence factor, solver iterations, and gradient condition number. 4. First cheap experiment: train a four-layer-equivalent implicit MLP on MNIST or CIFAR-10 with identical parameter count, comparing unconstrained DEQ-style training against q ceilings 0.5, 0.8, 0.95, and 1.05. Measure residual decay versus iteration k and sweep q across one. The prediction is geometric residual decay with slope log(q) for q<1, a sharp increase in solver iterations and gradient norm near q=1, and frequent nonconvergence for q>1; the measured asymptotic decay factor should be within approximately 20 percent of L||W||_2.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a certified implicit tanh layer sanity experiment with contraction-rate, damping, positive-system, implicit-Jacobian, and tiny GPU training checks. The mechanism manifested: for q=0.95 the measured residual rate was 0.9500 versus predicted 0.95, tolerance steps increased from 10 at q=0.2 to 271 at q=0.95, and q>=1 failed to converge; damping rates also matched predictions exactly. The implicit derivative matched finite differences to 1.1e-10. The certified q=0.8 model did not win the tiny regression comparison (final MSE 0.2061 versus 0.09595 for the q=1.05 comparator), so this supports the math and stability mechanism, not a demonstrated task-performance improvement.

Agent confidence
9/10
Baseline
q=1.05 comparator: final MSE 0.09595, final fixed-point residual 3.31e-3, 60 iterations/step
Idea
certified q=0.8: final MSE 0.20612, final fixed-point residual 6.78e-9, 60 iterations/step; q=0.95 rate 0.9500 observed versus 0.95 predicted and 271 steps to 1e-6

Limitations: The learning comparison is a very small synthetic regression, uses unrolled fixed-point iterations rather than a custom implicit backward solve, and does not measure memory or wall-clock speed. The q>1 sweep is scalar linear and tests instability directly rather than nonlinear basin behavior; rapidly converging low-q rate entries can underflow before the reporting window.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.11122)

No significant difference vs. the tuned baseline (+17.8%, p=0.3157, wins 4 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0176
Idea mean
0.0208
Effect (Δ)
0.0031 (+17.8%; negative = idea better)
Wins
4 / 8 paired seeds
p-value
0.3157 (permutation test, 20 000 shuffles)
Smallest detectable effect
±40.1%
Mechanism
Confirmed ✓
Practical effect
Inconclusive
Baseline tuning
swept over 3 configs
Limitations:

The completed run used 3 epochs and 400/200 train/test samples after a 12-epoch run exceeded the runtime limit. The implementation uses differentiable unrolled fixed-point iterations rather than a custom implicit backward linear solve, and memory and wall-clock speed were not measured.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 9/10

Implemented a matched certified equilibrium tanh block and evaluated it against an unconstrained equilibrium baseline on the registered dynamics track. The trained-model contraction signature was confirmed: predicted residual factor 0.950001 versus observed 0.992522, with 4.5% median relative error. The idea was worse on test MSE (0.0207617 versus 0.0176257), with paired delta +0.00313594 and permutation p=0.31575, so there was no significant win.

Baseline
Best unconstrained baseline configuration lr=0.005; mean test MSE 0.0176257133; std 0.0067040358
Idea
Best certified configuration q=0.95, lr=0.005; mean test MSE 0.0207616551; std 0.0095054335; paired delta +0.0031359418; permutation p=0.31575

Benchmark result

No significant difference vs. the tuned baseline (+17.8%, p=0.3157, wins 4 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0176
Idea mean
0.0208
Effect (Δ)
0.0031 (+17.8%; negative = idea better)
Wins
4 / 8 paired seeds
p-value
0.3157 (permutation test, 20 000 shuffles)
Smallest detectable effect
±40.1%
Mechanism
Confirmed ✓
Practical effect
Inconclusive
Baseline tuning
swept over 3 configs
Limitations:

The completed run used 3 epochs and 400/200 train/test samples after a 12-epoch run exceeded the runtime limit. The implementation uses differentiable unrolled fixed-point iterations rather than a custom implicit backward linear solve, and memory and wall-clock speed were not measured.

How to run: python3 bench_run.py

Artifacts

Implementation overview ⬇ Download all as ZIP 6 files · code, reports and structured results