✗ Failed on benchmark 2026

Positive-real rational resolvent mixer

Implementation & benchmark of arXiv:2609.03054 — The classes of bivariate Schur and Herglotz matrix-valued rational functions: realizations, symmetrizations, and related determinantal representations

Usefulness7/10

Difficulty6/10

Novelty7/10

Architecture Stability Accuracy Scalability Ssm Transformer Training Linear algebra Control theory Dynamical systems

Source paper: The classes of bivariate Schur and Herglotz matrix-valued rational functions: realizations, symmetrizations, and related determinantal representations arXiv:2609.03054 ⓘ · analyzed Sep 4, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace an unconstrained token mixer or feed-forward residual map with a feature-conditioned rational operator whose transfer matrix is positive real on the positive orthant. Apply it through a resolvent, rather than an additive residual, so that the accretivity certificate yields a nonexpansive implicit update and suppresses activation explosions.

Formulas

$$H(\zeta)=D+C\bigl(P(\zeta)+A\bigr)^{-1}B,\qquad P(\zeta)=\zeta_{1}P_{1}+\zeta_{2}P_{2}.$$

$$\begin{bmatrix}A+A^{*}&C^{*}-B\\ C-B^{*}&D+D^{*}\end{bmatrix}\geq 0.$$

$$\operatorname{Re}H(\zeta)=\frac{H(\zeta)+H(\zeta)^{*}}{2}\geq0,\qquad \left\|(I_d+\eta H(\zeta))^{-1}\right\|_2\leq1.$$

$$z_j(x)=\epsilon+\operatorname{softplus}(w_j^{\top}q(x)),\qquad y=(I_d+\eta[D+C(z_1P_1+z_2P_2+A)^{-1}B])^{-1}x.$$

Mathematical statement

The paper realizes a matrix-valued Herglotz function as $H(\zeta)=D+C(P(\zeta)+A)^{-1}B$, where $\zeta=(\zeta_1,\zeta_2)\in\mathbb{C}_+^2$, $P(\zeta)=\zeta_1P_1+\zeta_2P_2$, $A\in\mathbb{C}^{r\times r}$ is the state matrix, $B\in\mathbb{C}^{r\times d}$, $C\in\mathbb{C}^{d\times r}$, $D\in\mathbb{C}^{d\times d}$, and $P_1,P_2\in\mathbb{C}^{r\times r}$ are orthogonal projections satisfying $P_j=P_j^*$, $P_j^2=P_j$, and $P_1+P_2=I_r$. The paper's KYP condition is $\begin{bmatrix}A+A^*&C^*-B\\C-B^*&D+D^*\end{bmatrix}\geq0$, where $X^*$ denotes conjugate transpose and $X\geq0$ means positive semidefinite. Under this condition, $H$ has nonnegative Hermitian part on the bihalfplane: $\operatorname{Re}H(\zeta):=(H(\zeta)+H(\zeta)^*)/2\geq0$. For real neural tensors use real matrices and evaluate only $\zeta_j\geq\epsilon>0$. The adapted layer is $y=(I_d+\eta H(\zeta(x)))^{-1}x$ with $\eta>0$; accretivity implies the resolvent bound $\|(I_d+\eta H(\zeta))^{-1}\|_2\leq1$, giving a direct stability target.

Implementation notes

Integrate this at the token-mixing or FFN-residual point of a small Transformer or sequence model. Let $x\in\mathbb{R}^d$ be one token vector, let $q(x)$ be either $x$ or a cheap pooled context vector, and compute two strictly positive scalar coordinates $z_j=\epsilon+\operatorname{softplus}(w_j^\top q)$. Use a shared state size $r\ll d$ across all tokens. Initialize $P_1$ as a diagonal projection onto the first $r/2$ state coordinates and set $P_2=I_r-P_1$; optionally replace these by $P_1=Q\operatorname{diag}(1_{r/2},0)Q^\top$ and $P_2=I-P_1$, with $Q$ parameterized by a Cayley transform. For each token, form $M=A+z_1P_1+z_2P_2$, solve $MU=B x$, and compute $h(x)=D x+C U$. Then solve $(I_d+\eta H)x'=x$ for the output $x'$, using a Cholesky solve when the symmetric part is explicitly symmetrized, or a stable LU solve otherwise; add a standard gated projection around this block if dimensionality changes. During training enforce the paper's KYP certificate by forming $K=\begin{bmatrix}A+A^\top&C^\top-B\\C-B^\top&D+D^\top\end{bmatrix}$ and adding $\lambda\sum_i\operatorname{softplus}(\delta-\lambda_i(K))^2$ to the loss, where $\lambda_i(K)$ are eigenvalues and $\delta>0$ is a desired margin. A stronger variant projects $K$ onto the PSD cone after every optimizer step and reconstructs the block variables. The paper's mathematical computation is the rational realization and KYP constraint; $z_j$, the KYP margin, and $\eta$ are learned or tuned. First test on CIFAR-10 with a 6-layer width-128 Transformer and on the Adding or Copying synthetic long-range sequence task, comparing against a standard FFN/token mixer with equal parameter count. Measure loss versus optimizer steps, maximum activation norm, gradient norm, failed linear solves, and validation accuracy. Success is a substantially lower tail of activation and gradient norms with no loss of accuracy, or faster loss descent at equal FLOPs; also report the overhead from the $r\times r$ solve and test whether $r=16$ matches a much larger unconstrained mixer.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a NumPy MVP of the positive-real rational resolvent mixer with an exactly certified KYP realization, rational state-space solves, and implicit output solves. The numerical check found KYP minimum eigenvalue 0.24, sampled Hermitian-part minimum eigenvalue 0.1231, and maximum sampled resolvent singular value 0.9104, confirming the nonexpansive bound on the tested positive coordinates. In the fixed-operator toy update, the implicit method reduced RMS activation from 0.952 to 6.55e-6, while the explicit additive control exploded to RMS 1.67e21 at eta=0.8; this demonstrates the claimed stability effect, but not trained-model accuracy.

Agent confidence: 8/10

Baseline: Explicit additive residual at eta=0.8: final RMS 1.6729e21, final loss 2.7985e42, maximum sample norm 1.2222e22.
Idea: Implicit resolvent at eta=0.8: final RMS 6.545e-6, final loss 4.284e-11, maximum sample norm 0.9518. Across 500 positive-coordinate samples, maximum resolvent singular value was 0.9104 and maximum sampled energy ratio was 0.8903.
Step sweep: Explicit residual final RMS: 137.8 at eta=0.05, 2.014e4 at 0.1, 9.481e7 at 0.2, 3.869e13 at 0.4, and 1.673e21 at 0.8.

Limitations: Only a small fixed-operator NumPy toy experiment was tested. CIFAR-10, a long-range sequence task, learned feature conditioning, parameter-matched Transformer or FFN training, gradient analysis, speed, and solve overhead were not tested. The explicit control is an update-rule comparison rather than a complete modern neural residual block.

How to run: python3 resolvent_mixer_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.03054) Sep 4, 2026

Ahead of the baseline (−23.9%) but not statistically significant (p=0.2357, wins 6 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0005
Idea mean: 0.0004
Effect (Δ): -0.0001 (−23.9%; negative = idea better)
Wins: 6 / 8 paired seeds
p-value: 0.2357 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±43.9%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in dynamics track was tested because the idea targets stability/control. The implementation did not enforce the full KYP PSD constraint during optimization, so the trained operator was not certified. Activation and gradient tail distributions, FLOP/solve overhead, and longer training budgets were not measured.

How to run: python3 bench_resolvent.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented and trained a positive-real rational resolvent recurrent model on the structurally matched built-in dynamics track, using the canonical rnn_small GRU as the paired baseline. The idea reduced mean test MSE, but the paired permutation test was not significant (p=0.2357), and the trained-model nonexpansiveness signature was not confirmed because the observed maximum ratio was 5.2935.

Baseline: Best baseline lr=0.006: mean test MSE 0.0005385369, std 0.0001364130.
Idea: Best idea lr=0.001: mean test MSE 0.0004099761, std 0.0002596498; paired delta=-0.0001285607, 6/8 wins, p=0.2357.

Benchmark result