✓✓ Beats tuned baseline 2026

Pole-safe rational neural layer

Implementation & benchmark of arXiv:2607.00097 — Construction of Pole Cancellation Functions at Ordinary Poles of Operator-Valued Functions

Usefulness6/10

Difficulty5/10

Novelty8/10

Architecture Stability Accuracy Ssm Mlp Training dynamics Functional analysis Linear algebra Approximation theory

Source paper: Construction of Pole Cancellation Functions at Ordinary Poles of Operator-Valued Functions arXiv:2607.00097 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace an unconstrained feature vector entering a rational or resolvent-like neural operator by a polynomial feature whose first nonzero Taylor coefficient lies in a pole-safe subspace. For a pole of order m, the simplest guaranteed construction is psi(z)=(z-beta)^m v, which makes Q(z)psi(z) bounded even when Q(z) diverges. For lower-order cancellation, solve linear constraints among Taylor coefficients of psi so that all negative Laurent powers vanish.

Formulas

$$Q(z)=\sum_{j=-m}^{\infty}Q_j(z-\beta)^j.$$

$$\boldsymbol{\psi}(z)=(z-\beta)^m\boldsymbol{\psi}_m+(z-\beta)^{m+1}\boldsymbol{\psi}_{m+1}+O((z-\beta)^{m+2}),\qquad \boldsymbol{\psi}_m\neq 0.$$

$$Q_{-m}\boldsymbol{\psi}_m\neq 0.$$

$$\boldsymbol{\hat{\varphi}}(z):=Q(z)\boldsymbol{\psi}(z),\qquad \boldsymbol{\hat{\varphi}}(\beta)=Q_{-m}\boldsymbol{\psi}_m.$$

Mathematical statement

The paper defines a pole of order m at beta by the Laurent expansion Q(z)=sum_{j=-m}^{infinity} Q_j (z-beta)^j, where Q_j are bounded linear operators and Q_{-m} is nonzero. A maximal-order pole-cancellation function has the form psi(z)=(z-beta)^m psi_m+(z-beta)^{m+1} psi_{m+1}+O((z-beta)^{m+2}), with psi_m nonzero and Q_{-m} psi_m nonzero. The product phi_hat(z):=Q(z)psi(z) is finite at beta, with phi_hat(beta)=Q_{-m} psi_m. More generally, if psi(z)=sum_{l=k}^{infinity} psi_l (z-beta)^l, the coefficient of (z-beta)^r in Q(z)psi(z) is sum_{j+l=r} Q_j psi_l; cancellation requires these sums to equal zero for every r<0. Here z is the scalar spectral variable, beta is the pole location, Q_j are Laurent coefficients, and psi_l are vector-valued Taylor coefficients. The order-two example in the paper reduces the maximal-order condition to Q_{-2} psi_2 not equal to zero.

Implementation notes

(1) Integration point: use this construction in a rational spectral layer or diagonal/state-space layer that computes y(z,x)=Q_theta(z)h_theta(x) for scalar frequencies z, especially when z can approach a learned pole beta. Replace the unconstrained vector h_theta(x) by psi_theta(z,x), and return y=Q_theta(z)psi_theta(z,x). Store Laurent coefficients Q[-m],...,Q[R] explicitly, or obtain them from a rational parameterization.

(2) MVP pseudocode:
inputs: x, z, beta, pole_order m, Laurent matrices Q[-m:R]
v = h_theta(x)
psi_m = v / (norm(v)+epsilon)
psi = (z-beta)^m * psi_m
if lower_order_cancellation:
predict psi[k],...,psi[R] with neural heads
for r in {-m,...,-1}:
enforce sum over j+l=r of Q[j] @ psi[l] = 0
phi = sum over j of Q[j] @ psi(z)
y = phi

For the guaranteed maximal-order version, only psi_m is needed; higher coefficients can be added as residual features. For lower-order cancellation, parameterize the feasible coefficient vectors with an SVD-derived nullspace of the linear constraint matrix instead of using only a soft penalty. If beta and z are complex, represent real and imaginary parts as a doubled real system.

(3) Computed directly from the paper's mathematics: multiplication by (z-beta)^m, the Laurent-product coefficient constraints, and the finite limit Q_{-m} psi_m. Estimated empirically: pole locations beta and order m when they are unknown. Fit a local Laurent model to Q_theta(z) over sampled z, select m using held-out reconstruction error and residual growth, and estimate the relevant nullspace with SVD. Monitor the smallest singular value of the constraint matrix to detect ill-conditioning.

(4) First experiment: construct a 32-dimensional synthetic rational state-space operator Q(z)=Q_{-2}(z-beta)^{-2}+Q_{-1}(z-beta)^{-1}+Q_0 with beta=1, then train a small MLP to regress sequences sampled with z in [0.95,1.05]. Compare an unconstrained branch, gradient clipping, and the pole-safe branch at equal parameter count. Measure maximum activation norm, gradient norm, NaN frequency, validation MSE, and convergence speed. Success means bounded activations and gradients near z=beta, no numerical blow-ups, and lower validation error than clipping at equal training cost. Then test the module in a small rational or S4-like SSM on sequential MNIST or a Long Range Arena task, measuring stability and accuracy at sequence lengths where the unconstrained baseline becomes ill-conditioned.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a pole-safe order-two rational layer using psi(z,x)=(z-beta)^2 h_theta(x)/||h_theta(x)|| and compared it with unconstrained and gradient-clipped controls. The math check reproduced the expected inverse-square divergence for the unsafe product and a bounded, near-constant safe product converging to Q_-2 v. In the synthetic regression, the safe branch achieved validation MSE 0.0227 versus 7.996e11 unconstrained and 1.337e11 with clipping, with output norm about 1.49 versus roughly 1.8e7–2.0e7. This is a clear stability and optimization signal, but only on a synthetic one-seed benchmark.

Agent confidence: 8/10

Baseline: Unconstrained: validation MSE 7.996e11, maximum output norm 1.837e7, near-pole norm 5.304e7; gradient-clipped control: validation MSE 1.337e11, maximum output norm 2.049e7, near-pole norm 2.239e7.
Idea: Pole-safe: validation MSE 2.273e-2, maximum output norm 1.491, near-pole norm 1.327; direct math unsafe slope -1.9995, safe slope 0.00046, safe limit error 1.22e-5 at |z-beta|=1e-5.

Limitations: Only a synthetic 32-dimensional order-two operator, one fixed random seed, and one short regression setup were tested. No real state-space model, sequential benchmark, learned pole, lower-order nullspace parameterization, ablation over pole orders, or repeated-seed statistical evaluation was performed.

How to run: python3 pole_safe_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.00097) Sep 3, 2026

Beats the tuned baseline by 99.5% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 118.6588
Idea mean: 0.5461
Effect (Δ): -118.1126 (−99.5%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±40.1%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

The test used the built-in 400-sample dynamics track, 12 epochs, and a fixed known scalar pole of order two. Learned pole estimation, lower-order nullspace cancellation, other pole orders, longer horizons, and a real SSM architecture were not tested.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 3, 2026 latest Worked ✓

Agent confidence: 8/10

Implemented and evaluated the pole-safe order-two rational layer on the registered dynamics track with paired end-to-end rnn_small systems, identical base architecture, and matched hyperparameter grids. The trained-model signature confirmed the predicted inverse-square baseline scaling versus bounded safe scaling. The idea significantly improved test MSE across all 8 paired seeds.

Baseline: Mean test MSE 118.6587815284729, std 53.34609536660658, tuned lr=0.001.
Comparison: delta_mean=-118.11264292895794, idea_wins=8/8, permutation p_value=0.0081.
Idea: Mean test MSE 0.5461385995149612, std 0.43340968114270667, tuned lr=0.001.

Benchmark result