# Pole-safe rational neural layer

- ID: 297
- Canonical URL: https://synthcore.org/idea/297/pole-safe-rational-neural-layer
- API JSON: https://synthcore.org/api/idea/297.json
- API Markdown: https://synthcore.org/api/idea/297.md
- Verification status: beats_baseline
- Source: [arXiv:2607.00097](https://arxiv.org/abs/2607.00097)
- Category: architecture
- Solves: stability, accuracy
- ML areas: ssm, mlp, training-dynamics
- Math tags: functional-analysis, linear-algebra, approximation-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 8/10

## Idea description

Replace an unconstrained feature vector entering a rational or resolvent-like neural operator by a polynomial feature whose first nonzero Taylor coefficient lies in a pole-safe subspace. For a pole of order m, the simplest guaranteed construction is psi(z)=(z-beta)^m v, which makes Q(z)psi(z) bounded even when Q(z) diverges. For lower-order cancellation, solve linear constraints among Taylor coefficients of psi so that all negative Laurent powers vanish.

## Mathematical statement

The paper defines a pole of order m at beta by the Laurent expansion Q(z)=sum_{j=-m}^{infinity} Q_j (z-beta)^j, where Q_j are bounded linear operators and Q_{-m} is nonzero. A maximal-order pole-cancellation function has the form psi(z)=(z-beta)^m psi_m+(z-beta)^{m+1} psi_{m+1}+O((z-beta)^{m+2}), with psi_m nonzero and Q_{-m} psi_m nonzero. The product phi_hat(z):=Q(z)psi(z) is finite at beta, with phi_hat(beta)=Q_{-m} psi_m. More generally, if psi(z)=sum_{l=k}^{infinity} psi_l (z-beta)^l, the coefficient of (z-beta)^r in Q(z)psi(z) is sum_{j+l=r} Q_j psi_l; cancellation requires these sums to equal zero for every r<0. Here z is the scalar spectral variable, beta is the pole location, Q_j are Laurent coefficients, and psi_l are vector-valued Taylor coefficients. The order-two example in the paper reduces the maximal-order condition to Q_{-2} psi_2 not equal to zero.

## Key formulas

- $$Q(z)=\sum_{j=-m}^{\infty}Q_j(z-\beta)^j.$$
- $$\boldsymbol{\psi}(z)=(z-\beta)^m\boldsymbol{\psi}_m+(z-\beta)^{m+1}\boldsymbol{\psi}_{m+1}+O((z-\beta)^{m+2}),\qquad \boldsymbol{\psi}_m\neq 0.$$
- $$Q_{-m}\boldsymbol{\psi}_m\neq 0.$$
- $$\boldsymbol{\hat{\varphi}}(z):=Q(z)\boldsymbol{\psi}(z),\qquad \boldsymbol{\hat{\varphi}}(\beta)=Q_{-m}\boldsymbol{\psi}_m.$$

## Implementation notes

(1) Integration point: use this construction in a rational spectral layer or diagonal/state-space layer that computes y(z,x)=Q_theta(z)h_theta(x) for scalar frequencies z, especially when z can approach a learned pole beta. Replace the unconstrained vector h_theta(x) by psi_theta(z,x), and return y=Q_theta(z)psi_theta(z,x). Store Laurent coefficients Q[-m],...,Q[R] explicitly, or obtain them from a rational parameterization.

(2) MVP pseudocode:
inputs: x, z, beta, pole_order m, Laurent matrices Q[-m:R]
v = h_theta(x)
psi_m = v / (norm(v)+epsilon)
psi = (z-beta)^m * psi_m
if lower_order_cancellation:
    predict psi[k],...,psi[R] with neural heads
    for r in {-m,...,-1}:
        enforce sum over j+l=r of Q[j] @ psi[l] = 0
phi = sum over j of Q[j] @ psi(z)
y = phi

For the guaranteed maximal-order version, only psi_m is needed; higher coefficients can be added as residual features. For lower-order cancellation, parameterize the feasible coefficient vectors with an SVD-derived nullspace of the linear constraint matrix instead of using only a soft penalty. If beta and z are complex, represent real and imaginary parts as a doubled real system.

(3) Computed directly from the paper's mathematics: multiplication by (z-beta)^m, the Laurent-product coefficient constraints, and the finite limit Q_{-m} psi_m. Estimated empirically: pole locations beta and order m when they are unknown. Fit a local Laurent model to Q_theta(z) over sampled z, select m using held-out reconstruction error and residual growth, and estimate the relevant nullspace with SVD. Monitor the smallest singular value of the constraint matrix to detect ill-conditioning.

(4) First experiment: construct a 32-dimensional synthetic rational state-space operator Q(z)=Q_{-2}(z-beta)^{-2}+Q_{-1}(z-beta)^{-1}+Q_0 with beta=1, then train a small MLP to regress sequences sampled with z in [0.95,1.05]. Compare an unconstrained branch, gradient clipping, and the pole-safe branch at equal parameter count. Measure maximum activation norm, gradient norm, NaN frequency, validation MSE, and convergence speed. Success means bounded activations and gradients near z=beta, no numerical blow-ups, and lower validation error than clipping at equal training cost. Then test the module in a small rational or S4-like SSM on sequential MNIST or a Long Range Arena task, measuring stability and accuracy at sequence lengths where the unconstrained baseline becomes ill-conditioned.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built a pole-safe order-two rational layer using psi(z,x)=(z-beta)^2 h_theta(x)/||h_theta(x)|| and compared it with unconstrained and gradient-clipped controls. The math check reproduced the expected inverse-square divergence for the unsafe product and a bounded, near-constant safe product converging to Q_-2 v. In the synthetic regression, the safe branch achieved validation MSE 0.0227 versus 7.996e11 unconstrained and 1.337e11 with clipping, with output norm about 1.49 versus roughly 1.8e7–2.0e7. This is a clear stability and optimization signal, but only on a synthetic one-seed benchmark.

### Mechanism check

- Verdict: Built a pole-safe order-two rational layer using psi(z,x)=(z-beta)^2 h_theta(x)/||h_theta(x)|| and compared it with unconstrained and gradient-clipped controls. The math check reproduced the expected inverse-square divergence for the unsafe product and a bounded, near-constant safe product converging to Q_-2 v. In the synthetic regression, the safe branch achieved validation MSE 0.0227 versus 7.996e11 unconstrained and 1.337e11 with clipping, with output norm about 1.49 versus roughly 1.8e7–2.0e7. This is a clear stability and optimization signal, but only on a synthetic one-seed benchmark.
- Confidence: 8/10
- Limitations: Only a synthetic 32-dimensional order-two operator, one fixed random seed, and one short regression setup were tested. No real state-space model, sequential benchmark, learned pole, lower-order nullspace parameterization, ablation over pole orders, or repeated-seed statistical evaluation was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [README.md](https://synthcore.org/code/1257/README.md)
- [bench_report.json](https://synthcore.org/code/1257/bench_report.json)
- [pole_safe_experiment.py](https://synthcore.org/code/1257/pole_safe_experiment.py)
- [report.md](https://synthcore.org/code/1257/report.md)
- [report_bench_2026-09-03T225017.md](https://synthcore.org/code/1257/report_bench_2026-09-03T225017.md)
- [results.json](https://synthcore.org/code/1257/results.json)
- [stage2_bench.py](https://synthcore.org/code/1257/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1257)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.