# Zoomed and Pole-Safe Rational Activation

- ID: 684
- Canonical URL: https://synthcore.org/idea/684/zoomed-and-pole-safe-rational-activation
- API JSON: https://synthcore.org/api/idea/684.json
- API Markdown: https://synthcore.org/api/idea/684.md
- Verification status: unverified
- Source: [arXiv:2607.10377](https://arxiv.org/abs/2607.10377)
- Category: regularization
- Solves: stability, accuracy, generalization
- ML areas: mlp, transformer, regularization, training-dynamics
- Math tags: approximation-theory, spectral-theory, linear-algebra, numerical-analysis
- Ratings: usefulness 5/10; difficulty 5/10; novelty 6/10

## Idea description

Use a barycentric rational activation or filter whose interpolation nodes are periodically zoomed into the range of preactivations or eigenvalues actually encountered by the network. Protect the layer from catastrophic poles by monitoring the associated generalized eigenproblem and penalizing poles close to the active input interval. This targets rational networks whose expressivity comes from localized poles but whose training is destabilized by denominator zeros.

## Mathematical statement

The paper uses the barycentric rational form q(s)=[sum_{k=1}^K w_k f_k/(s-z_k)]/[sum_{k=1}^K w_k/(s-z_k)], where z_k are distinct support points, f_k are function values, and w_k are barycentric weights. It extracts roots through the generalized eigenproblem A-lambda B, where the displayed pencil has A with first row [1,0,...,0] and subsequent rows containing [1,z_k] in the arrowhead structure, while B has a zero first diagonal entry followed by ones. A backward-stable QZ solve computes roots as exact eigenvalues of a perturbed pencil. Theorem 14 states that if |z_k|=O(1) and a computed eigenvalue lambda-hat is separated from every support point except possibly one, then min_{k not equal to k-hat}|lambda-hat-z_k|=Theta(1) implies a backward-stable root. The neural adaptation uses this separation condition as a pole-safety constraint. For an active preactivation interval [a,b], normalize with u=(2s-a-b)/(b-a), place nodes in the normalized coordinate, and map back with s=(b-a)u/2+(a+b)/2.

## Key formulas

- $$q(s)=\frac{\displaystyle\sum_{k=1}^{K}\frac{w_k f_k}{s-z_k}}{\displaystyle\sum_{k=1}^{K}\frac{w_k}{s-z_k}}$$
- $$A-\lambda B=\begin{bmatrix}1&0&\cdots&0\\1&z_1&&\\1&&z_2&\\\vdots&&&\ddots\end{bmatrix}-\lambda\begin{bmatrix}0&&&\\&1&&\\&&\ddots&\\&&&1\end{bmatrix}$$
- $$\min_{1\le k\ne\widehat{k}\le K}|\widehat{\lambda}-z_k|=\Theta(1)$$
- $$u=\frac{2s-a-b}{b-a},\qquad s=\frac{b-a}{2}u+\frac{a+b}{2}$$

## Implementation notes

Replace GELU in a small MLP or transformer feed-forward block with a K-node barycentric rational activation. Maintain support nodes z[k], values f[k], and weights w[k], and evaluate the displayed barycentric formula directly using float32 or float64 compensated summation. At each batch, compute detached preactivation quantiles a and b, using the 1st and 99th percentiles, then update exponential moving averages `a=.99*a+.01*q01` and `b=.99*b+.01*q99`. Every 500 optimizer steps, relocate the nodes by applying u=(2s-a-b)/(b-a) to the old nodes and mapping the normalized coordinates into the new interval. Keep two support nodes outside [a,b] by a margin m=.1*(b-a). Add a pole-safety penalty `L_pole=lambda*sum_k softplus(m-|p_k-c|)^2`, where c=(a+b)/2 and p_k are generalized eigenvalues of the barycentric pencil A-lambda B. Compute p_k with a QZ or generalized eigensolver once per relocation, not during every forward pass. Refit f[k] after relocation by evaluating the previous activation on 128 points and solving a small least-squares interpolation problem; keep w[k] fixed initially, then unfreeze them in a second experiment. Clip the denominator magnitude with epsilon=1e-6. The paper supplies the pencil and separation criterion; activation quantiles, interval margins, and penalty weights are empirical. First test on CIFAR-10 with a four-layer MLP or tiny ViT, comparing GELU, a fixed rational activation, and the zoomed pole-safe version at equal parameter count. Record validation accuracy, loss spikes, gradient-norm outliers, and denominator minima. Success is equal-or-better accuracy with fewer outlier batches and no denominator blow-ups.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.