# Pivot-separation barrier for polynomial neurons

- ID: 954
- Canonical URL: https://synthcore.org/idea/954/pivot-separation-barrier-for-polynomial-neurons
- API JSON: https://synthcore.org/api/idea/954.json
- API Markdown: https://synthcore.org/api/idea/954.md
- Verification status: unverified
- Source: [arXiv:2607.15173](https://arxiv.org/abs/2607.15173)
- Category: regularization
- Solves: stability, accuracy, generalization
- ML areas: mlp, regularization, training-dynamics
- Math tags: linear-algebra, algebra, optimization, approximation-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 8/10

## Idea description

Add a width- and degree-aware regularizer that prevents hidden polynomial neurons from collapsing to the same pivot. The paper's critical-point analysis says that non-global local minima and nontrivial saddles for cubic activation occur only when all pivots coincide, while global representations require at least d distinct active and visible pivots; the barrier directly targets this degeneracy.

## Mathematical statement

For a unit-weight representation h_j(x)=(x+b_j)^d, the hidden pivots are b_j. The paper uses the Vandermonde matrix with entries V_{i,j}=b_j^i, i=0,\ldots,d-1, and its invertibility is equivalent to distinct pivots. Its determinant is \det V=\prod_{1\le r<s\le d}(b_s-b_r), so a collapsed-pivot configuration has zero determinant and cannot provide the full polynomial basis. The paper's landscape result states that global minimizers require at least d distinct active and visible hidden neurons, whereas for width m<d the zero-loss infimum is not attained and every minimizing sequence diverges. Here m is hidden width, d is monomial degree, an active neuron has nonzero inner and outer weights, and a visible neuron contributes nontrivially to the realization. The adapted regularizer uses a logarithmic repulsion on normalized pivots: R_{sep}=-\frac{1}{m(m-1)}\sum_{r<s}\log((    ilde b_r-    ilde b_s)^2+\varepsilon^2).

## Key formulas

- $$V=\begin{pmatrix}1&1&\cdots&1\\ b_1&b_2&\cdots&b_m\\ b_1^2&b_2^2&\cdots&b_m^2\\ \vdots&\vdots&\ddots&\vdots\\ b_1^{d-1}&b_2^{d-1}&\cdots&b_m^{d-1}\end{pmatrix},\qquad \det V=\prod_{1\le r<s\le d}(b_s-b_r)\quad (m=d).$$
- $$\mathcal{L}_{total}=\mathcal{L}_{data}+\lambda_{sep}R_{sep},\qquad R_{sep}=-\frac{1}{m(m-1)}\sum_{r<s}\log\left((\tilde b_r-\tilde b_s)^2+\varepsilon^2\right).$$
- $$\inf_{\theta}\mathcal{L}(\theta)=0,\qquad m<d\ \Longrightarrow\ \text{the infimum is not attained and every minimizing parameter sequence is unbounded}.$$
- $$\frac{\partial R_{sep}}{\partial \tilde b_r}=-\frac{2}{m(m-1)}\sum_{s\ne r}\frac{\tilde b_r-\tilde b_s}{(\tilde b_r-\tilde b_s)^2+\varepsilon^2}.$$

## Implementation notes

Apply the regularizer to the bias-to-weight pivot b_j/w_j for each polynomial neuron with nonzero inner weight, rather than blindly regularizing raw biases. In an MVP, constrain w_j=1 and use b_j directly. At each minibatch, compute normalized pivots     ilde b_j=    anh(b_j/B) so the repulsion cannot be reduced by sending all biases to infinity; calculate all pairwise differences, evaluate R_sep, add lambda_sep times this term to the data loss, and backpropagate using the displayed gradient. Set epsilon to 1e-3 and begin with lambda_sep between 1e-4 and 1e-2 relative to the data-loss scale. For widths larger than d, repulse all units but optionally use a soft activity gate q_j=|v_jw_j^d|/(|v_jw_j^d|+tau) and weight each pair by q_rq_s, so inactive spare units do not distort the basis. Do not use this term when d=1. The paper supplies the structural criterion and Vandermonde determinant; the engineer estimates only the current pivots, activity scores, and the conditioning proxy kappa(V), computed by SVD on a minibatch or periodically. First test cubic and quintic scalar regression on polynomials plus noisy sinusoids, using widths m=d and m=2d. Compare an unregularized polynomial MLP, standard weight decay, and the barrier at equal training budgets. Track pivot minimum separation, log singular-value spread of V, loss, gradient norm, and held-out error. Success is fewer collapsed-pivot runs, bounded parameter norms, and lower validation error or faster convergence at the same width.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.