Unverified 2026

Pivot-separation barrier for polynomial neurons

Implementation & benchmark of arXiv:2607.15173 — Landscape analysis for shallow neural networks: Complete classification of critical points for cubic activation and affine target functions

Usefulness6/10
Difficulty5/10
Novelty8/10

Source paper: Landscape analysis for shallow neural networks: Complete classification of critical points for cubic activation and affine target functions arXiv:2607.15173 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a width- and degree-aware regularizer that prevents hidden polynomial neurons from collapsing to the same pivot. The paper's critical-point analysis says that non-global local minima and nontrivial saddles for cubic activation occur only when all pivots coincide, while global representations require at least d distinct active and visible pivots; the barrier directly targets this degeneracy.

Formulas

$$V=\begin{pmatrix}1&1&\cdots&1\\ b_1&b_2&\cdots&b_m\\ b_1^2&b_2^2&\cdots&b_m^2\\ \vdots&\vdots&\ddots&\vdots\\ b_1^{d-1}&b_2^{d-1}&\cdots&b_m^{d-1}\end{pmatrix},\qquad \det V=\prod_{1\le r<s\le d}(b_s-b_r)\quad (m=d).$$
$$\mathcal{L}_{total}=\mathcal{L}_{data}+\lambda_{sep}R_{sep},\qquad R_{sep}=-\frac{1}{m(m-1)}\sum_{r<s}\log\left((\tilde b_r-\tilde b_s)^2+\varepsilon^2\right).$$
$$\inf_{\theta}\mathcal{L}(\theta)=0,\qquad m<d\ \Longrightarrow\ \text{the infimum is not attained and every minimizing parameter sequence is unbounded}.$$
$$\frac{\partial R_{sep}}{\partial \tilde b_r}=-\frac{2}{m(m-1)}\sum_{s\ne r}\frac{\tilde b_r-\tilde b_s}{(\tilde b_r-\tilde b_s)^2+\varepsilon^2}.$$

Mathematical statement

For a unit-weight representation h_j(x)=(x+b_j)^d, the hidden pivots are b_j. The paper uses the Vandermonde matrix with entries V_{i,j}=b_j^i, i=0,\ldots,d-1, and its invertibility is equivalent to distinct pivots. Its determinant is \det V=\prod_{1\le r<s\le d}(b_s-b_r), so a collapsed-pivot configuration has zero determinant and cannot provide the full polynomial basis. The paper's landscape result states that global minimizers require at least d distinct active and visible hidden neurons, whereas for width m<d the zero-loss infimum is not attained and every minimizing sequence diverges. Here m is hidden width, d is monomial degree, an active neuron has nonzero inner and outer weights, and a visible neuron contributes nontrivially to the realization. The adapted regularizer uses a logarithmic repulsion on normalized pivots: R_{sep}=-\frac{1}{m(m-1)}\sum_{r<s}\log(( ilde b_r- ilde b_s)^2+\varepsilon^2).

Implementation notes

Apply the regularizer to the bias-to-weight pivot b_j/w_j for each polynomial neuron with nonzero inner weight, rather than blindly regularizing raw biases. In an MVP, constrain w_j=1 and use b_j directly. At each minibatch, compute normalized pivots ilde b_j= anh(b_j/B) so the repulsion cannot be reduced by sending all biases to infinity; calculate all pairwise differences, evaluate R_sep, add lambda_sep times this term to the data loss, and backpropagate using the displayed gradient. Set epsilon to 1e-3 and begin with lambda_sep between 1e-4 and 1e-2 relative to the data-loss scale. For widths larger than d, repulse all units but optionally use a soft activity gate q_j=|v_jw_j^d|/(|v_jw_j^d|+tau) and weight each pair by q_rq_s, so inactive spare units do not distort the basis. Do not use this term when d=1. The paper supplies the structural criterion and Vandermonde determinant; the engineer estimates only the current pivots, activity scores, and the conditioning proxy kappa(V), computed by SVD on a minibatch or periodically. First test cubic and quintic scalar regression on polynomials plus noisy sinusoids, using widths m=d and m=2d. Compare an unregularized polynomial MLP, standard weight decay, and the barrier at equal training budgets. Track pivot minimum separation, log singular-value spread of V, loss, gradient norm, and held-out error. Success is fewer collapsed-pivot runs, bounded parameter norms, and lower validation error or faster convergence at the same width.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.