Pivot-separation barrier for polynomial neurons
Implementation & benchmark of arXiv:2607.15173 — Landscape analysis for shallow neural networks: Complete classification of critical points for cubic activation and affine target functions
Source paper: Landscape analysis for shallow neural networks: Complete classification of critical points for cubic activation and affine target functions arXiv:2607.15173 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Add a width- and degree-aware regularizer that prevents hidden polynomial neurons from collapsing to the same pivot. The paper's critical-point analysis says that non-global local minima and nontrivial saddles for cubic activation occur only when all pivots coincide, while global representations require at least d distinct active and visible pivots; the barrier directly targets this degeneracy.
Formulas
Mathematical statement
For a unit-weight representation h_j(x)=(x+b_j)^d, the hidden pivots are b_j. The paper uses the Vandermonde matrix with entries V_{i,j}=b_j^i, i=0,\ldots,d-1, and its invertibility is equivalent to distinct pivots. Its determinant is \det V=\prod_{1\le r<s\le d}(b_s-b_r), so a collapsed-pivot configuration has zero determinant and cannot provide the full polynomial basis. The paper's landscape result states that global minimizers require at least d distinct active and visible hidden neurons, whereas for width m<d the zero-loss infimum is not attained and every minimizing sequence diverges. Here m is hidden width, d is monomial degree, an active neuron has nonzero inner and outer weights, and a visible neuron contributes nontrivially to the realization. The adapted regularizer uses a logarithmic repulsion on normalized pivots: R_{sep}=-\frac{1}{m(m-1)}\sum_{r<s}\log(( ilde b_r- ilde b_s)^2+\varepsilon^2).
Implementation notes
Apply the regularizer to the bias-to-weight pivot b_j/w_j for each polynomial neuron with nonzero inner weight, rather than blindly regularizing raw biases. In an MVP, constrain w_j=1 and use b_j directly. At each minibatch, compute normalized pivots ilde b_j= anh(b_j/B) so the repulsion cannot be reduced by sending all biases to infinity; calculate all pairwise differences, evaluate R_sep, add lambda_sep times this term to the data loss, and backpropagate using the displayed gradient. Set epsilon to 1e-3 and begin with lambda_sep between 1e-4 and 1e-2 relative to the data-loss scale. For widths larger than d, repulse all units but optionally use a soft activity gate q_j=|v_jw_j^d|/(|v_jw_j^d|+tau) and weight each pair by q_rq_s, so inactive spare units do not distort the basis. Do not use this term when d=1. The paper supplies the structural criterion and Vandermonde determinant; the engineer estimates only the current pivots, activity scores, and the conditioning proxy kappa(V), computed by SVD on a minibatch or periodically. First test cubic and quintic scalar regression on polynomials plus noisy sinusoids, using widths m=d and m=2d. Compare an unregularized polynomial MLP, standard weight decay, and the barrier at equal training budgets. Track pivot minimum separation, log singular-value spread of V, loss, gradient norm, and held-out error. Success is fewer collapsed-pivot runs, bounded parameter norms, and lower validation error or faster convergence at the same width.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.