Unverified 2026

Vandermonde polynomial initialization

Implementation & benchmark of arXiv:2607.15173 — Landscape analysis for shallow neural networks: Complete classification of critical points for cubic activation and affine target functions

Usefulness6/10
Difficulty4/10
Novelty7/10

Source paper: Landscape analysis for shallow neural networks: Complete classification of critical points for cubic activation and affine target functions arXiv:2607.15173 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Initialize a univariate polynomial-activation hidden layer to realize a prescribed polynomial exactly, rather than relying on gradient descent to learn the required cancellation between shifted monomials. This provides an analytically controlled starting point for polynomial MLPs, polynomial feature extractors, and teacher-to-student initialization when the desired local map is known or fitted from data.

Formulas

$$\sum_{j=1}^{d}\mathfrak{a}_{j}[\mathfrak{b}_{j}]^{i}=p_{d-i}\binom{d}{i}^{-1},\qquad i=0,1,\ldots,d-1.$$
$$\mathfrak{c}+\sum_{j=1}^{d}\mathfrak{a}_{j}(x+\mathfrak{b}_{j})^{d}=p_0+\sum_{k=1}^{d}p_kx^k,$$
$$\begin{pmatrix}1&\cdots&1\\ \mathfrak{b}_1&\cdots&\mathfrak{b}_d\\ \vdots&\ddots&\vdots\\ \mathfrak{b}_1^{d-1}&\cdots&\mathfrak{b}_d^{d-1}\end{pmatrix}\begin{pmatrix}\mathfrak{a}_1\\ \vdots\\ \mathfrak{a}_d\end{pmatrix}=\begin{pmatrix}p_d\binom{d}{0}^{-1}\\ p_{d-1}\binom{d}{1}^{-1}\\ \vdots\\ p_1\binom{d}{d-1}^{-1}\end{pmatrix}.$$
$$(a+b)^d=\sum_{i=0}^{d}\binom{d}{i}a^ib^{d-i}.$$

Mathematical statement

The paper's representability construction states that for any polynomial P(x)=p_0+p_1x+\cdots+p_dx^d, one can choose d distinct pivots \mathfrak{b}_1,\ldots,\mathfrak{b}_d and coefficients \mathfrak{a}_1,\ldots,\mathfrak{a}_d such that P(x)=\mathfrak{c}+\sum_{j=1}^d\mathfrak{a}_j(x+\mathfrak{b}_j)^d. The coefficients solve the Vandermonde system \sum_{j=1}^d\mathfrak{a}_j\mathfrak{b}_j^i=p_{d-i}\binom{d}{i}^{-1} for i=0,\ldots,d-1, with \mathfrak{c}=p_0-\sum_j\mathfrak{a}_j\mathfrak{b}_j^d. The system is invertible because the pivots are distinct. In the neural-network adaptation, d is the activation degree, \mathfrak{b}_j are hidden biases with unit inner weights, \mathfrak{a}_j are output weights, and \mathfrak{c} is the output bias.

Implementation notes

Integrate this at initialization of a scalar polynomial layer whose hidden units compute h_j(x)=(w_jx+b_j)^d and whose output is y=c+\sum_jv_jh_j(x). The simplest stable construction sets every w_j=1, chooses d pivots b_j on a bounded interval such as equally spaced points in [-1,1], and solves the Vandermonde linear system before training. Given desired polynomial coefficients p_0,\ldots,p_d, construct the matrix V[i,j]=b_j^i for i=0,\ldots,d-1, solve V a=r with r_i=p_{d-i}/\binom{d}{i}, set v_j=a_j, and set c=p_0-\sum_j a_jb_j^d. Use QR or SVD rather than an explicit inverse; rescale x into [-1,1] first because equally spaced Vandermonde systems become ill-conditioned at large degree. For a target affine map, use p_0 and p_1 from a linear regression and set p_k=0 for k\ge2. Optionally freeze this layer for a warm-up period, then unfreeze all parameters. What is computed analytically is the exact coefficient synthesis; what is estimated empirically is the fitted polynomial coefficients and whether later optimization preserves useful conditioning. First test d=3,5,7 on a one-dimensional regression benchmark such as fitting sin(x) on [-1,1], comparing random Xavier initialization against Vandermonde initialization at equal width and optimizer settings. Measure initial function error, loss after a fixed number of updates, gradient norms, and frequency of divergence. Success is lower initial loss and faster loss reduction without large output-weight explosions.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.