Initialization ideas

Initialization schemes and early-training interventions that change the loss landscape the optimizer sees.

2 ideas found

Category: Initialization ML: Training dynamics Math: Linear algebra Clear all

Unverified 2026

Vandermonde polynomial initialization

Initialize a univariate polynomial-activation hidden layer to realize a prescribed polynomial exactly, rather than relying on gradient descent to learn the required cancellation between shifted monomials. This provides an analytically controlled starting point for polynomial MLPs, polynomial feature extractors, and teacher-to-student initialization when the desired local map is known or fitted from data.

Useful6/10

Difficulty4/10

Novelty7/10

Initialization Accuracy Sample efficiency Stability

Paper: Landscape analysis for shallow neural networks: Complete classification of critical points for cubic activation and affine target functions arXiv:2607.15173

Unverified 2026

Projective stationary-energy initialization

Split a recurrent state into two blocks and initialize their variances and cross-correlation according to the stationary projective energy distribution induced by the transition. This places the initial hidden state near the typical invariant direction of the dynamics instead of forcing a long transient from zero or isotropic noise.

Useful5/10

Difficulty5/10

Novelty7/10

Initialization Stability Sample efficiency Accuracy

Paper: Quantitative Furstenberg Theory for Large Random Matrices arXiv:2608.22543