Initialization ideas

Initialization schemes and early-training interventions that change the loss landscape the optimizer sees.

2 ideas found

Category: Initialization ML: Training dynamics Math: Probability Clear all

Unverified 2026

Projective stationary-energy initialization

Split a recurrent state into two blocks and initialize their variances and cross-correlation according to the stationary projective energy distribution induced by the transition. This places the initial hidden state near the typical invariant direction of the dynamics instead of forcing a long transient from zero or isotropic noise.

Useful5/10

Difficulty5/10

Novelty7/10

Initialization Stability Sample efficiency Accuracy

Paper: Quantitative Furstenberg Theory for Large Random Matrices arXiv:2608.22543

Unverified 2026

Semicircle-Calibrated Additive Initialization

Calibrate the scales of several additive self-adjoint residual or attention operators so their aggregate spectrum has a controlled higher-moment Berry–Esseen certificate. Penalize unusually large normalized (2+δ)-moments, which should reduce spectral outliers and make the summed operator closer to a predictable semicircle-shaped spectrum.

Useful4/10

Difficulty5/10

Novelty7/10

Initialization Stability Accuracy Initialization

Paper: The Berry--Esseen Estimate in the Free Central Limit Theorem arXiv:2608.05866