Initialization ideas

Initialization schemes and early-training interventions that change the loss landscape the optimizer sees.

2 ideas found

Category: Initialization ML: Mlp Math: Spectral theory Clear all

△ Mechanism confirmed, baseline not beaten 2026

Spectral subspace initialization for nonlinear teachers

Use a bounded function of the response to form a supervised, label-weighted covariance of the input and initialize the first neural layer from its leading outlier eigenspace. For vector-valued responses, use a matrix-valued response preprocessing map so several label statistics are combined in one lifted spectral estimator.

Useful8/10

Difficulty5/10

Novelty7/10

Initialization Sample efficiency Accuracy Stability

Paper: Spectral phase transitions in Gaussian multi-index models arXiv:2608.12183

Unverified 2026

Marchenko–Pastur Tensor Initialization

Use Marchenko–Pastur spectral edges to calibrate tensorized random features even when the base vector has exchangeable, sign-symmetric dependent coordinates. Rescale the tensor features and select their retained dimension so the predicted covariance bulk remains well-conditioned instead of assuming independent Gaussian coordinates.

Useful6/10

Difficulty5/10

Novelty7/10

Initialization Stability Accuracy Scalability

Paper: Marchenko-Pastur law for tensor powers of exchangeable unconditional vectors arXiv:2607.21759