Initialization ideas

Initialization schemes and early-training interventions that change the loss landscape the optimizer sees.

2 ideas found

Status: Unverified Category: Initialization Math: Spectral theory Clear all

Unverified 2026

Marchenko–Pastur Tensor Initialization

Use Marchenko–Pastur spectral edges to calibrate tensorized random features even when the base vector has exchangeable, sign-symmetric dependent coordinates. Rescale the tensor features and select their retained dimension so the predicted covariance bulk remains well-conditioned instead of assuming independent Gaussian coordinates.

Useful6/10

Difficulty5/10

Novelty7/10

Initialization Stability Accuracy Scalability

Paper: Marchenko-Pastur law for tensor powers of exchangeable unconditional vectors arXiv:2607.21759

Unverified 2026

Semicircle-Calibrated Additive Initialization

Calibrate the scales of several additive self-adjoint residual or attention operators so their aggregate spectrum has a controlled higher-moment Berry–Esseen certificate. Penalize unusually large normalized (2+δ)-moments, which should reduce spectral outliers and make the summed operator closer to a predictable semicircle-shaped spectrum.

Useful4/10

Difficulty5/10

Novelty7/10

Initialization Stability Accuracy Initialization

Paper: The Berry--Esseen Estimate in the Free Central Limit Theorem arXiv:2608.05866