Initialization ideas

Initialization schemes and early-training interventions that change the loss landscape the optimizer sees.

Unverified 2026

Separated Digital-Net Codebook Initialization

Initialize VQ-VAE, product-quantization, or prototype embeddings from a matrix-scrambled digital net after mapping points into the data latent region. This aims to prevent early codebook collisions and dead entries by giving codewords broad coverage and controlled minimum separation, rather than relying on Gaussian initialization or random samples that contain increasingly large local gaps.

Useful6/10
Difficulty4/10
Novelty7/10
Paper: Separation properties of scrambled digital nets and related random point sets arXiv:2607.29063
Unverified 2026

Marchenko–Pastur Tensor Initialization

Use Marchenko–Pastur spectral edges to calibrate tensorized random features even when the base vector has exchangeable, sign-symmetric dependent coordinates. Rescale the tensor features and select their retained dimension so the predicted covariance bulk remains well-conditioned instead of assuming independent Gaussian coordinates.

Useful6/10
Difficulty5/10
Novelty7/10
Paper: Marchenko-Pastur law for tensor powers of exchangeable unconditional vectors arXiv:2607.21759
Unverified 2026

Projective stationary-energy initialization

Split a recurrent state into two blocks and initialize their variances and cross-correlation according to the stationary projective energy distribution induced by the transition. This places the initial hidden state near the typical invariant direction of the dynamics instead of forcing a long transient from zero or isotropic noise.

Useful5/10
Difficulty5/10
Novelty7/10
Paper: Quantitative Furstenberg Theory for Large Random Matrices arXiv:2608.22543
Unverified 2026

Semicircle-Calibrated Additive Initialization

Calibrate the scales of several additive self-adjoint residual or attention operators so their aggregate spectrum has a controlled higher-moment Berry–Esseen certificate. Penalize unusually large normalized (2+δ)-moments, which should reduce spectral outliers and make the summed operator closer to a predictable semicircle-shaped spectrum.

Useful4/10
Difficulty5/10
Novelty7/10
Paper: The Berry--Esseen Estimate in the Free Central Limit Theorem arXiv:2608.05866