Initialization ideas

Initialization schemes and early-training interventions that change the loss landscape the optimizer sees.

2 ideas found

Category: Initialization Solves: Sample efficiency Math: Probability Clear all

Unverified 2026

Separated Digital-Net Codebook Initialization

Initialize VQ-VAE, product-quantization, or prototype embeddings from a matrix-scrambled digital net after mapping points into the data latent region. This aims to prevent early codebook collisions and dead entries by giving codewords broad coverage and controlled minimum separation, rather than relying on Gaussian initialization or random samples that contain increasingly large local gaps.

Useful6/10

Difficulty4/10

Novelty7/10

Initialization Stability Accuracy Sample efficiency

Paper: Separation properties of scrambled digital nets and related random point sets arXiv:2607.29063

Unverified 2026

Projective stationary-energy initialization

Split a recurrent state into two blocks and initialize their variances and cross-correlation according to the stationary projective energy distribution induced by the transition. This places the initial hidden state near the typical invariant direction of the dynamics instead of forcing a long transient from zero or isotropic noise.

Useful5/10

Difficulty5/10

Novelty7/10

Initialization Stability Sample efficiency Accuracy

Paper: Quantitative Furstenberg Theory for Large Random Matrices arXiv:2608.22543