Initialization ideas

Initialization schemes and early-training interventions that change the loss landscape the optimizer sees.

2 ideas found

Category: Initialization Math: Geometry Clear all

Unverified 2026

Separated Digital-Net Codebook Initialization

Initialize VQ-VAE, product-quantization, or prototype embeddings from a matrix-scrambled digital net after mapping points into the data latent region. This aims to prevent early codebook collisions and dead entries by giving codewords broad coverage and controlled minimum separation, rather than relying on Gaussian initialization or random samples that contain increasingly large local gaps.

Useful6/10

Difficulty4/10

Novelty7/10

Initialization Stability Accuracy Sample efficiency

Paper: Separation properties of scrambled digital nets and related random point sets arXiv:2607.29063

Unverified 2026

Spherical-Design Prototype Initialization

Initialize directional prototypes, cosine-classifier weights, or angular attention directions with a spherical t-design rather than iid random vectors. Exact matching of spherical moments through degree t should provide uniform angular coverage and reduce initialization anisotropy, especially when the number of prototypes is small.

Useful5/10

Difficulty4/10

Novelty5/10

Initialization Accuracy Sample efficiency Stability

Paper: Dual Geometry of Spherical Designs: Polarity, Self-Polar Rigidity, and Quadrature Structure arXiv:2609.02439