SynthCore

Initialization ideas

Initialization schemes and early-training interventions that change the loss landscape the optimizer sees.

2 ideas found

Category: Initialization ML: Mlp Math: Probability Clear all
Unverified 2026

Effective-Scale Sparse Projection

Replace a dense token-mixing, MLP, or low-rank-adapter projection with a Bernoulli-signed sparse matrix normalized by the paper's predicted operator scale. Estimate the active representation dimension and use the effective scale to detect regimes in which extreme sparsity is likely to cause unstable amplification or dead rows.

Useful6/10
Difficulty4/10
Novelty5/10
Initialization StabilityMemoryInference speedup
Paper: Level-set entropy and sparse randomized embeddings arXiv:2607.23017
Unverified 2026

Marchenko–Pastur Tensor Initialization

Use Marchenko–Pastur spectral edges to calibrate tensorized random features even when the base vector has exchangeable, sign-symmetric dependent coordinates. Rescale the tensor features and select their retained dimension so the predicted covariance bulk remains well-conditioned instead of assuming independent Gaussian coordinates.

Useful6/10
Difficulty5/10
Novelty7/10
Initialization StabilityAccuracyScalability
Paper: Marchenko-Pastur law for tensor powers of exchangeable unconditional vectors arXiv:2607.21759