Spectral Budgeted Embeddings

Integrate this at model construction for every categorical embedding table, before full training. Let table j have N_j categories and maximum candidate dimension D_max. First train a cheap pilot model with all tables at D_max for a small number of steps, or initialize each table with a random or learned matrix E_j in R^{N_j x D_max}. Compute a truncated SVD E_j=U_j diag(sigma_j)V_j^T; if a pilot table is unreliable, use category-conditioned hidden-state means or target/residual statistics instead. Set c_j=N_j, or set c_j=N_j multiplied by the number of optimizer states if the budget is intended to count training memory rather than weights. Estimate V_j=sum_k sigma_{j,k}^2, then compute the continuous dimensions using d_j=B*sqrt(V_j/c_j)/sum_l sqrt(V_l c_l). Clamp each dimension to [d_min,D_max], round to integers, and repair the budget by repeatedly adding one dimension to the table with largest marginal gain sigma_{j,d_j+1}^2 per cost c_j, or removing one from the smallest such ratio. Pseudocode: inputs are E_j, N_j, B; obtain singular values; initialize d_j=max(d_min,round(B*sqrt(V_j/c_j)/Z)); while sum_j c_j d_j>B remove argmin_j sigma_{j,d_j}^2/c_j; while budget remains add argmax_j sigma_{j,d_j+1}^2/c_j; replace each embedding table by its selected-width parameter matrix and train normally. The paper-derived quantities are the spectral-tail approximation proxy, parameter budget, and square-root allocation; the pilot spectrum, minimum dimensions, and rounding policy are empirical engineering choices. Start with a small Transformer or tabular MLP on Criteo, Avazu, or a synthetic task with heterogeneous category effects. Compare uniform dimensions and common cardinality heuristics at exactly equal total embedding parameters. Success is higher validation accuracy or lower log loss at fixed parameter count, especially when tables have different category counts and spectral decay; also measure calibration and whether the pilot allocation remains stable across random seeds.