Training ideas

Curricula, sampling schedules, pseudo-labeling and multi-task mixing, with the standard training recipe as the control.

3 ideas found

Category: Training Solves: Generalization ML: Attention Clear all

Unverified 2026

Sparse-Cost Temperature Calibration

Use a small set of known or trusted pairwise costs to estimate the effective entropic temperature of a Sinkhorn attention or mixture-of-experts routing layer directly from its observed transport plan. This provides a calibration controller that can detect over-concentrated routing and adjust epsilon without backpropagating through a costly temperature search.

Useful6/10

Difficulty4/10

Novelty8/10

Training Stability Accuracy Generalization

Paper: An exact and fast solution of the inverse Regularized Optimal Transport problem arXiv:2609.01278

Unverified 2026

Parity-Copula High-Order Interaction Benchmark

Construct training and evaluation examples whose every (d-1)-variable marginal is exactly independent, but whose full d-variable distribution contains a parity interaction. This isolates genuine high-order reasoning from shortcuts based on pairwise or lower-order statistics and can expose whether attention or MLP architectures learn the intended interaction.

Useful6/10

Difficulty3/10

Novelty6/10

Training Accuracy Generalization Sample efficiency

Paper: How far are $d$-dimensional copulas with uniform $(d-1)$-marginals from (total) independence? arXiv:2608.18286

Unverified 2026

Order-Parameter Mode Activation Schedule

Use the paper's order-parameter dynamics to initialize spectral feature modes with deliberately separated activation times. This creates a controlled progressive-learning curriculum in which dominant modes become available first and weaker modes activate later, potentially reducing early gradient interference.

Useful6/10

Difficulty6/10

Novelty6/10

Training Stability Accuracy Generalization

Paper: Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws arXiv:2608.13335