Training ideas

Curricula, sampling schedules, pseudo-labeling and multi-task mixing, with the standard training recipe as the control.

2 ideas found

Category: Training Solves: Memory ML: Transformer Clear all

△ Mechanism confirmed, baseline not beaten 2026

Matrix-Free Krylov Backpropagation Through Solver Layers

Turn an iterative optimization or equilibrium computation inside a neural network into a differentiable layer whose backward pass solves the implicit adjoint system with conjugate gradients or GMRES using only automatic-differentiation matrix-vector products. This avoids storing unrolled iterations and avoids explicit Hessian or Jacobian construction, enabling longer solver horizons and lower-memory implicit architectures.

Useful8/10

Difficulty6/10

Novelty5/10

Training Memory Speedup Stability

Paper: PANDA: A Matrix-Free Differentiable NMPC Solver via Proximal Averaged Quasi-Newton with Adaptive Linesearch Algorithm arXiv:2608.16280

Unverified 2026

Stochastic Operator Stability Monitor

Instrument selected neural-network operators with cheap stochastic perturbations and estimate how much their outputs change under finite-precision perturbations. Use the resulting per-operator score to identify unstable kernels and selectively switch them to FP32, compensated accumulation, or a stable reformulation instead of running the entire model at high precision.

Useful6/10

Difficulty5/10

Novelty6/10

Training Stability Speedup Memory

Paper: Automated Numerical Stability Analysis of Deep Learning Operators arXiv:2607.25494