Training ideas

Curricula, sampling schedules, pseudo-labeling and multi-task mixing, with the standard training recipe as the control.

2 ideas found

Category: Training Solves: Memory ML: Mlp Clear all

△ Mechanism confirmed, baseline not beaten 2026

Matrix-Free Krylov Backpropagation Through Solver Layers

Turn an iterative optimization or equilibrium computation inside a neural network into a differentiable layer whose backward pass solves the implicit adjoint system with conjugate gradients or GMRES using only automatic-differentiation matrix-vector products. This avoids storing unrolled iterations and avoids explicit Hessian or Jacobian construction, enabling longer solver horizons and lower-memory implicit architectures.

Useful8/10

Difficulty6/10

Novelty5/10

Training Memory Speedup Stability

Paper: PANDA: A Matrix-Free Differentiable NMPC Solver via Proximal Averaged Quasi-Newton with Adaptive Linesearch Algorithm arXiv:2608.16280

Unverified 2026

Inverted recurrent surprise layers

Replace globally backpropagated hidden-layer losses with independent recurrent layers that receive bottom-up, top-down, and lateral inputs. Train each layer to assign low activity to correctly paired input-label examples and high activity to mismatched examples, then classify by selecting the label with the lowest accumulated surprise.

Useful6/10

Difficulty5/10

Novelty6/10

Training Memory Sample efficiency Stability

Paper: From Local Learning to Global Prediction Through Layered Surprise Cascades arXiv:2608.05481