Training ideas

Curricula, sampling schedules, pseudo-labeling and multi-task mixing, with the standard training recipe as the control.

2 ideas found

Category: Training Solves: Memory Math: Stochastic processes Clear all

△ Mechanism confirmed, baseline not beaten 2026

Input-Subspace Perturbation Learning

Replace full-dimensional node or weight perturbation with perturbations in an input-conditioned d-dimensional tangent subspace, where d is the input or feature dimension and is much smaller than the reservoir width or parameter count. Estimate the update using only scalar self-supervised losses from positive and negative perturbations, then map the low-dimensional update back to the trainable parameters.

Useful8/10

Difficulty5/10

Novelty7/10

Training Speedup Stability Memory

Paper: Scalable Perturbation Learning for Online Self-Supervised Learning in Echo State Networks arXiv:2607.06079

✗ Mechanism failed 2026

Two-level local/coarse GNN training

Partition a large graph into induced subgraphs and perform most parameter updates using only local subgraphs, interleaving them with inexpensive global updates on a randomly subsampled coarse graph. The coarse correction preserves information about cross-partition dependencies while reducing full-graph message passing and communication cost.

Useful8/10

Difficulty5/10

Novelty6/10

Training Speedup Scalability Memory

Paper: Two-level domain-decomposition AdaGrad method for scalable training of graph neural networks arXiv:2608.22575