Training ideas

Curricula, sampling schedules, pseudo-labeling and multi-task mixing, with the standard training recipe as the control.

2 ideas found

Status: Mechanism failed Category: Training Solves: Memory Clear all

✗ Mechanism failed 2026

Two-level local/coarse GNN training

Partition a large graph into induced subgraphs and perform most parameter updates using only local subgraphs, interleaving them with inexpensive global updates on a randomly subsampled coarse graph. The coarse correction preserves information about cross-partition dependencies while reducing full-graph message passing and communication cost.

Useful8/10

Difficulty5/10

Novelty6/10

Training Speedup Scalability Memory

Paper: Two-level domain-decomposition AdaGrad method for scalable training of graph neural networks arXiv:2608.22575

✗ Mechanism failed 2026

Recorded-Mesh Neural ODE Backpropagation

Run an adaptive neural ODE solver once to determine accepted step sizes, then train using a regular fixed-length replay of those steps rather than differentiating through adaptive accept/reject logic. The replay can be fused across a batch of trajectories and differentiated with an ordinary reverse sweep, giving the exact discrete gradient of the replayed solver and predictable GPU work.

Useful7/10

Difficulty5/10

Novelty6/10

Training Speedup Memory Scalability

Paper: GRADSOLVE: fast exact gradients for ODE ensembles on GPUs arXiv:2609.02876