Memory ideas

External memories, state compression, retrieval-style read/write policies and calibrated stopping rules.

2 ideas found

Category: Memory ML: Optimizer Math: Stochastic processes Clear all

✓✓ Beats tuned baseline 2026

Residual-only unbiased gradient compression

Compress only the difference between the current client gradient and a persistent control variate, rather than compressing the full gradient. As the control variate tracks the client gradient, the residual shrinks and the same communication budget produces less compression noise than direct gradient quantization.

Useful7/10

Difficulty4/10

Novelty5/10

Memory Memory Speedup Scalability

Paper: Theoretical Foundations of Communication-Efficient, Robust, and Practical Distributed and Federated Optimization arXiv:2608.06563

Unverified 2026

Decoder-Side Refinement for Heavy-Tailed Gradients

Replace ordinary sign-gradient or fixed-range gradient quantization in distributed training with a non-adaptive one-bit refinement block. Workers generate all localization and multiscale residue bits using public randomness before the server knows the current aggregate gradient; after decoding a coarse center, the server chooses which stored phase and scale interpretation to use. This should reduce clipping and variance for heavy-tailed gradients without requiring a second communication round.

Useful6/10

Difficulty6/10

Novelty7/10

Memory Memory Scalability Stability

Paper: Universal Refinement without Interaction: Order-Optimal 1-Bit Mean Estimation arXiv:2607.24358