Optimizer ideas

Optimizer update-rule variants and parameter-specific step sizes, benchmarked against a learning-rate-tuned baseline.

1 idea found

Category: Optimizer ML: Mlp Math: Statistics Clear all

✗ Failed on benchmark 2026

Two-sided conditioned DFA

Replace the raw DFA outer-product update with a damped left-right preconditioned update that whitens both presynaptic activity directions and local-error directions. The activity factor removes nuisance-dominated input anisotropy, while the error factor equalizes postsynaptic credit coordinates; separate damping prevents noisy error covariances from destabilizing training.

Useful7/10

Difficulty5/10

Novelty5/10

Optimizer Accuracy Stability Sample efficiency

Paper: Conditioned Direct Feedback Alignment via Activity and Error Geometry arXiv:2607.18574