Dropout and Random Gradient Masking Are Asymptotically Equivalent in Large ResNets

What the math gives to ML

The paper identifies an asymptotic invariance in residual networks: multiplicative activation dropout and unbiased random masking of residual gradients converge to the same limiting training dynamics as depth and width grow in the complete feature-learning regime. This suggests replacing forward-pass dropout with backward-only masking, preserving the unperturbed representation seen by later layers while still injecting stochasticity into parameter updates. The most practical transfer is to compare activation dropout, stochastic depth, and residual-branch gradient masking at matched mask rates, measuring whether backward-only noise provides comparable regularization with lower activation-memory or forward-compute costs.