Width-Independent Compressibility of Deep Neural Networks

What the math gives to ML

The paper gives a constructive explanation for why a fixed, very wide analytic-activation MLP can be replaced by a much narrower network while preserving its function uniformly on a bounded input domain. The key transferable asset is width scaling that depends on effective input dimension and error tolerance, rather than teacher width: O((log(1/epsilon))^d_in). Its derivative-matching construction suggests a data-free compression or distillation method that matches local input derivatives of each layer, while layer-wise reweighting prevents approximation errors from compounding through depth. The most promising engineering test is a teacher-network compressor that uses automatic differentiation to fit narrow layers against values and derivatives at domain anchors.