Null-Space-Preserving Consensus Optimizer

Integrate this into a distributed or multi-replica training loop, such as four data-parallel workers that each maintain a local copy of an MLP or small transformer and communicate through a sparse graph. At every communication step, compute a raw nonnegative edge-weight matrix \(W\), then form a row-sum-preserving Laplacian \(L=D-W\), with \(D_{ii}=\sum_jW_{ij}\). If communication dropout, pruning, or quantization produces a perturbed adjacency \(\widetilde W\), rebuild its degree matrix as \(\widetilde D_{ii}=\sum_j\widetilde W_{ij}\) instead of perturbing \(D\) independently; this guarantees \(\widetilde L\mathbb{1}=0\). Apply g_i = local_gradient_i + gamma * sum_j L[i,j] * P(theta_j) and then theta_i <- theta_i - eta * P(g_i). For comparison, implement an adjacency-only baseline that keeps the old degree matrix while perturbing edges. The paper supplies the null-space mechanism and modal decomposition; estimate \(H\) with Hessian-vector products, obtain \(P\) from the optimizer, and estimate spectral radii by power iteration on the update operator. First run CIFAR-10 with four replicas of a two-layer MLP or small ResNet, using random edge dropout and heterogeneous communication delays. Measure the average parameter trajectory \(\bar\theta_k\) against a no-perturbation reference and the disagreement norm \(\|(I-\Pi)\Theta_k\|\), where \(\Pi=(\mathbb{1}\mathbb{1}^T/N)\otimes I_p\). The quantitative prediction is that null-space-preserving perturbations keep common-mode trajectory error near the unperturbed level while changing only disagreement decay, whereas adjacency-only perturbations produce an error proportional to \(\|\Delta L\mathbb{1}\|\). Divergence should begin when the largest estimated modal spectral radius crosses one.