# Null-Space-Preserving Consensus Optimizer

- ID: 1127
- Canonical URL: https://synthcore.org/idea/1127/null-space-preserving-consensus-optimizer
- API JSON: https://synthcore.org/api/idea/1127.json
- API Markdown: https://synthcore.org/api/idea/1127.md
- Verification status: beats_baseline
- Source: [arXiv:2607.18913](https://arxiv.org/abs/2607.18913)
- Category: dynamics
- Solves: stability, scalability, accuracy
- ML areas: optimizer, training-dynamics, federated
- Math tags: dynamical-systems, control-theory, linear-algebra, spectral-theory
- Ratings: usefulness 8/10; difficulty 5/10; novelty 6/10

## Idea description

Use a consensus-coupled optimizer for replicated model parameters, but construct every communication perturbation so that the all-ones consensus direction remains in the Laplacian null space. This prevents topology noise, pruning, or heterogeneous communication weights from changing the common parameter trajectory while still allowing disagreement modes to be damped.

## Mathematical statement

Let \(\theta_i\in\mathbb{R}^p\) be the parameters of replica \(i\), stack them as \(\Theta=[\theta_1;\ldots;\theta_N]\), and let \(L\) be a communication Laplacian. The consensus subspace is \(\mathcal C=\{\mathbb{1}\otimes\theta:\theta\in\mathbb{R}^p\}\), because \(L\mathbb{1}=0\). A structure-preserving perturbation \(\Delta L\) must satisfy \(\Delta L\mathbb{1}=0\); an adjacency perturbation that changes row sums generally violates this condition. For a linearized optimizer with local Hessian \(H\), preconditioner \(P\), learning rate \(\eta\), and consensus gain \(\gamma\), the discrete update has modal multipliers \(I-\eta PH-\gamma\lambda_jP\), where \(\lambda_j\) are eigenvalues of the perturbed Laplacian. Stability requires the spectral radius of every disagreement multiplier to be below one, while the consensus multiplier is unchanged when \(\Delta L\mathbb{1}=0\).

## Key formulas

- $$\mathcal C=\operatorname{im}(\mathbb{1}\otimes I_p),\qquad L\mathbb{1}=0,\qquad \Delta L\mathbb{1}=0\ \Longrightarrow\ (L+\Delta L)\mathbb{1}=0.$$
- $$\Theta_{k+1}=\left[I-\eta(I_N\otimes PH)-\gamma(L+\Delta L)\otimes P\right]\Theta_k.$$
- $$\rho\!\left(I-\eta PH-\gamma\lambda_jP\right)<1\quad\text{for every disagreement mode }j\geq2.$$
- $$\Delta L\mathbb{1}\neq0\ \Longrightarrow\ \text{the consensus trajectory receives the forcing }-\gamma(\Delta L\mathbb{1})\otimes P\theta.$$

## Implementation notes

Integrate this into a distributed or multi-replica training loop, such as four data-parallel workers that each maintain a local copy of an MLP or small transformer and communicate through a sparse graph. At every communication step, compute a raw nonnegative edge-weight matrix \(W\), then form a row-sum-preserving Laplacian \(L=D-W\), with \(D_{ii}=\sum_jW_{ij}\). If communication dropout, pruning, or quantization produces a perturbed adjacency \(\widetilde W\), rebuild its degree matrix as \(\widetilde D_{ii}=\sum_j\widetilde W_{ij}\) instead of perturbing \(D\) independently; this guarantees \(\widetilde L\mathbb{1}=0\). Apply `g_i = local_gradient_i + gamma * sum_j L[i,j] * P(theta_j)` and then `theta_i <- theta_i - eta * P(g_i)`. For comparison, implement an adjacency-only baseline that keeps the old degree matrix while perturbing edges. The paper supplies the null-space mechanism and modal decomposition; estimate \(H\) with Hessian-vector products, obtain \(P\) from the optimizer, and estimate spectral radii by power iteration on the update operator. First run CIFAR-10 with four replicas of a two-layer MLP or small ResNet, using random edge dropout and heterogeneous communication delays. Measure the average parameter trajectory \(\bar\theta_k\) against a no-perturbation reference and the disagreement norm \(\|(I-\Pi)\Theta_k\|\), where \(\Pi=(\mathbb{1}\mathbb{1}^T/N)\otimes I_p\). The quantitative prediction is that null-space-preserving perturbations keep common-mode trajectory error near the unperturbed level while changing only disagreement decay, whereas adjacency-only perturbations produce an error proportional to \(\|\Delta L\mathbb{1}\|\). Divergence should begin when the largest estimated modal spectral radius crosses one.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built a self-contained NumPy implementation comparing null-space-preserving and adjacency-only consensus perturbations in a paired replicated quadratic-training experiment. The mechanism manifested: rebuilt-degree Laplacians preserved the null space to 3.14e-16, forcing scaled with row-sum defect at correlation 1.0, and the observed stability boundary was within 1.02% of prediction. Common-mode error was reduced from 2.04e-2 to 2.24e-17 on average while disagreement remained damped.

### Mechanism check

- Verdict: Built a self-contained NumPy implementation comparing null-space-preserving and adjacency-only consensus perturbations in a paired replicated quadratic-training experiment. The mechanism manifested: rebuilt-degree Laplacians preserved the null space to 3.14e-16, forcing scaled with row-sum defect at correlation 1.0, and the observed stability boundary was within 1.02% of prediction. Common-mode error was reduced from 2.04e-2 to 2.24e-17 on average while disagreement remained damped.
- Confidence: 9/10
- Limitations: Only a controlled linear/quadratic NumPy experiment was tested. CIFAR-10 or MLP training, directed nonsymmetric graphs, quantization, delays, stochastic nonlinear gradients, adaptive preconditioners, communication cost, and large-scale distributed execution were not evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [REPORT.md](https://synthcore.org/code/218/REPORT.md)
- [bench_experiment.py](https://synthcore.org/code/218/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/218/bench_report.json)
- [experiment.py](https://synthcore.org/code/218/experiment.py)
- [report.md](https://synthcore.org/code/218/report.md)
- [report_bench_2026-09-02T214032.md](https://synthcore.org/code/218/report_bench_2026-09-02T214032.md)
- [results.json](https://synthcore.org/code/218/results.json)
- [stage2_bench.py](https://synthcore.org/code/218/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/218)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.