# Blockwise anisotropic consensus optimizer

- ID: 677
- Canonical URL: https://synthcore.org/idea/677/blockwise-anisotropic-consensus-optimizer
- API JSON: https://synthcore.org/api/idea/677.json
- API Markdown: https://synthcore.org/api/idea/677.md
- Verification status: unverified
- Source: [arXiv:2607.10205](https://arxiv.org/abs/2607.10205)
- Category: optimization
- Solves: stability, sample-efficiency, accuracy
- ML areas: optimizer, fine-tuning, training-dynamics
- Math tags: optimization, stochastic-processes, linear-algebra, geometry
- Ratings: usefulness 5/10; difficulty 6/10; novelty 7/10

## Idea description

Apply consensus-based derivative-free optimization independently in parameter blocks that are expected to contribute additively to the objective, using noise projected into each block rather than isotropic noise over all parameters. The method is most suitable for low-dimensional trainable objects such as LoRA adapters, soft prompts, calibration vectors, or neural architecture hyperparameters, where maintaining a small population of particles is feasible.

## Mathematical statement

Assumption 1.1 writes the objective as ${\cal E}(x)=\sum_{\ell=1}^{L}{\cal E}_{\ell}({\cal B}_{\ell}^{\top}x)$, where $x\in\mathbb{R}^{D}$ is the complete optimization vector, ${\cal I}_{1},\ldots,{\cal I}_{L}$ partition the coordinates, ${\cal B}_{\ell}^{\top}x=x|_{{\cal I}_{\ell}}\in\mathbb{R}^{d_{\ell}}$ selects block $\ell$, and $\sum_{\ell}d_{\ell}=D$. The associated projector is $P_{\ell}={\cal B}_{\ell}{\cal B}_{\ell}^{\top}$, which keeps only coordinates in block $\ell$. The transferable property is that exploration can be aligned with these orthogonal low-dimensional subspaces instead of injecting noise in all $D$ directions. The adapted optimizer maintains particles $x_i\in\mathbb{R}^{D}$, evaluates ${\cal E}(x_i)$, forms the weighted consensus $x_{\alpha}=\sum_i w_i x_i/\sum_iw_i$ with $w_i=\exp(-\alpha{\cal E}(x_i))$, and updates each particle by deterministic attraction plus blockwise projected Gaussian noise. Here $\lambda>0$ is the consensus attraction rate, $\sigma_{\ell}\geq0$ is the noise scale for block $\ell$, $\Delta t$ is the step size, and $\xi_{i\ell}\sim{\cal N}(0,I_{d_{\ell}})$ is independent Gaussian noise in block coordinates. Exact additive separability is not required for an empirical version, but the method should degrade toward ordinary CBO when blockwise losses are strongly coupled.

## Key formulas

- $${\cal E}(x)=\sum_{{\ell}=1}^{L}{\cal E}_{\ell}({\cal B}^{\top}_{\ell}x),\qquad {\cal B}_{\ell}^{\top}x=x|_{{\cal I}_{\ell}},\qquad \sum_{\ell=1}^{L}d_{\ell}=D.$$
- $$P_{\ell}={\cal B}_{\ell}{\cal B}_{\ell}^{\top},$$
- $$x_{\alpha}=\frac{\sum_{j=1}^{N}\exp(-\alpha {\cal E}(x_j))x_j}{\sum_{j=1}^{N}\exp(-\alpha {\cal E}(x_j))},$$
- $$x_{i}^{t+1}=x_i^t+\lambda\Delta t\,(x_{\alpha}^t-x_i^t)+\sigma\sqrt{\Delta t}\sum_{\ell=1}^{L}{\cal B}_{\ell}\,\operatorname{diag}(s_{\ell})\,\xi_{i\ell}^{t},\qquad \xi_{i\ell}^{t}\sim{\cal N}(0,I_{d_{\ell}}).$$

## Implementation notes

(1) Integration point: use this optimizer instead of AdamW only for a small trainable parameter set, initially a rank-4 or rank-8 LoRA adapter on one or two transformer layers, a soft prompt, or a vector of 10-100 calibration parameters. Flatten the trainable tensors into $x\in\mathbb{R}^{D}$ and define fixed blocks by adapter, layer, or tensor rather than by individual scalar coordinates. Keep the pretrained backbone frozen. (2) Pseudocode: initialize $N=32$ particles $x_i$ around the AdamW solution or around zero; at every iteration evaluate the minibatch loss ${\cal E}(x_i)$ for all particles; compute $w_i=\exp[-\alpha({\cal E}(x_i)-\min_j{\cal E}(x_j))]$ and $x_\alpha=\sum_iw_ix_i/(\sum_iw_i)$; for every particle and block draw $\xi_{i\ell}\sim{\cal N}(0,I)$ and apply $x_i\leftarrow x_i+\lambda\Delta t(x_\alpha-x_i)+\sigma_\ell\sqrt{\Delta t}{\cal B}_\ell\xi_{i\ell}$. Evaluate the final consensus or best particle on held-out data. (3) Computed from the paper: the block selectors ${\cal B}_\ell$, projectors $P_\ell$, and blockwise noise structure. Estimated empirically: loss values, consensus weights, and optionally $\sigma_\ell$; start with equal $\sigma_\ell$, then set $\sigma_\ell\propto1/\sqrt{v_\ell+\epsilon}$ where $v_\ell$ is an exponential moving average of squared block displacement from consensus. Clip weights using the stabilized exponent above. (4) First experiment: compare AdamW, isotropic CBO, and this blockwise CBO on a frozen ViT or small decoder with LoRA fine-tuning on a 10-class subset of CIFAR-100 or a small language-model classification task. Use equal numbers of forward evaluations and report validation accuracy, best loss versus wall-clock time, particle diversity per block, and sensitivity to random seed. A positive result is faster best-loss descent or higher held-out accuracy at equal forward-evaluation budget, especially when adapter blocks are initialized with deliberately different scales.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.