# Parity-block curvature preconditioner

- ID: 3026
- Canonical URL: https://synthcore.org/idea/3026/parity-block-curvature-preconditioner
- API JSON: https://synthcore.org/api/idea/3026.json
- API Markdown: https://synthcore.org/api/idea/3026.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2609.01857](https://arxiv.org/abs/2609.01857)
- Category: architecture
- Solves: stability, speedup, generalization
- ML areas: optimizer, cnn, transformer, training-dynamics
- Math tags: linear-algebra, spectral-theory, geometry, optimization
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

When a model, dataset, or parameterization has an involutive symmetry, transform gradients and curvature into symmetry sectors before applying adaptive updates. The Hessian and optimizer can then be handled as independent even and odd blocks, preventing curvature from one sector from forcing an unnecessarily conservative learning rate in the other.

## Mathematical statement

For a reflection operator $R$ satisfying $R^2=I$, the paper constructs bases $B_+$ and $B_-$ for the $+1$ and $-1$ eigenspaces and, in the orthogonal case, forms $Q=[B_+\mid B_-]\in O(6)$. Reflection-invariant quadratic forms become block diagonal in these coordinates: $Q^THQ=\operatorname{diag}(H_+,H_-)$, so generalized eigenvalues and degeneracies can be computed independently. The paper also emphasizes that a symmetry action may be non-orthogonal, in which case $Q$ is invertible but $Q^TQ\ne I$; the correct reduction is by congruence using metric matrices rather than by assuming Euclidean orthogonality. For neural networks, let $S\in\mathbb R^{d\times d}$ be an involutive parameter transformation preserving the loss locally or exactly. Define projectors $P_+=(I+S)/2$ and $P_-=(I-S)/2$. If $S^THS=H$, then $P_+HP_-=0$ and the local quadratic model separates into even and odd sectors. Sector-specific preconditioners $A_+$ and $A_-$ can use separate step sizes with stability ceilings $\eta_\pm<2/\lambda_{\max}(A_\pm^{1/2}H_\pm A_\pm^{1/2})$.

## Key formulas

- $$R^2=I,\qquad P_{+}=\frac{I+R}{2},\qquad P_{-}=\frac{I-R}{2},\qquad P_{+}P_{-}=0.$$
- $$Q=[B_{+}\mid B_{-}],\qquad Q^{T}HQ=\begin{pmatrix}H_{+}&0\\0&H_{-}\end{pmatrix},\qquad H_{\pm}=B_{\pm}^{T}HB_{\pm}.$$
- $$\eta_{\pm,c}=\frac{2}{\lambda_{\max}\!\left(A_{\pm}^{1/2}H_{\pm}A_{\pm}^{1/2}\right)},\qquad \Delta\theta_{\pm}=-\eta_{\pm}A_{\pm}P_{\pm}g.$$
- $$S^{T}HS=H\ \Longrightarrow\ P_{+}HP_{-}=0;\qquad S^{T}S\ne I\ \Longrightarrow\text{use the generalized pencil }(H,G),\ G=Q^{T}Q.$$

## Implementation notes

Choose a concrete involution with a known network meaning. The cheapest test is a two-branch MLP or CNN with branch-swap symmetry: $S$ swaps the branch parameters, so $P_+$ gives shared or even parameters and $P_-$ gives antisymmetric or odd parameters. A second test is a transformer with paired attention heads, where $S$ exchanges each head pair. At every step compute $g_+=P_+g$ and $g_-=P_-g$, maintain separate RMS or diagonal curvature estimates $A_+$ and $A_-$, and update $\theta\leftarrow\theta-\eta_+A_+g_+-\eta_-A_-g_-$. Estimate each sector's top generalized curvature by one Hutchinson or Lanczos pass restricted to that sector. If the implemented symmetry is only a coordinate relabeling with a non-Euclidean parameter metric, construct a basis $Q=[B_+|B_-]$ by eigendecomposition of $S$ and solve $H_\pm v=\lambda G_\pm v$ with $G_\pm=B_\pm^TB_\pm$ rather than treating the basis as orthonormal. The paper supplies the parity-sector and non-orthogonal congruence reduction; curvature estimates and optimizer states are empirical. First experiment: train a two-branch CIFAR-10 CNN with tied initialization and branch-swap-invariant loss, comparing AdamW against sector-wise AdamW. Log $\lambda_{\max,+}$, $\lambda_{\max,-}$, gradient norms, and validation accuracy. The prediction is that the measured stable step-size boundary for independently scaled sectors follows $\eta_{\pm,c}=2/\lambda_{\max,\pm}$ within 20%, and that one sector can tolerate at least a 1.5-fold different learning rate whenever its top curvature differs by that factor.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a readable parity-sector quadratic preconditioner experiment with orthogonal and non-orthogonal involutions. The orthogonal check observed numerical block decoupling (cross-block norm 7.4e-15), a 15x sector curvature ratio, and stability boundaries matching 2/lambda_max: 0.99x was stable while 1.01x was unstable in both sectors. Sector-specific steps reduced loss substantially faster than the conservative global step, but this is a toy quadratic result rather than evidence from a neural-network dataset.

### Mechanism check

- Verdict: Built a readable parity-sector quadratic preconditioner experiment with orthogonal and non-orthogonal involutions. The orthogonal check observed numerical block decoupling (cross-block norm 7.4e-15), a 15x sector curvature ratio, and stability boundaries matching 2/lambda_max: 0.99x was stable while 1.01x was unstable in both sectors. Sector-specific steps reduced loss substantially faster than the conservative global step, but this is a toy quadratic result rather than evidence from a neural-network dataset.
- Confidence: 8/10
- Limitations: No CIFAR-10 CNN, AdamW implementation, stochastic training, wall-clock/FLOP measurement, validation accuracy, or learned curvature estimator was tested. The observed win depends on an exactly known reflection symmetry and a deliberately constructed 15x sector-curvature mismatch.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/1217/bench_report.json)
- [official_stage2.py](https://synthcore.org/code/1217/official_stage2.py)
- [parity_experiment.py](https://synthcore.org/code/1217/parity_experiment.py)
- [report.md](https://synthcore.org/code/1217/report.md)
- [report_bench_2026-09-03T122941.md](https://synthcore.org/code/1217/report_bench_2026-09-03T122941.md)
- [stage2_bench.py](https://synthcore.org/code/1217/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1217)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.