# Cayley Midpoint Optimizer for Adversarial Heads

- ID: 660
- Canonical URL: https://synthcore.org/idea/660/cayley-midpoint-optimizer-for-adversarial-heads
- API JSON: https://synthcore.org/api/idea/660.json
- API Markdown: https://synthcore.org/api/idea/660.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.09950](https://arxiv.org/abs/2607.09950)
- Category: optimization
- Solves: stability, accuracy, speedup
- ML areas: optimizer, gan, rl, training-dynamics
- Math tags: dynamical-systems, linear-algebra, spectral-theory, optimization
- Ratings: usefulness 8/10; difficulty 6/10; novelty 6/10

## Idea description

Replace simultaneous descent-ascent on a bilinear adversarial subproblem by an implicit midpoint step. The update is a Cayley transform of the skew-symmetric game Jacobian, so it rotates rather than amplifies oscillatory modes and remains bounded for arbitrarily large positive step sizes in the exact bilinear case.

## Mathematical statement

For a zero-sum bilinear utility, define the joint variable x = (x_1, x_2), the affine game operator F(x) = Gx + c, and the skew-symmetric matrix G satisfying G^T = -G. The implicit midpoint rule is x^{t+1} = x^t - eta F((x^t + x^{t+1})/2), where eta > 0 is the step size and c contains the linear terms. Rearranging gives a linear system for x^{t+1}. Around an equilibrium x^star, the shifted error z^t = x^t - x^star obeys z^{t+1} = R_eta z^t, with R_eta = (I + eta G/2)^{-1}(I - eta G/2). Since G is skew-symmetric, R_eta is orthogonal, so every nonzero mode preserves its Euclidean norm instead of exhibiting Euler's artificial growth. In a real Schur block J(omega), where omega is the magnitude of the imaginary eigenvalue, the update is a planar rotation with angle theta = 2 arctan(eta omega / 2).

## Key formulas

- $$x^{t+1}=x^t-\eta F\left(\frac{x^t+x^{t+1}}{2}\right),\qquad F(x)=Gx+c,\qquad G^\top=-G.$$
- $$\left(I+\frac{\eta}{2}G\right)x^{t+1}=\left(I-\frac{\eta}{2}G\right)x^t-\eta c.$$
- $$z^{t+1}=R_\eta z^t,\qquad R_\eta=\left(I+\frac{\eta}{2}G\right)^{-1}\left(I-\frac{\eta}{2}G\right),\qquad R_\eta^\top R_\eta=I.$$
- $$J(\omega)=\begin{pmatrix}0&\omega\\-\omega&0\end{pmatrix},\qquad \theta=2\arctan\left(\frac{\eta\omega}{2}\right).$$

## Implementation notes

1. Integration point: apply the method to a coupled generator-discriminator, actor-critic, or adversarial-head parameter vector whose local interaction is approximately bilinear. Begin with two small linear heads w_1 and w_2 on top of frozen neural feature extractors, and concatenate them into x = [w_1; w_2]. Estimate the cross-Jacobian block A from minibatch gradients and construct G = [[0, A], [-A^T, 0]]. 2. Pseudocode: compute the minibatch affine operator F_t(x_t); refresh G_t; solve (I + eta G_t / 2) x_next = (I - eta G_t / 2) x_t - eta c_t; then assign x_t = x_next. For a matrix-free implementation, use fixed-point iteration x^(r+1) = x_t - eta F_t((x_t + x^(r))/2), initialized with x^(0) = x_t, and stop when the linear-system residual is small. 3. The paper supplies the Cayley transform and exact orthogonality result. Estimate A, G_t, and the solve residual using automatic Jacobian-vector products or finite differences; do not assume the guarantee remains exact for a changing nonlinear Jacobian. 4. First experiment: train a small GAN on MNIST or CIFAR-10 and compare simultaneous gradient descent/ascent, extragradient, and this optimizer using identical minibatches and matched wall-clock budgets. Sweep eta over 0.1, 1, 10, and 100. Track parameter norms, gradient norms, loss oscillation, FID, mode coverage, and game residual. Success means no orbit explosion at large eta and better FID or residual at equal compute.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built and ran a deterministic Cayley midpoint versus simultaneous Euler descent-ascent experiment for skew-symmetric bilinear games. All three predictions were confirmed: Cayley norm preservation, exact Euler amplification, and the predicted rotation angle; maximum errors were approximately 2.2e-16, 5.7e-14, and 5.9e-14 respectively. At eta=10 for 100 steps, Cayley retained norm 1.0770329614 while Euler grew to 1.77e100, demonstrating the claimed stability mechanism. This is a real win for exact bilinear dynamics, but it does not establish improved performance for nonlinear GAN or adversarial-head training.

### Mechanism check

- Verdict: Built and ran a deterministic Cayley midpoint versus simultaneous Euler descent-ascent experiment for skew-symmetric bilinear games. All three predictions were confirmed: Cayley norm preservation, exact Euler amplification, and the predicted rotation angle; maximum errors were approximately 2.2e-16, 5.7e-14, and 5.9e-14 respectively. At eta=10 for 100 steps, Cayley retained norm 1.0770329614 while Euler grew to 1.77e100, demonstrating the claimed stability mechanism. This is a real win for exact bilinear dynamics, but it does not establish improved performance for nonlinear GAN or adversarial-head training.
- Confidence: 10/10
- Limitations: Only exact low-dimensional bilinear games were tested. No nonlinear GAN, neural adversarial heads, changing Jacobian estimation, wall-clock benchmark, FID, mode coverage, or comparison with extragradient was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.90395
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/124/bench_report.json)
- [cayley_bench.py](https://synthcore.org/code/124/cayley_bench.py)
- [cayley_experiment.py](https://synthcore.org/code/124/cayley_experiment.py)
- [report.md](https://synthcore.org/code/124/report.md)
- [report_bench_2026-08-30T151855.md](https://synthcore.org/code/124/report_bench_2026-08-30T151855.md)
- [results.json](https://synthcore.org/code/124/results.json)
- [Download all files as ZIP](https://synthcore.org/download/124)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.