△ Mechanism confirmed, baseline not beaten 2026

Cayley Midpoint Optimizer for Adversarial Heads

Implementation & benchmark of arXiv:2607.09950 — Implicit Midpoint Gradient Descent: Fast and Learning rate free convergence for Zero-Sum Games

Usefulness8/10

Difficulty6/10

Novelty6/10

Optimization Stability Accuracy Speedup Optimizer Gan Rl Training dynamics Dynamical systems Linear algebra Spectral theory Optimization

Source paper: Implicit Midpoint Gradient Descent: Fast and Learning rate free convergence for Zero-Sum Games arXiv:2607.09950 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace simultaneous descent-ascent on a bilinear adversarial subproblem by an implicit midpoint step. The update is a Cayley transform of the skew-symmetric game Jacobian, so it rotates rather than amplifies oscillatory modes and remains bounded for arbitrarily large positive step sizes in the exact bilinear case.

Formulas

$$x^{t+1}=x^t-\eta F\left(\frac{x^t+x^{t+1}}{2}\right),\qquad F(x)=Gx+c,\qquad G^\top=-G.$$

$$\left(I+\frac{\eta}{2}G\right)x^{t+1}=\left(I-\frac{\eta}{2}G\right)x^t-\eta c.$$

$$z^{t+1}=R_\eta z^t,\qquad R_\eta=\left(I+\frac{\eta}{2}G\right)^{-1}\left(I-\frac{\eta}{2}G\right),\qquad R_\eta^\top R_\eta=I.$$

$$J(\omega)=\begin{pmatrix}0&\omega\\-\omega&0\end{pmatrix},\qquad \theta=2\arctan\left(\frac{\eta\omega}{2}\right).$$

Mathematical statement

For a zero-sum bilinear utility, define the joint variable x = (x_1, x_2), the affine game operator F(x) = Gx + c, and the skew-symmetric matrix G satisfying G^T = -G. The implicit midpoint rule is x^{t+1} = x^t - eta F((x^t + x^{t+1})/2), where eta > 0 is the step size and c contains the linear terms. Rearranging gives a linear system for x^{t+1}. Around an equilibrium x^star, the shifted error z^t = x^t - x^star obeys z^{t+1} = R_eta z^t, with R_eta = (I + eta G/2)^{-1}(I - eta G/2). Since G is skew-symmetric, R_eta is orthogonal, so every nonzero mode preserves its Euclidean norm instead of exhibiting Euler's artificial growth. In a real Schur block J(omega), where omega is the magnitude of the imaginary eigenvalue, the update is a planar rotation with angle theta = 2 arctan(eta omega / 2).

Implementation notes

1. Integration point: apply the method to a coupled generator-discriminator, actor-critic, or adversarial-head parameter vector whose local interaction is approximately bilinear. Begin with two small linear heads w_1 and w_2 on top of frozen neural feature extractors, and concatenate them into x = [w_1; w_2]. Estimate the cross-Jacobian block A from minibatch gradients and construct G = [[0, A], [-A^T, 0]]. 2. Pseudocode: compute the minibatch affine operator F_t(x_t); refresh G_t; solve (I + eta G_t / 2) x_next = (I - eta G_t / 2) x_t - eta c_t; then assign x_t = x_next. For a matrix-free implementation, use fixed-point iteration x^(r+1) = x_t - eta F_t((x_t + x^(r))/2), initialized with x^(0) = x_t, and stop when the linear-system residual is small. 3. The paper supplies the Cayley transform and exact orthogonality result. Estimate A, G_t, and the solve residual using automatic Jacobian-vector products or finite differences; do not assume the guarantee remains exact for a changing nonlinear Jacobian. 4. First experiment: train a small GAN on MNIST or CIFAR-10 and compare simultaneous gradient descent/ascent, extragradient, and this optimizer using identical minibatches and matched wall-clock budgets. Sweep eta over 0.1, 1, 10, and 100. Track parameter norms, gradient norms, loss oscillation, FID, mode coverage, and game residual. Success means no orbit explosion at large eta and better FID or residual at equal compute.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 10/10

Built and ran a deterministic Cayley midpoint versus simultaneous Euler descent-ascent experiment for skew-symmetric bilinear games. All three predictions were confirmed: Cayley norm preservation, exact Euler amplification, and the predicted rotation angle; maximum errors were approximately 2.2e-16, 5.7e-14, and 5.9e-14 respectively. At eta=10 for 100 steps, Cayley retained norm 1.0770329614 while Euler grew to 1.77e100, demonstrating the claimed stability mechanism. This is a real win for exact bilinear dynamics, but it does not establish improved performance for nonlinear GAN or adversarial-head training.

Agent confidence: 10/10

Baseline: Simultaneous Euler descent-ascent: one-step norm factor exactly sqrt(1+(eta*omega)^2); at omega=1, eta=10, factor=10.0498756 and 100-step norm=1.7713e100.
Idea: Cayley midpoint: norm factor=1.0 and orthogonality error <=3.51e-16 across eta in {0.01,0.1,1,10,100} and omega in {0.1,1,3}; at omega=1, eta=10, 100-step norm=1.0770329614 versus predicted 1.0770329614. Rotation-angle maximum absolute error=5.87e-14; affine equilibrium identity error=1.11e-16.

Limitations: Only exact low-dimensional bilinear games were tested. No nonlinear GAN, neural adversarial heads, changing Jacobian estimation, wall-clock benchmark, FID, mode coverage, or comparison with extragradient was performed.

How to run: python3 cayley_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.09950) Sep 2, 2026

Ahead of the baseline (−1.4%) but not statistically significant (p=0.904, wins 5 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.001
Idea mean: 0.001
Effect (Δ): -0 (−1.4%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.904 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 6 configs

Limitations:

Only the built-in dynamics track was tested with rnn_small, 400 training examples, 200 test examples, 8 epochs, and 8 paired seeds. GANs, larger adversarial heads, extragradient, wall-clock speed, FID, and mode coverage were not tested.

How to run: python3 cayley_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 9/10

The Cayley midpoint optimizer was implemented on the structurally matching dynamics track with the shared rnn_small architecture. Its trained-model signature confirmed norm preservation, but the task-MSE improvement was not significant (paired delta -1.3968e-05, permutation p=0.90395), so no neural-training win was demonstrated.

Baseline: Adam best lr=0.01, weight_decay=0.0; 8-seed test MSE mean 0.0009744616800162476.
Idea: Cayley midpoint lr=0.01, weight_decay=0.0; 8-seed test MSE mean 0.0009604936167306732; paired delta -1.3968063285574317e-05, p=0.90395, 5/8 wins.

Benchmark result