✗ Failed on benchmark 2026

Fejer reflection accelerator for fixed-point layers

Implementation & benchmark of arXiv:2607.24643 — Anderson acceleration of the proximal point method: the exact adaptive minimax, a spectral phase transition, and optimal safeguarding

Usefulness8/10

Difficulty4/10

Novelty6/10

Dynamics Speedup Stability Optimizer Training dynamics Ssm Diffusion sampling Optimization Spectral theory Dynamical systems

Source paper: Anderson acceleration of the proximal point method: the exact adaptive minimax, a spectral phase transition, and optimal safeguarding arXiv:2607.24643 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a slow sequence of resolvent or contractive fixed-point updates by a blockwise averaged-reflection extrapolation. The method computes reflected iterates R^j y_0, averages them with equal weights, and uses the result as the next macro-iterate. Unlike unconstrained Anderson acceleration, this construction has a uniform residual guarantee for every maximal monotone operator.

Formulas

$$J=(I+M)^{-1},\qquad r(y)=J(y)-y,\qquad R=2J-I.$$

$$\widehat y_K=\frac{1}{K+1}\sum_{j=0}^{K}R^j y_0.$$

$$r(\widehat y_K)=\frac{R^{K+1}y_0-y_0}{2(K+1)},\qquad \|r(\widehat y_K)\|\leq\frac{d_0}{K+1}.$$

$$\min_{P(1)=1}\max_{|u|=1}|(u-1)P(u)|=\frac{2}{K+1}.$$

Mathematical statement

Let M be a maximal monotone operator on a Hilbert space, J=(I+M)^(-1) its resolvent, R=2J-I the reflected resolvent, and r(y)=J(y)-y the fixed-point residual. The estimator is y_hat_K=(K+1)^(-1) sum from j=0 to K of R^j y_0. Because J=(I+R)/2, the residual telescopes as r(y_hat_K)=(R^(K+1)y_0-y_0)/(2(K+1)). If y_star is a fixed point of J and d_0=||y_0-y_star||, nonexpansiveness of R implies ||r(y_hat_K)|| <= d_0/(K+1). The paper proves that this rate is minimax-optimal over all adaptive affine combinations of K+1 resolvent outputs. The transferable property is therefore a cheap, stable polynomial accelerator for neural fixed-point solvers.

Implementation notes

Integrate this into a repeated fixed-point module: a DEQ solver, an implicit residual block, a proximal neural layer, a monotone learned denoiser, or an iterative diffusion or energy-minimization sampler. The existing update tensor J(y) is the only neural operation that must be modified. For each macro-step, initialize z=y and accumulator a=y. Repeat K times: compute z=2*J(z)-z and update a=a+z. Return y_new=a/(K+1). This directly implements z_j=R^j y and y_new=(K+1)^(-1) sum_j z_j. The exact theorem requires J to be a resolvent of a maximal monotone operator; for a generic neural block this assumption should be treated as a hypothesis, not as a guarantee. Measure the normalized residual rho(y)=||J(y)-y||/max(1,||y||) after every macro-step. Add a safeguard: accept y_new only if rho(y_new) <= rho(y); otherwise use one ordinary J update or damp the result with y_new=(1-eta)y+eta*y_new, where eta starts at 0.25 and is doubled after successful steps up to 1. The first experiment should use a 1M-parameter DEQ MLP on CIFAR-10 or MNIST, comparing K in {3,7,15} against plain fixed-point iteration and Anderson acceleration at equal numbers of J evaluations. Record residual versus evaluations, wall-clock time, backward memory, solver failures, and task loss. Success means lower residual or equal accuracy with fewer evaluations and no divergence spikes.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built the Fejer averaged-reflection solver and an optional normalized-residual safeguard. On exact linear resolvents induced by 2D rotations, the telescoping identity held with maximum absolute error 1.19e-16; the worst-case residual matched 1/(K+1) exactly for K=1,3,7,15,31, and the 90% transition matched the predicted theta=2 asin(0.9)/(K+1) within 2.4e-5 relative error. The low-frequency prediction residual approximately theta/2 matched within 4.2e-6 relative error, confirming the mechanism. At theta=0.35 and equal J evaluations, Fejer residual versus plain PPM was 0.1611 vs 0.1663 (K=3), 0.1232 vs 0.1563 (K=7), 0.02094 vs 0.1382 (K=15), and 0.01973 vs 0.1080 (K=31), so the toy effect is real under the resolvent assumptions.

Agent confidence: 9/10

Baseline: Plain proximal/fixed-point iteration residual after K evaluations at rotation theta=0.35: 0.16625, 0.15633, 0.13821, 0.10804 for K=3,7,15,31.
Idea: Fejer residual after equal K evaluations: 0.16105, 0.12318, 0.02094, 0.01973 for K=3,7,15,31; ratios to baseline 0.969, 0.788, 0.151, 0.183. Identity error 1.19e-16.

Limitations: Only small deterministic linear rotation-resolvent and one nonlinear callable smoke test were evaluated. No CIFAR/MNIST, neural DEQ training, Anderson acceleration, wall-clock benchmark, GPU run, backward-memory measurement, or large-scale solver-failure study was performed; the exact theorem assumptions are not guaranteed for generic neural blocks.

How to run: python3 fejer_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.24643) Sep 2, 2026

Significantly WORSE than the tuned baseline (+94.3%, p=0.0228, wins 1 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0012
Idea mean: 0.0023
Effect (Δ): 0.0011 (+94.3%; negative = idea better)
Wins: 1 / 8 paired seeds
p-value: 0.0228 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±125.9%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 9 configs

Limitations:

Only the registered dynamics track was tested, with 400 training examples, 200 test examples, 16 epochs, and the shared 3x3 learning-rate/K grid. No vision, sequence, tabular, Anderson, wall-clock, backward-memory, or larger-scale evaluation was run.

How to run: python3 stage2_fejer_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

The Fejer reflection recurrent system was tested on the registered dynamics track with a tuned baseline and 8 paired seeds. The baseline MSE was 0.0011932104 versus 0.0023188923 for the best idea configuration; delta_mean was +0.0011256819 with permutation p=0.02285, so the idea was significantly worse. The trained-model mechanism signature was also not confirmed: predicted residual ratio 0.125, observed 1.1270424, and identity error 13.3236.

Baseline: lr=0.005, K=15; mean test MSE 0.0011932104389416054
Delta mean: 0.0011256818834226578
Idea: lr=0.005, K=3; mean test MSE 0.0023188923223642632
P value: 0.02285

Benchmark result