# Fejer reflection accelerator for fixed-point layers

- ID: 1369
- Canonical URL: https://synthcore.org/idea/1369/fejer-reflection-accelerator-for-fixed-point-layers
- API JSON: https://synthcore.org/api/idea/1369.json
- API Markdown: https://synthcore.org/api/idea/1369.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.24643](https://arxiv.org/abs/2607.24643)
- Category: dynamics
- Solves: speedup, stability
- ML areas: optimizer, training-dynamics, ssm, diffusion-sampling
- Math tags: optimization, spectral-theory, dynamical-systems
- Ratings: usefulness 8/10; difficulty 4/10; novelty 6/10

## Idea description

Replace a slow sequence of resolvent or contractive fixed-point updates by a blockwise averaged-reflection extrapolation. The method computes reflected iterates R^j y_0, averages them with equal weights, and uses the result as the next macro-iterate. Unlike unconstrained Anderson acceleration, this construction has a uniform residual guarantee for every maximal monotone operator.

## Mathematical statement

Let M be a maximal monotone operator on a Hilbert space, J=(I+M)^(-1) its resolvent, R=2J-I the reflected resolvent, and r(y)=J(y)-y the fixed-point residual. The estimator is y_hat_K=(K+1)^(-1) sum from j=0 to K of R^j y_0. Because J=(I+R)/2, the residual telescopes as r(y_hat_K)=(R^(K+1)y_0-y_0)/(2(K+1)). If y_star is a fixed point of J and d_0=||y_0-y_star||, nonexpansiveness of R implies ||r(y_hat_K)|| <= d_0/(K+1). The paper proves that this rate is minimax-optimal over all adaptive affine combinations of K+1 resolvent outputs. The transferable property is therefore a cheap, stable polynomial accelerator for neural fixed-point solvers.

## Key formulas

- $$J=(I+M)^{-1},\qquad r(y)=J(y)-y,\qquad R=2J-I.$$
- $$\widehat y_K=\frac{1}{K+1}\sum_{j=0}^{K}R^j y_0.$$
- $$r(\widehat y_K)=\frac{R^{K+1}y_0-y_0}{2(K+1)},\qquad \|r(\widehat y_K)\|\leq\frac{d_0}{K+1}.$$
- $$\min_{P(1)=1}\max_{|u|=1}|(u-1)P(u)|=\frac{2}{K+1}.$$

## Implementation notes

Integrate this into a repeated fixed-point module: a DEQ solver, an implicit residual block, a proximal neural layer, a monotone learned denoiser, or an iterative diffusion or energy-minimization sampler. The existing update tensor J(y) is the only neural operation that must be modified. For each macro-step, initialize z=y and accumulator a=y. Repeat K times: compute z=2*J(z)-z and update a=a+z. Return y_new=a/(K+1). This directly implements z_j=R^j y and y_new=(K+1)^(-1) sum_j z_j. The exact theorem requires J to be a resolvent of a maximal monotone operator; for a generic neural block this assumption should be treated as a hypothesis, not as a guarantee. Measure the normalized residual rho(y)=||J(y)-y||/max(1,||y||) after every macro-step. Add a safeguard: accept y_new only if rho(y_new) <= rho(y); otherwise use one ordinary J update or damp the result with y_new=(1-eta)y+eta*y_new, where eta starts at 0.25 and is doubled after successful steps up to 1. The first experiment should use a 1M-parameter DEQ MLP on CIFAR-10 or MNIST, comparing K in {3,7,15} against plain fixed-point iteration and Anderson acceleration at equal numbers of J evaluations. Record residual versus evaluations, wall-clock time, backward memory, solver failures, and task loss. Success means lower residual or equal accuracy with fewer evaluations and no divergence spikes.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built the Fejer averaged-reflection solver and an optional normalized-residual safeguard. On exact linear resolvents induced by 2D rotations, the telescoping identity held with maximum absolute error 1.19e-16; the worst-case residual matched 1/(K+1) exactly for K=1,3,7,15,31, and the 90% transition matched the predicted theta=2 asin(0.9)/(K+1) within 2.4e-5 relative error. The low-frequency prediction residual approximately theta/2 matched within 4.2e-6 relative error, confirming the mechanism. At theta=0.35 and equal J evaluations, Fejer residual versus plain PPM was 0.1611 vs 0.1663 (K=3), 0.1232 vs 0.1563 (K=7), 0.02094 vs 0.1382 (K=15), and 0.01973 vs 0.1080 (K=31), so the toy effect is real under the resolvent assumptions.

### Mechanism check

- Verdict: Built the Fejer averaged-reflection solver and an optional normalized-residual safeguard. On exact linear resolvents induced by 2D rotations, the telescoping identity held with maximum absolute error 1.19e-16; the worst-case residual matched 1/(K+1) exactly for K=1,3,7,15,31, and the 90% transition matched the predicted theta=2 asin(0.9)/(K+1) within 2.4e-5 relative error. The low-frequency prediction residual approximately theta/2 matched within 4.2e-6 relative error, confirming the mechanism. At theta=0.35 and equal J evaluations, Fejer residual versus plain PPM was 0.1611 vs 0.1663 (K=3), 0.1232 vs 0.1563 (K=7), 0.02094 vs 0.1382 (K=15), and 0.01973 vs 0.1080 (K=31), so the toy effect is real under the resolvent assumptions.
- Confidence: 9/10
- Limitations: Only small deterministic linear rotation-resolvent and one nonlinear callable smoke test were evaluated. No CIFAR/MNIST, neural DEQ training, Anderson acceleration, wall-clock benchmark, GPU run, backward-memory measurement, or large-scale solver-failure study was performed; the exact theorem assumptions are not guaranteed for generic neural blocks.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.02285
- Paired wins: 1/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/362/bench_report.json)
- [fejer_experiment.py](https://synthcore.org/code/362/fejer_experiment.py)
- [fejer_solver.py](https://synthcore.org/code/362/fejer_solver.py)
- [report.md](https://synthcore.org/code/362/report.md)
- [report_bench_2026-08-31T092854.md](https://synthcore.org/code/362/report_bench_2026-08-31T092854.md)
- [results.json](https://synthcore.org/code/362/results.json)
- [stage2_fejer_bench.py](https://synthcore.org/code/362/stage2_fejer_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/362)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.