Cap-free golden-ratio primal-dual optimizer
Implementation & benchmark of arXiv:2607.08174 — Two Adaptive Accelerated Golden Ratio Primal--Dual Algorithms With an Application to Poisson Imaging Problem
Source paper: Two Adaptive Accelerated Golden Ratio Primal--Dual Algorithms With an Application to Poisson Imaging Problem arXiv:2607.08174 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace a standard proximal-gradient or Adam-style update for a composite neural-network objective with a golden-ratio primal-dual iteration. The optimizer separates a nonsmooth regularizer from a locally smooth loss, estimates local curvature from successive gradients, and uses dual variables for explicit constraints instead of forcing all structure into penalty coefficients. The experiment is falsifiable: at equal gradient evaluations, the method should tolerate larger initial steps and show fewer divergence events than fixed-step proximal gradient or AdamW on objectives with sharp and flat regions.
Formulas
Mathematical statement
The paper studies $\min_x f(x)+g(Kx)+h(x)$, where $f$ and $g$ are proper convex lower-semicontinuous functions, $K$ is linear, and $h$ is convex differentiable. Its local-smoothness assumption is $\|\nabla h(x)-\nabla h(y)\|\le L_B\|x-y\|$ for all $x,y$ in every compact set $B$, without requiring one global known Lipschitz constant. Use $\varphi=(1+\sqrt5)/2$ and the golden-ratio extrapolation $z_{n+1}=\frac{\varphi-1}{\varphi}x_{n+1}+\frac1\varphi z_n$. The implementable primal-dual splitting is $x_{n+1}=\operatorname{prox}_{\lambda_n f}(z_n-\lambda_n(\nabla h(x_n)+K^*y_n))$ and $y_{n+1}=\operatorname{prox}_{\sigma_n g^*}(y_n+\sigma_nKz_{n+1})$, where $g^*$ is the convex conjugate, $K^*$ is the adjoint, and $\lambda_n,\sigma_n>0$ are primal and dual step sizes. Estimate local smoothness by $\widehat L_n=\|\nabla h(x_n)-\nabla h(x_{n-1})\|/(\|x_n-x_{n-1}\|+\epsilon)$ and choose $\lambda_n=\eta/(\widehat L_n+\|K\|^2\sigma_n+\epsilon)$ with $0<\eta<1$; estimate $\|K\|$ by a few power iterations. This is an engineering adaptation of the paper's cap-free adaptive principle; the paper's convex guarantees apply directly only when the neural subproblem is convex.
Implementation notes
(1) Integration point: implement this as an optimizer for one parameter block $x$ of a neural network, not as a replacement for the forward pass. Start with a fixed-feature classifier or LoRA block. Let $h(x)$ be the minibatch cross-entropy or squared loss, let $f(x)=\alpha\|x\|_1+\frac{\rho}{2}\|x\|_2^2$ so its proximal map is soft-thresholding followed by shrinkage, and let $Kx$ expose a constrained quantity such as per-layer group norms or a residual parameter vector. For a simple inequality constraint $\|Kx\|_\infty\le c$, use $g$ as the indicator of the box and $\operatorname{prox}_{\sigma g^*}(y)=y-\sigma\,\operatorname{clip}(y/\sigma,-c,c)$ via Moreau's identity. (2) Pseudocode: initialize $x,z=x$, $y=0$, $\lambda=\lambda_0$, and $\sigma=\sigma_0$. At each minibatch, compute $q=\nabla h(x)$; if this is not the first step, set $\widehat L=\|q-q_{old}\|/(\|x-x_{old}\|+10^{-8})$; estimate $\widehat{\|K\|^2}$ once every 100 steps by two power iterations; set $\lambda=\eta/(\widehat L+\sigma\widehat{\|K\|^2}+10^{-8})$; compute $x_{new}=\operatorname{prox}_{\lambda f}(z-\lambda(q+K^Ty))$; compute $z_{new}=(\varphi-1)x_{new}/\varphi+z/\varphi$; compute $y_{new}=\operatorname{prox}_{\sigma g^*}(y+\sigma Kz_{new})$; then shift old variables. (3) Compute $q$ with autograd, $K$ and $K^T$ as explicit tensor operators, and all proximal maps analytically where possible. The local curvature estimate and operator norm are empirical; log them along with the step size, gradient norm, constraint violation $\max(0,\|Kx\|_\infty-c)$, and objective. Smooth $\widehat L$ with an EMA only for minibatch-noise robustness, not as a hidden manual step-size cap. (4) First experiment: train a logistic-regression head and a rank-8 LoRA adapter on CIFAR-10 or a small language-model classification task using frozen features. Compare this optimizer against proximal gradient with tuned constant step size, AdamW, and an untuned conservative step. Use identical minibatch order and gradient-evaluation budgets. Success means faster loss decrease at equal FLOPs, fewer NaN or divergence runs across learning-rate initializations, and lower final constraint violation or parameter norm at matched validation accuracy. Then test the update on a small MLP, explicitly reporting that the convex convergence theorem no longer applies.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 8/10
Built a cap-free golden-ratio primal-dual optimizer with elastic-net proximal regularization, box-constraint dual updates via Moreau's identity, local curvature adaptation, and controls for proximal gradient, projected proximal gradient, and AdamW. The math sanity check passed: the finite-difference curvature estimate matched the analytic Hessian action with 0.10% relative error, and the golden-ratio coefficients summed to one. GRPD maintained the explicit constraint at all tested initial steps and reached loss 0.07735 versus 0.10555 for projected PG at step 0.02, but the claimed fewer-divergence/larger-step stability win was not demonstrated because every method remained finite across the tested sweep; therefore worked=false.
- Agent confidence
- 8/10
- Baseline
- Projected PG final losses: 0.10555, 0.08904, 0.08438, 0.07971, 0.07702, 0.07685, 0.07685 for initial steps 0.02, 0.05, 0.1, 0.2, 0.5, 1.0, 2.0; constraint violation was zero throughout.
- Idea
- GRPD final losses: 0.07735, 0.07735, 0.07732, 0.07729, 0.07725, 0.07730, 0.07752 at the same steps; violations were zero except approximately 0.00035 at steps 0.5 and 1.0. Curvature relative error was 0.000975.
Limitations: Only a small full-batch convex logistic problem and one fixed random seed were tested. No neural network, minibatch noise, CIFAR/features, multiple independent runs, GPU execution, runtime/FLOP comparison, or genuinely divergent regime was evaluated; the convex paper guarantee was not tested beyond this toy setting.
How to run: python3 experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- experiment.py 5.0 KB View Raw
- report.md 1.8 KB View
- results.json 6.3 KB View Raw