# Inexact High-Order Moreau DC Optimizer

- ID: 270
- Canonical URL: https://synthcore.org/idea/270/inexact-high-order-moreau-dc-optimizer
- API JSON: https://synthcore.org/api/idea/270.json
- API Markdown: https://synthcore.org/api/idea/270.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2606.30991](https://arxiv.org/abs/2606.30991)
- Category: optimization
- Solves: stability, accuracy, speedup
- ML areas: optimizer, training, regularization
- Math tags: convex-analysis, optimization, approximation-theory, dynamical-systems
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Represent a parameter objective locally as a difference of convex terms, compute approximate proximal points for both terms, and update parameters using the difference of their high-order Moreau-envelope gradients rather than the raw DC gradient. Start with the quadratic case p=2, then test p=4 as a sharper penalty for large proximal residuals; solve each proximal subproblem with a small fixed number of inner steps and decrease the smoothing scale during training.

## Mathematical statement

The paper studies a DC objective \(\varphi(x)=g(x)-h(x)\), where \(g,h:\mathbb{R}^n\to\overline{\mathbb{R}}\) are proper, lower-semicontinuous, convex functions and \(D=\operatorname{dom}(g)\cap\operatorname{dom}(h)\neq\varnothing\). For an implementable high-order Moreau construction, define \(q=p/(p-1)\) for \(p>1\), the generalized proximal point \(\operatorname{prox}^{p}_{\gamma f}(s)=\arg\min_x\{f(x)+\|x-s\|^p/(p\gamma)\}\), and the envelope \(E^p_{\gamma f}(s)=\min_x\{f(x)+\|x-s\|^p/(p\gamma)\}\). If \(u=\operatorname{prox}^{p}_{\gamma g}(s)\) and \(v=\operatorname{prox}^{p}_{\gamma h}(s)\) are unique, their envelope gradients are \(\nabla E^p_{\gamma f}(s)=\gamma^{-1}\|s-\operatorname{prox}^{p}_{\gamma f}(s)\|^{p-2}(s-\operatorname{prox}^{p}_{\gamma f}(s))\). The adapted HOME-DC gradient is therefore \(G(s)=\gamma^{-1}[\|s-u\|^{p-2}(s-u)-\|s-v\|^{p-2}(s-v)]\), and the descent update is \(s_{t+1}=s_t-\eta_tG(s_t)\). The paper's lifted critical-set example uses \(J_q(\xi)=\|\xi\|^{q-2}\xi\) and gives \(\mathcal{L}_{\gamma}^{p}(0)=\{\gamma^{q-1}J_q(\xi):\xi\in[-1,1]\}=[-\gamma^{q-1},\gamma^{q-1}]\) for \(g(x)=|x|,h(x)=2|x|\); thus smoothing can introduce nonisolated critical points. For inexact proximal points \(\tilde u,\tilde v\), use residuals \(r_g=\tilde u-\operatorname{prox}^{p}_{\gamma g}(s)\) and \(r_h=\tilde v-\operatorname{prox}^{p}_{\gamma h}(s)\) conceptually, and terminate inner solves when the computable first-order residuals \(\|\nabla f(z)+\gamma^{-1}\|z-s\|^{p-2}(z-s)\|\leq\varepsilon_t\) are small. The resulting gradient error is controlled by the proximal errors, so set \(\varepsilon_t\) proportional to the outer gradient norm or decrease it with \(t\).

## Key formulas

- $$\varphi(x)=g(x)-h(x),\qquad g,h:\mathbb{R}^n\to\overline{\mathbb{R}}\text{ proper, lower-semicontinuous, convex},$$
- $$\operatorname{prox}^{p}_{\gamma f}(s)=\arg\min_x\left\{f(x)+\frac{\|x-s\|^p}{p\gamma}\right\},\qquad \nabla E^p_{\gamma f}(s)=\frac{1}{\gamma}\|s-\operatorname{prox}^{p}_{\gamma f}(s)\|^{p-2}\bigl(s-\operatorname{prox}^{p}_{\gamma f}(s)\bigr),$$
- $$G(s)=\frac{1}{\gamma}\left[\|s-u\|^{p-2}(s-u)-\|s-v\|^{p-2}(s-v)\right],\quad u=\operatorname{prox}^{p}_{\gamma g}(s),\ v=\operatorname{prox}^{p}_{\gamma h}(s),\quad s_{t+1}=s_t-\eta_tG(s_t),$$
- $$\mathcal{L}_{\gamma}^{p}(0)=\left\{\gamma^{q-1}J_q(\xi):\xi\in[-1,1]\right\}=[-\gamma^{q-1},\gamma^{q-1}],\qquad q=\frac{p}{p-1},\quad J_q(\xi)=|\xi|^{q-2}\xi.$$

## Implementation notes

1. Integration point: replace the AdamW update for a selected parameter block, preferably LoRA/adapters or the final classifier, by an outer HOME-DC update. Choose a concrete DC objective \(F(\theta)=g(\theta)-h(\theta)\): for an initial experiment let \(g(\theta)=\ell(\theta)+\lambda\|\theta\|_1+\frac{L}{2}\|\theta\|^2\) and \(h(\theta)=\frac{L}{2}\|\theta\|^2\), where \(L\) is a conservative curvature estimate; alternatively use a standard DC sparsity penalty such as capped-L1 written as a convex term minus a convex hinge term. 2. Pseudocode: set \(s=\theta\), smoothing \(\gamma\), order \(p\in\{2,4\}\). At every outer step, define \(u\leftarrow s\), \(v\leftarrow s\); run K=3--10 inner gradient or proximal-gradient steps on \(g(x)+\|x-s\|^p/(p\gamma)\) to obtain \(u\), and on \(h(x)+\|x-s\|^p/(p\gamma)\) to obtain \(v\). Compute \(a_g=\gamma^{-1}\|s-u\|^{p-2}(s-u)\), \(a_h=\gamma^{-1}\|s-v\|^{p-2}(s-v)\), then \(G=a_g-a_h\) and \(s\leftarrow s-\eta G\). Copy \(\theta\leftarrow s\) before the next minibatch. 3. Mathematics versus estimation: the proximal-gradient residual formula is exact for the chosen surrogate; the proximal points are estimated by the K inner iterations, \(L\) is estimated from a few Hessian-vector power iterations or a backtracking test, and \(\gamma\) is a hyperparameter. Use residual-based stopping if extra inner work is allowed: stop when both \(\|\nabla g(u)+a_g\|\) and \(\|\nabla h(v)+a_h\|\) fall below \(\varepsilon_t\). Use gradient clipping only after forming \(G\), not separately on \(a_g\) and \(a_h\). 4. First experiment: train a small two-layer MLP and a 2-layer ViT on CIFAR-10 with a nonconvex capped-L1 or group-sparsity regularizer, comparing SGD, AdamW, ordinary DCA, and the proposed p=2 and p=4 optimizers at equal outer minibatch FLOPs. Sweep \(\gamma\in\{1,0.3,0.1\}\) and geometrically decrease it after validation loss plateaus. Measure training-loss variance, gradient-norm spikes, final accuracy, achieved parameter sparsity, and wall-clock cost. Success means fewer divergence events and a smoother loss curve, or at least 0.5--1 percentage-point better accuracy at the same sparsity and total FLOPs. Also test continuation versus fixed \(\gamma\), since the interval-valued critical-set example predicts that fixed heavy smoothing may converge to a surrogate artifact rather than an original DC stationary point.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built an inexact high-order Moreau DC optimizer for a capped-L1-like toy DC objective, including p=2 and p=4 proximal updates with fixed inner iterations and decreasing smoothing. The envelope-gradient identity was numerically verified for both orders, with finite-difference error around 3e-7 and proximal residuals below 8e-6. In the fixed-seed comparison, p=4 reached a lower final objective than raw DC descent (1.8981 vs 1.9394) and had much lower RMS update norm (0.2005 vs 1.0892); p=2 was smoother but ended worse, so the signal is promising but not uniform.

### Mechanism check

- Verdict: Built an inexact high-order Moreau DC optimizer for a capped-L1-like toy DC objective, including p=2 and p=4 proximal updates with fixed inner iterations and decreasing smoothing. The envelope-gradient identity was numerically verified for both orders, with finite-difference error around 3e-7 and proximal residuals below 8e-6. In the fixed-seed comparison, p=4 reached a lower final objective than raw DC descent (1.8981 vs 1.9394) and had much lower RMS update norm (0.2005 vs 1.0892); p=2 was smoother but ended worse, so the signal is promising but not uniform.
- Confidence: 7/10
- Limitations: Only a 24-dimensional synthetic capped-L1-like DC problem was tested. No CIFAR/MLP experiment, wall-clock or FLOP accounting, hyperparameter sweep, multiple-seed statistical test, or residual-based adaptive inner termination was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/1247/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/1247/bench_report.json)
- [experiment.py](https://synthcore.org/code/1247/experiment.py)
- [report.md](https://synthcore.org/code/1247/report.md)
- [report_bench_2026-09-03T222105.md](https://synthcore.org/code/1247/report_bench_2026-09-03T222105.md)
- [results.json](https://synthcore.org/code/1247/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1247)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.