# Proximal Spherical Cubic Step

- ID: 1347
- Canonical URL: https://synthcore.org/idea/1347/proximal-spherical-cubic-step
- API JSON: https://synthcore.org/api/idea/1347.json
- API Markdown: https://synthcore.org/api/idea/1347.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.24046](https://arxiv.org/abs/2607.24046)
- Category: optimization
- Solves: speedup, stability, accuracy
- ML areas: optimizer, training-dynamics, mlp, fine-tuning
- Math tags: tensor-decomposition, optimization, numerical-analysis
- Ratings: usefulness 7/10; difficulty 7/10; novelty 7/10

## Idea description

Replace the inner step of a neural optimizer with a safeguarded cubic local-model solve. Represent the cubic Taylor model as a homogeneous tensor in an augmented coordinate, solve proximal unit-sphere subproblems by alternating tensor contractions, decode a candidate step, and accept it only when the actual neural loss confirms the predicted decrease.

## Mathematical statement

For parameters x and step s, the cubic Taylor model is T_3(s)=c+g^T s+(1/2)s^T Hs+(1/6)C[s,s,s], where c=f(x), g is the gradient, H is the Hessian, and C is the third-derivative tensor. With the augmented vector     ilde{s}=(1,s), an order-3 tensor A is defined so that T_3(s)=A[    ilde{s}]^3. For a trilinear representation F_h(u,v,w)=A[u,v,w] and unit vectors u,v,w, the proximal block problem is u^+=argmin_{||u||=1}[F_h(u,v,w)+(beta_1/2)||u-u_old||^2]. Because F_h is linear in u and both vectors have unit norm, the solution is u^+=normalize(beta_1 u_old-A(.,v,w)), with analogous updates for v and w. Alternating exact block minimization gives non-increasing proximal objectives and stationary accumulation points. A quartic safeguard can be added as m_3(s)=T_3(s)+(sigma/4)||s||^4.

## Key formulas

- $$T_3(s)=c+g^Ts+\frac{1}{2}s^THs+\frac{1}{6}C[s,s,s],\qquad T_3(s)={\cal A}[\tilde{s}]^3,\quad \tilde{s}=(1,s).$$
- $$u^{(\kappa+1)}=\arg\min_{\|u\|=1}\left[F_h(u,v^{(\kappa)},w^{(\kappa)})+\frac{\beta_1}{2}\|u-u^{(\kappa)}\|^2\right].$$
- $$u^{(\kappa+1)}=\frac{\beta_1u^{(\kappa)}-\mathcal{A}(\cdot,v^{(\kappa)},w^{(\kappa)})}{\left\|\beta_1u^{(\kappa)}-\mathcal{A}(\cdot,v^{(\kappa)},w^{(\kappa)})\right\|_2}.$$
- $$m_3(s)=T_3(s)+\frac{\sigma}{4}\|s\|^4,\qquad \rho=\frac{f(x)-f(x+s)}{T_3(0)-T_3(s)}.$$

## Implementation notes

Integrate this as an optional inner solver for a selected parameter block, such as a final MLP layer, LoRA adapter, or small projection matrix; do not initially apply it to an entire large Transformer. At optimizer step k, flatten the block into x, compute g, Hessian-vector products, and approximate third-order directional contractions, then construct a low-rank tensor-contraction oracle for A rather than materializing a dense tensor. Choose a radius Delta and represent candidate directions with three unit vectors u,v,w using the augmented vector     ilde{s}=(1,s). Initialize candidates from the negative-gradient direction, Adam's direction, and random unit vectors. For each PAM sweep, compute a_u=A(.,v,w) and update u <- normalize(beta_1 u-a_u); then compute a_v=A(u,.,w) and a_w=A(u,v,.) and apply the corresponding updates with beta_2 and beta_3. Decode the best candidate and scale it to ||s|| <= Delta. Evaluate rho=(f(x)-f(x+s))/(T_3(0)-T_3(s)); accept when rho >= eta, enlarge the radius for high rho, and shrink it after rejection. Estimate third derivatives using forward-over-reverse autodiff or finite differences of Hessian-vector products. First test a 2-layer MLP on CIFAR-10 against AdamW and a gradient-only cubic optimizer at matched gradient-equivalent FLOPs. Measure loss decrease per wall-clock second, final accuracy, gradient norm, rejected-step rate, and stability across seeds. Success is faster descent or fewer rejected steps on ill-conditioned blocks without loss spikes.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a readable proximal spherical cubic solver with exact homogeneous tensor construction, tensor contractions, PAM updates, quartic safeguard ratio testing, and deterministic toy verification. The mechanism manifested: representation error was 1.6e-14, maximum proximal block-objective increase was at numerical zero, and all quartic-strength acceptance patterns matched the analytically predicted radius transitions. The secondary comparison did not show a performance win: gradient descent reached -0.59694 versus -0.58612 for the safeguarded method, with 74/80 safeguarded attempts rejected.

### Mechanism check

- Verdict: Built a readable proximal spherical cubic solver with exact homogeneous tensor construction, tensor contractions, PAM updates, quartic safeguard ratio testing, and deterministic toy verification. The mechanism manifested: representation error was 1.6e-14, maximum proximal block-objective increase was at numerical zero, and all quartic-strength acceptance patterns matched the analytically predicted radius transitions. The secondary comparison did not show a performance win: gradient descent reached -0.59694 versus -0.58612 for the safeguarded method, with 74/80 safeguarded attempts rejected.
- Confidence: 8/10
- Limitations: No CIFAR-10, neural-network training, Hessian-vector or third-derivative autodiff, matched-FLOP timing, GPU test, multiple random seeds, or high-dimensional/low-rank tensor benchmark was performed. The implementation materializes a small dense tensor and uses a fixed toy radius rather than a full adaptive trust-region optimizer.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.6102
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/527/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/527/bench_report.json)
- [proximal_cubic.py](https://synthcore.org/code/527/proximal_cubic.py)
- [report.md](https://synthcore.org/code/527/report.md)
- [report_bench_2026-08-31T165804.md](https://synthcore.org/code/527/report_bench_2026-08-31T165804.md)
- [results.json](https://synthcore.org/code/527/results.json)
- [run_output.json](https://synthcore.org/code/527/run_output.json)
- [Download all files as ZIP](https://synthcore.org/download/527)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.