# Removable-Pole Negative-Shifted Optimizer

- ID: 1285
- Canonical URL: https://synthcore.org/idea/1285/removable-pole-negative-shifted-optimizer
- API JSON: https://synthcore.org/api/idea/1285.json
- API Markdown: https://synthcore.org/api/idea/1285.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.22474](https://arxiv.org/abs/2607.22474)
- Category: optimization
- Solves: accuracy, sample-efficiency, speedup
- ML areas: optimizer, fine-tuning, training-dynamics
- Math tags: spectral-theory, dynamical-systems, optimization, linear-algebra
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace ordinary gradient descent in a chosen approximately linear parameter block with gradient descent plus a controlled negative quadratic penalty, and stop before the unstable directions explode. The finite-time spectral filter can amplify well-supported directions while retaining shrinkage or limited exposure on weak directions, which is unavailable to a stable negative-ridge endpoint.

## Mathematical statement

For squared loss with design matrix X, sample count n, labels y, parameters beta, empirical covariance S=\widehat{\Sigma}=X^    op X/n, and gradient vector b=X^    op y/n, the paper's negative-shifted gradient flow is \dot{\beta}_t=b-(S-
u I)\beta_t with initialization \beta_0=0 and shift 
u\ge 0. If S has eigenpair (u,s), then the coefficient along u is multiplied by g_{
u,t}(s)=(1-e^{-(s-
u)t})/(s-
u), with removable value g_{
u,t}(
u)=t; the corresponding prediction filter is f_{
u,t}(s)=s g_{
u,t}(s), whose removable value is f_{
u,t}(
u)=
u t. Thus the apparent pole of the endpoint is harmless at finite time, while the sign of f_{
u,t}(s)-1 can change across the spectrum, allowing a high-eigenvalue prefix to be anti-shrunk and weaker directions to remain shrunk. The discrete neural update is an Euler discretization: beta_{k+1}=beta_k-eta(
abla L(beta_k)-
u beta_k), where eta is the step size and K is the stopping iteration. Stability is intentionally supplied by finite K, validation selection, gradient clipping, and a cap on eta(nu-s_max), rather than by requiring S-nu I to be positive semidefinite.

## Key formulas

- $$\dot{\beta}_{t}=n^{-1}X^{\top}(y-X\beta_{t})+\nu\beta_{t}=b-(\widehat{\Sigma}-\nu I)\beta_{t},\qquad\beta_{0}=0.$$
- $$g_{\nu,t}(s)=\frac{1-e^{-(s-\nu)t}}{s-\nu},\qquad g_{\nu,t}(\nu)=t,\qquad f_{\nu,t}(s)=s\,g_{\nu,t}(s),\qquad f_{\nu,t}(\nu)=\nu t.$$
- $$\beta_{k+1}=\beta_k-\eta\left(\nabla_{\beta}L(\beta_k)-\nu\beta_k\right)=(1+\eta\nu)\beta_k-\eta\nabla_{\beta}L(\beta_k).$$
- $$\text{For }S=U\operatorname{diag}(s_i)U^{\top},\quad \beta_t=U\operatorname{diag}\!\left(g_{\nu,t}(s_i)\right)U^{\top}b;\quad f_{\nu,t}(s_i)=s_i g_{\nu,t}(s_i).$$

## Implementation notes

1. Integration point: use this optimizer only on an approximately linear parameter block, initially a frozen-backbone linear classifier, a LoRA matrix, or the final transformer block; leave the rest of the network on AdamW. Center the block at its pretrained value beta0 and apply the shift to the displacement delta=beta-beta0, so the update does not encourage raw weights to grow away from a useful initialization. For squared or locally quadratic loss, set delta <- (1+eta*nu)*delta - eta*grad_delta. For a general loss, use the same update with the minibatch gradient and optionally clip delta's norm.

2. Pseudocode: choose a grid of nu values, for example {0, 0.1, 0.3, 1.0} times an estimated largest Hessian eigenvalue, and stopping horizons K in {100, 300, 1000}. Initialize delta=0. At every minibatch, compute g=gradient(loss(theta0+delta)); set delta=(1+eta*nu)*delta-eta*g; apply global clipping if ||delta|| exceeds a preset multiple of the baseline delta norm; evaluate validation loss at each checkpoint and retain the best (nu,K) pair. The continuous-time spectral prediction is g_nu,t(s)=(1-exp(-(s-nu)t))/(s-nu), with value t when s=nu.

3. Quantities computed from the mathematics versus estimated: eta, nu, and K are hyperparameters in the first MVP. Estimate the top curvature scale s_max with 5-10 Hessian-vector power iterations, using Hessian-vector products through the selected block. Optionally estimate a few eigenvalues s_i and compare observed parameter or prediction amplification against f_nu,t(s_i). The paper supplies the finite-time filter and removable-pole mechanism; the nonlinear network's changing Hessian and minibatch noise must be measured empirically.

4. First experiment: use CIFAR-10 with a frozen pretrained small ViT or ResNet and a trainable linear head, comparing SGD, SGD with positive weight decay, and this method at matched learning-rate schedules and FLOPs. Run a second test on LoRA fine-tuning of a small language model. Record validation accuracy versus optimizer steps, final accuracy, norm growth, and divergence rate. Success is lower validation loss or higher accuracy than SGD or AdamW at equal steps, particularly in low-data fine-tuning, without more than 2x norm growth or materially higher instability. An additional success signal is faster early loss descent caused by amplification of high-curvature, signal-bearing directions.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a readable spectral NS-GD MVP and an anisotropic quadratic-regression comparison in experiment.py. The removable-pole limit matched f(nu)=nu*t exactly, off-pole error scaled linearly, mixed-sign crossover roots agreed with dense sweeps within 2.2e-5, and the Euler stability boundary matched eta=2/(s-nu) within 2.6e-4. The regression run did not show an accuracy win: NS-GD test MSE was 0.6523 versus 0.5875 for GD and 0.5859 for positive ridge, with 2.04x the GD parameter norm.

### Mechanism check

- Verdict: Built a readable spectral NS-GD MVP and an anisotropic quadratic-regression comparison in experiment.py. The removable-pole limit matched f(nu)=nu*t exactly, off-pole error scaled linearly, mixed-sign crossover roots agreed with dense sweeps within 2.2e-5, and the Euler stability boundary matched eta=2/(s-nu) within 2.6e-4. The regression run did not show an accuracy win: NS-GD test MSE was 0.6523 versus 0.5875 for GD and 0.5859 for positive ridge, with 2.04x the GD parameter norm.
- Confidence: 9/10
- Limitations: Only a deterministic full-batch linear regression toy was tested; no CIFAR-10, neural-network or LoRA block, minibatch noise, validation-based nu/K selection, Hessian-vector estimation, clipping, or wall-clock/FLOP comparison was run.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.1835
- Paired wins: 6/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/511/bench_report.json)
- [experiment.py](https://synthcore.org/code/511/experiment.py)
- [report.md](https://synthcore.org/code/511/report.md)
- [report_bench_2026-08-31T163415.md](https://synthcore.org/code/511/report_bench_2026-08-31T163415.md)
- [results.json](https://synthcore.org/code/511/results.json)
- [stage2_negative_shift_bench.py](https://synthcore.org/code/511/stage2_negative_shift_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/511)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.