# Primal-Dual Active-Set Optimizer Filter

- ID: 2265
- Canonical URL: https://synthcore.org/idea/2265/primal-dual-active-set-optimizer-filter
- API JSON: https://synthcore.org/api/idea/2265.json
- API Markdown: https://synthcore.org/api/idea/2265.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.13841](https://arxiv.org/abs/2608.13841)
- Category: optimization
- Solves: stability, speedup
- ML areas: optimizer, training-dynamics, speedup
- Math tags: optimization, control-theory, linear-algebra, numerical-analysis
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Use the paper's structure-exploiting primal-dual active-set strategy to solve barrier-constrained neural updates without invoking a generic quadratic-program solver at every step. The active constraints identify which layers or state statistics are actually close to instability, while warm-started multipliers and active sets should make the safety correction nearly constant-cost when the training trajectory changes smoothly.

## Mathematical statement

Write the barrier-filter quadratic program as \(\min_v \frac12v^THv+c^Tv\) subject to \(Av\ge b\), where \(H=(1+\lambda)I\succ0\), \(c=-v_{\rm nom}\), row \(a_j^T=\nabla h_j(\theta)^T\), and \(b_j=-\alpha_jh_j(\theta)\). Introduce multipliers \(\mu\ge0\) for constraints \(Av-b\ge0\). The KKT conditions are \(Hv+c-A^T\mu=0\), \(Av-b\ge0\), \(\mu\ge0\), and \(\mu_j(Av-b)_j=0\). For a guessed active set \(\mathcal A\), solve the equality-constrained saddle system, then add violated inactive constraints or remove active constraints with negative multipliers. Warm starting from the previous \((v,\mu,\mathcal A)\) is the transferable real-time mechanism.

## Key formulas

- $$\min_v\;\frac12v^THv+c^Tv\quad\mathrm{s.t.}\quad Av\ge b,\qquad H\succ0$$
- $$Hv+c-A^T\mu=0,\quad Av-b\ge0,\quad \mu\ge0,\quad \mu_j(Av-b)_j=0$$
- $$\begin{bmatrix}H&-A_{\mathcal A}^T\\A_{\mathcal A}&0\end{bmatrix}\begin{bmatrix}v\\\mu_{\mathcal A}\end{bmatrix}=\begin{bmatrix}-c\\b_{\mathcal A}\end{bmatrix}$$

## Implementation notes

1. Integration point: place the solver between gradient computation and the parameter update. Keep the neural update in a low-dimensional control space when possible: one velocity per layer, per residual branch, or per activation-statistic constraint. This makes the KKT system small even when the network has millions of parameters. If full-parameter constraints are required, use matrix-free products \(Av\) and \(A^T\mu\) through autodiff. 2. Pseudocode: initialize the active set and multipliers from the previous step; construct \(A\) from barrier gradients and \(b_j=-\alpha_jh_j\); solve the saddle system for the current active set; if any inactive residual \(a_j^Tv-b_j<0\), add the most violated row; if any active multiplier \(\mu_j<0\), remove the most negative row; repeat until primal residuals and multiplier violations are below tolerance; apply \(\theta\leftarrow\theta+\eta v\). 3. Computed exactly: KKT residuals, active-set changes, projected velocity, and dual multipliers. Estimated empirically: barrier gradients and the number of active constraints. Use diagonal or block-diagonal \(H\) so each KKT solve is cheap. 4. First cheap experiment: use a 4-layer MLP on CIFAR-10 with layerwise norm and activation barriers, comparing a generic projected-gradient approximation against the warm-started active-set solver. Measure wall-clock time, number of KKT iterations, active-set changes, constraint violations, and training loss. The quantitative prediction is that after an initial transient, consecutive minibatches will share the same active set and require at most 1-3 active-set corrections per step; solver iterations and active-set changes should spike precisely when the nominal optimizer crosses a barrier. Verify that KKT residuals stay below \(10^{-5}\) and that the active-set method reaches the same constrained solution as a reference OSQP solve while reducing per-step quadratic-program time.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a primal-dual active-set solver for strictly convex inequality QPs with warm-started active sets, multiplier checks, violated-constraint additions, and negative-multiplier removals. The activity transition matched the predicted alpha=0.4625 within the 0.05 sweep resolution, dual scaling matched the predicted slope 0.8 to machine precision, and smooth trajectories used one iteration on 92.5% of steps with 2 active-set changes and maximum KKT residual 5.6e-17. On the small toy comparison, PDAS reached the same objective as sequential projected gradient while averaging 1.59 ms versus 44.16 ms, but this is not evidence of a neural-training speedup.

### Mechanism check

- Verdict: Built a primal-dual active-set solver for strictly convex inequality QPs with warm-started active sets, multiplier checks, violated-constraint additions, and negative-multiplier removals. The activity transition matched the predicted alpha=0.4625 within the 0.05 sweep resolution, dual scaling matched the predicted slope 0.8 to machine precision, and smooth trajectories used one iteration on 92.5% of steps with 2 active-set changes and maximum KKT residual 5.6e-17. On the small toy comparison, PDAS reached the same objective as sequential projected gradient while averaging 1.59 ms versus 44.16 ms, but this is not evidence of a neural-training speedup.
- Confidence: 8/10
- Limitations: Only small synthetic QPs were tested. No CIFAR-10/MLP integration, full-parameter or autodiff barrier gradients, OSQP comparison, GPU benchmark, scaling study, infeasibility handling, or high-dimensional matrix-free implementation was tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/894/bench_report.json)
- [pdas_bench.py](https://synthcore.org/code/894/pdas_bench.py)
- [pdas_filter.py](https://synthcore.org/code/894/pdas_filter.py)
- [report.md](https://synthcore.org/code/894/report.md)
- [report_bench_2026-09-01T135343.md](https://synthcore.org/code/894/report_bench_2026-09-01T135343.md)
- [results.json](https://synthcore.org/code/894/results.json)
- [run_experiment.py](https://synthcore.org/code/894/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/894)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.