✓✓ Beats tuned baseline 2026

Active-Set Reduced Differentiable QP Layer

Implementation & benchmark of arXiv:2608.04189 — Structured Differentiable Optimization for Efficient Decision-focused Learning in Power Systems

Usefulness8/10

Difficulty5/10

Novelty5/10

Optimization Speed Memory Stability Optimizer Training dynamics Loss Optimization Linear algebra Numerical analysis

Source paper: Structured Differentiable Optimization for Efficient Decision-focused Learning in Power Systems arXiv:2608.04189 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace full-KKT implicit differentiation through a constrained quadratic-program layer with differentiation through only the equality constraints and inequalities active at the optimum. The forward solver still enforces all constraints, but the backward linear system scales with the active-set size rather than the total number of inequalities.

Formulas

$$\min_z\;\frac12 z^\top P(\phi)z+q(\phi)^\top z+c(\phi)\quad\text{s.t.}\quad A(\phi)z=b(\phi),\;G(\phi)z\le h(\phi).$$

$$K_{\mathcal A}=\begin{bmatrix}P&A^\top&G_{\mathcal A}^\top\\A&0&0\\G_{\mathcal A}&0&0\end{bmatrix},\qquad K_{\mathcal A}\begin{bmatrix}dz\\d\lambda\\d\nu_{\mathcal A}\end{bmatrix}=-\begin{bmatrix}d(Pz+q+A^\top\lambda+G_{\mathcal A}^\top\nu_{\mathcal A})\\d(Az-b)\\d(G_{\mathcal A}z-h_{\mathcal A})\end{bmatrix}_{dz,d\lambda,d\nu_{\mathcal A}\;\mathrm{held\ fixed}}.$$

$$\frac{\partial \mathcal L}{\partial\phi}=\frac{\partial\ell}{\partial\phi}+\frac{\partial\ell}{\partial z}\frac{\partial z^\star}{\partial\phi},\qquad \frac{\partial z^\star}{\partial\phi}\text{ is obtained by solving the reduced system for each seed direction}.$$

$$\mathrm{cost}_{\mathrm{backward}}\approx O\!\left((n+n_{eq}+|\mathcal A|)^3\right),\qquad \mathrm{cost}_{\mathrm{full}}\approx O\!\left((n+n_{eq}+m)^3\right),$$

Mathematical statement

Consider the convex quadratic program $z^\star(\phi)=\arg\min_z \frac{1}{2}z^\top P(\phi)z+q(\phi)^\top z+c(\phi)$ subject to $A(\phi)z=b(\phi)$ and $G(\phi)z\le h(\phi)$, where $z\in\mathbb{R}^n$ is the optimization variable and $\phi$ is a neural-network output. Let $\lambda$ be equality multipliers and $\nu\ge0$ inequality multipliers. The KKT equations are $Pz+q+A^\top\lambda+G^\top\nu=0$, $Az-b=0$, $Gz-h\le0$, and $\nu_i(G_i z-h_i)=0$. Define the active set $\mathcal A=\{i:G_i z^\star-h_i\ge-\tau\}$, with tolerance $\tau$, and let $G_{\mathcal A}$ and $h_{\mathcal A}$ contain only active rows. Under LICQ, strict complementarity, and a nonsingular reduced KKT matrix, perturbations satisfy $K_{\mathcal A}[dz,d\lambda,d\nu_{\mathcal A}]^\top=-[r_z,r_b,r_{\mathcal A}]^\top$, where $K_{\mathcal A}=\begin{bmatrix}P&A^\top&G_{\mathcal A}^\top\\A&0&0\\G_{\mathcal A}&0&0\end{bmatrix}$. The residual vector is the total differential of the three active KKT equations, excluding the unknown differentials $dz,d\lambda,d\nu_{\mathcal A}$. Inactive inequalities contribute no columns or multiplier variables because their multipliers are zero locally.

Implementation notes

Integrate this as a custom autograd layer whose forward pass solves a convex QP and whose backward pass uses the cached primal-dual solution. The network produces $\phi$, which parameterizes $P,q,A,b,G,h$; the QP returns $z^\star$ to the downstream loss. Pseudocode: solve the full QP; mark $\mathcal A=\{i:h_i-G_i z^\star\le\tau\}$ and optionally require $\nu_i>\tau_\nu$; assemble the reduced KKT matrix; factor it once; for each upstream vector, solve the transposed reduced system and apply the chain rule through all parameter maps. Do not differentiate the discrete active-set test. Compute $z^\star,\lambda,\nu$ and feasibility residuals from the QP solver; estimate $\tau$ from solver tolerances and validate the reduced Jacobian by finite differences away from active-set transitions. The first cheap experiment should use a batch of small QPs embedded after an MLP, with $n=32$, $m=256$, and active-set fractions of 5%, 20%, and 50%, comparing a full-KKT layer against the reduced layer in PyTorch. Measure backward time, peak memory, gradient cosine similarity, and failures near active-set changes. The prediction is that backward factorization time follows $(n+n_{eq}+|\mathcal A|)^3$ and that the reduced method is fastest when $|\mathcal A|/m\ll1$; gradients should agree within $10^{-3}$ relative error while the active set remains unchanged.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a readable NumPy MVP for active-set reduced implicit differentiation, including strict-complementarity QPs, finite-difference verification, full-vs-reduced Jacobian comparison, and timing sweeps. The core mechanism manifested: finite-difference relative error reached about 4e-10, gradient cosine similarity was 1.0 with approximately 1e-15 relative Jacobian error, and reduced solves were 6.4x–12.0x faster. The ideal cubic scaling prediction was not confirmed, with an observed exponent of about 1.75.

Agent confidence: 8/10

Baseline: Full complementarity KKT solve: 0.0191–0.0269 s; gradient cosine approximately 1.0; full-vs-reduced Jacobian relative error approximately 5e-17–3e-15.
Idea: Reduced KKT solve: 0.00182–0.00419 s; observed speedups 7.04x, 12.04x, and 6.41x for active fractions 5%, 20%, and 50%; finite-difference relative error 4.3e-14 to 4.1e-10.

Limitations: This is a NumPy verification MVP rather than the requested PyTorch custom-autograd layer. It does not test an MLP, GPU execution, batched QPs, peak memory, parameter-dependent P/A/G maps, a production forward QP solver, or active-set transitions. The cubic cost law was not quantitatively confirmed.

How to run: python3 active_set_qp_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.04189) Sep 2, 2026

Beats the tuned baseline by 13.2% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Directional polytope
Model: mlp
Paired seeds: 8
Baseline mean: 0.004
Idea mean: 0.0035
Effect (Δ): -0.0005 (−13.2%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±4.1%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

The registered track provides polytope-structured regression but the implementation uses a separable box-clamping QP rather than a general dense QP with parameter-dependent P, A, G, and h. Peak GPU memory, large inequality counts, active-set transitions, and production QP solver integration were not tested. The reported speed signature uses end-to-end training wall time, not an isolated backward timing; mean_backward_seconds is unavailable.

How to run: python3 qp_active_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Worked ✓

Agent confidence: 8/10

Implemented and benchmarked matched full-KKT and active-set reduced backward QP systems on the registered directional_polytope track with an identical MLP and equal learning-rate sweep. The idea improved mean test MSE from 0.00397446 to 0.00345121, with all 8 paired seeds improving and permutation p=0.0081. The mechanism signature also observed approximately 53.45x lower end-to-end training time for the reduced system, although the timing is not an isolated backward-kernel measurement.

Baseline: Full-KKT system, best lr=0.003 and 18 epochs: mean test MSE 0.0039744608; sweep means lr=0.001/0.003/0.01 = 0.04984972/0.00397446/0.00528108.
Idea: Reduced active-set system, best lr=0.01 and 18 epochs: mean test MSE 0.0034512121; sweep means lr=0.001/0.003/0.01 = 0.04921583/0.00389726/0.00345121; paired delta=-0.0005232487; 8/8 wins; p=0.0081.

Benchmark result