# LP-Embedded Input-Convex MLP

- ID: 2056
- Canonical URL: https://synthcore.org/idea/2056/lp-embedded-input-convex-mlp
- API JSON: https://synthcore.org/api/idea/2056.json
- API Markdown: https://synthcore.org/api/idea/2056.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.09707](https://arxiv.org/abs/2608.09707)
- Category: architecture
- Solves: scalability, stability, accuracy
- ML areas: mlp, optimizer, training
- Math tags: convex-analysis, optimization, linear-algebra
- Ratings: usefulness 8/10; difficulty 5/10; novelty 5/10

## Idea description

Replace a standard ReLU surrogate with an input convex neural network whose hidden-to-hidden weights are constrained to be nonnegative. The network remains piecewise linear and expressive, but its convexity allows downstream minimization to use continuous ReLU epigraph constraints instead of binary activation variables, potentially eliminating the integrality bottleneck of neural optimization.

## Mathematical statement

For input x in R^d, define z_0=x and hidden states z_{k+1}=ReLU(W_k z_k+U_k x+b_k), where z_k is the vector of hidden activations, W_k is the hidden-to-hidden matrix, U_k is the direct input matrix, b_k is a bias vector, and ReLU(t)=max(0,t) is applied coordinatewise. If W_k is componentwise nonnegative for every k and the output is y=w^T z_L+u^T x+c with w componentwise nonnegative, then y=f(x) is convex in x; u and c are unrestricted, and L is the number of hidden layers. Enforce W_k>=0 by parameterizing W_k=softplus(A_k), where A_k is unconstrained. For minimization, each ReLU can be represented by its epigraph constraints z_{k+1}>=W_k z_k+U_k x+b_k and z_{k+1}>=0. Because the output coefficients w are nonnegative, minimizing y drives every z_{k+1} to the smallest feasible value, making this LP embedding exact whenever the optimization model does not require upper-side activation constraints or nonconvex interactions. The supervised fitting objective is L_fit=(1/N)sum_{i=1}^N(f(x_i)-y_i)^2, optionally augmented with a small weight-decay term; convexity is architectural rather than estimated from sampled Hessians.

## Key formulas

- $$z_0=x,\qquad z_{k+1}=\operatorname{ReLU}(W_k z_k+U_kx+b_k),\qquad y=f(x)=w^\top z_L+u^\top x+c$$
- $$W_k\geq 0,\quad w\geq 0\quad\Longrightarrow\quad f(x)\ \text{is convex in }x$$
- $$z_{k+1}\geq W_kz_k+U_kx+b_k,\qquad z_{k+1}\geq 0,\qquad \min\; w^\top z_L+u^\top x+c$$
- $$W_k=\operatorname{softplus}(A_k),\qquad \mathcal{L}_{\mathrm{fit}}=\frac{1}{N}\sum_{i=1}^{N}\left(f(x_i)-y_i\right)^2+\lambda\sum_k\left(\lVert A_k\rVert_F^2+\lVert U_k\rVert_F^2\right)$$

## Implementation notes

Integrate this as a surrogate MLP in a two-stage pipeline: first fit the network to expensive simulator or historical data, then insert its equations into a downstream linear or mixed-integer optimization model. In PyTorch, create unconstrained parameters A_k, U_k, b_k, u, and c; compute W_k=softplus(A_k)+epsilon; run the recurrence z[0]=x and z[k+1]=relu(W_k @ z[k] + U_k @ x + b_k); use the squared fitting loss shown above. Pseudocode: inputs x_batch,y_batch; for each layer compute W=softplus(A)+1e-6 and z=relu(W*z+U*x+b); output y=w*z+u*x+c; backpropagate and update all unconstrained parameters with AdamW. Keep w nonnegative using w=softplus(a_w). After training, export variables x,z_k and constraints z[k+1]>=W_k z[k]+U_k x+b_k and z[k+1]>=0 to scipy.optimize.linprog, HiGHS, or a larger LP/MIP model. The mathematical guarantees are the convexity implication from W_k>=0 and w>=0 and the exactness of the epigraph embedding for minimization; estimate practical approximation quality, solve-time benefit, and whether the application’s feasible set preserves convexity empirically. Start with a 3-layer width-64 surrogate for a synthetic convex function such as f(x)=||Bx||_2^2 plus noise, then test the same architecture on a small tabular benchmark with a known convex downstream objective. Compare against a width-matched ReLU MLP embedded with binary ReLU variables. Success means similar validation MSE, fewer branch-and-bound nodes, and materially lower wall-clock solve time; the strongest result is root-node termination with no binary variables and no degradation in the downstream objective.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built an ICNN surrogate with softplus-constrained hidden-to-hidden weights, controlled convexity sweeps, exact LP epigraph checks, and a matched ReLU MLP fitting comparison. Nonnegative weights produced only numerical Jensen violations (about 4e-14), the controlled sign transition matched predictions exactly, and one- and two-layer LP epigraph objectives matched forward ReLU values within 1.8e-15. The ICNN had much worse validation MSE than the ordinary ReLU MLP, so the optimization/embedding mechanism is supported but comparable approximation quality was not achieved.

### Mechanism check

- Verdict: Built an ICNN surrogate with softplus-constrained hidden-to-hidden weights, controlled convexity sweeps, exact LP epigraph checks, and a matched ReLU MLP fitting comparison. Nonnegative weights produced only numerical Jensen violations (about 4e-14), the controlled sign transition matched predictions exactly, and one- and two-layer LP epigraph objectives matched forward ReLU values within 1.8e-15. The ICNN had much worse validation MSE than the ordinary ReLU MLP, so the optimization/embedding mechanism is supported but comparable approximation quality was not achieved.
- Confidence: 8/10
- Limitations: No full downstream application, binary ReLU MIP, branch-and-bound node count, or wall-clock solve-time comparison was tested. The fitting comparison used one small synthetic quadratic dataset and CPU execution; the ICNN architecture was not extensively hyperparameter-tuned.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.94485
- Paired wins: 4/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/772/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/772/bench_report.json)
- [experiment.py](https://synthcore.org/code/772/experiment.py)
- [report.md](https://synthcore.org/code/772/report.md)
- [report_bench_2026-09-01T100929.md](https://synthcore.org/code/772/report_bench_2026-09-01T100929.md)
- [results.json](https://synthcore.org/code/772/results.json)
- [Download all files as ZIP](https://synthcore.org/download/772)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.