✗ Failed on benchmark 2026

Algebraically smoothed ReLU for differentiable planning

Implementation & benchmark of arXiv:2607.13289 — Model predictive control for laser thermal processing: operator learning, closed-loop validation, and out-of-distribution analysis

Usefulness7/10

Difficulty3/10

Novelty4/10

Optimization Stability Accuracy World model Rl Training dynamics Loss Numerical analysis Optimization Approximation theory Dynamical systems

Source paper: Model predictive control for laser thermal processing: operator learning, closed-loop validation, and out-of-distribution analysis arXiv:2607.13289 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

When a neural network is placed inside a Newton, SQP, or interior-point optimization loop, replace its ReLUs only in the embedded inference graph by a smooth algebraic approximation. The approximation is uniformly close to ReLU but has well-defined first and second derivatives, improving Hessian-based action optimization without retraining or changing the learned weights.

Formulas

$$\sigma(z)=\max(0,z),\qquad \widetilde{\sigma}_\varepsilon(z)=\frac12\left(z+\sqrt{z^2+\varepsilon}\right).$$

$$\left|\widetilde{\sigma}_\varepsilon(z)-\sigma(z)\right|\le\frac{\sqrt{\varepsilon}}{2}.$$

$$\widetilde{\sigma}_\varepsilon'(z)=\frac12\left(1+\frac{z}{\sqrt{z^2+\varepsilon}}\right),\qquad \widetilde{\sigma}_\varepsilon''(z)=\frac{\varepsilon}{2(z^2+\varepsilon)^{3/2}}.$$

$$\min_{a,\eta}\;J(a)+w_\eta\sum_{i=1}^{N_h}\eta_i\quad\text{s.t.}\quad \widehat{g}_i(a)\le c_i,\;\widehat{g}_i(a)+\eta_i\ge \ell_i,\;\eta_i\ge0.$$

Mathematical statement

The paper replaces $\sigma(z)=\max(0,z)$ with $\widetilde\sigma_\varepsilon(z)=\frac12(z+\sqrt{z^2+\varepsilon})$, where $z\in\mathbb R$ is a preactivation and $\varepsilon>0$ is the smoothing parameter. The function is smooth, converges to ReLU as $\varepsilon\to0$, and obeys the uniform bound $|\widetilde\sigma_\varepsilon(z)-\sigma(z)|\le\sqrt{\varepsilon}/2$. Its derivatives are $\widetilde\sigma_\varepsilon'(z)=\frac12(1+z/\sqrt{z^2+\varepsilon})$ and $\widetilde\sigma_\varepsilon''(z)=\varepsilon/[2(z^2+\varepsilon)^{3/2}]$. Thus the value perturbation is bounded while curvature is explicitly available near the ReLU kink. Use the smoothed graph for differentiation and constrained optimization, while optionally retaining the original ReLU graph for ordinary predictions.

Implementation notes

Integrate at the neural-network forward pass used inside differentiable model-predictive control, trajectory optimization, constrained policy improvement, or any solver consuming network Jacobians and Hessians. Keep all trained weights unchanged and add a solver-only activation mode: every ReLU in the planning copy is evaluated as 0.5*(z + sqrt(z*z + eps)); ordinary training and unconstrained evaluation can continue using ReLU. Pseudocode is def smooth_relu(z,eps): return 0.5*(z+sqrt(z*z+eps)); then for each layer use z=linear(z) followed by z=smooth_relu(z,eps), compute prediction=net_smooth(state,action_sequence), define the control objective and constraints, and call an SQP or interior-point solver through automatic differentiation. The exact paper quantities are the activation and its uniform approximation bound; solver behavior must be measured empirically using gradient norms, Hessian condition numbers, line-search failures, iterations, final objective, and constraint violations. Start with standardized activations and sweep $\varepsilon\in\{10^{-8},10^{-6},10^{-4},10^{-2}\}$. Note that the curvature peak is $\widetilde\sigma''(0)=1/(2\sqrt\varepsilon)$, so overly small values can cause stiffness. Use a 2-layer ReLU dynamics surrogate on Pendulum or constrained CartPole, optimize an 8-step action sequence, and compare raw ReLU, this smoother, softplus, and tanh. Success means fewer failed solves and lower constraint violation at equal or better task cost, with negligible prediction drift.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built experiment.py implementing algebraically smoothed ReLU, analytic derivatives, exact scaling sweeps, and a fixed-weight constrained action-optimization comparison. The mathematical mechanism was confirmed: value error matched sqrt(epsilon)/2, curvature matched 1/(2 sqrt(epsilon)), observed log-log slopes were 0.5 and -0.5, and scaled transition derivatives were epsilon-invariant. In the toy optimizer, epsilon=1e-4 and 1e-2 matched or slightly improved ReLU loss, while epsilon<=1e-6 showed stiffness, so the practical gain is parameter-dependent rather than universal.

Agent confidence: 9/10

Baseline: ReLU final loss 0.006626, prediction 0.715527, maximum historical constraint violation 0.169500
Idea: Smooth ReLU epsilon=1e-4: final loss 0.006171 and prediction 0.715527; epsilon=1e-2: loss 0.006327; epsilon=1e-6 and 1e-8: losses 0.387207 and 0.387162, demonstrating small-epsilon stiffness
Math: Maximum value error exactly matched sqrt(epsilon)/2; curvature peak matched 1/(2 sqrt(epsilon)); observed log-log slopes were 0.500000 and -0.500000; scaled-transition derivative spread was 1.1e-16

Limitations: The practical comparison is a tiny synthetic two-action projected finite-difference optimizer, not a full SQP or interior-point MPC solver or a Pendulum/CartPole benchmark. It does not measure Hessian conditioning, line-search failures, FLOPs, GPU behavior, retraining effects, or multi-step dynamics; the constraint metric includes the initial trajectory and is therefore conservative.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.13289) Sep 2, 2026

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 0 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0014
Idea mean: 0.0014
Effect (Δ): 0 (−-0.0%; negative = idea better)
Wins: 0 / 0 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 0 configs

Limitations:

The built-in dynamics architecture has no explicit ReLU, so this run does not validate the intended activation mechanism. A custom ReLU-based dynamics track was not added.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented and ran the eight-seed dynamics benchmark with a common learning-rate sweep and identical training budgets. Baseline and smooth-ReLU systems had identical mean test MSE 0.0014198302, paired delta 0, and permutation p=1.0; there was no win. The result is also a structural mismatch because rnn_small uses GRU activations rather than explicit ReLU layers.

Baseline: best lr=0.01; 8-seed mean MSE 0.0014198302378645167; std 0.00039839114901047844
Idea: epsilon=1e-4; best lr=0.01; 8-seed mean MSE 0.0014198302378645167; std 0.00039839114901047844

Benchmark result