# Algebraically smoothed ReLU for differentiable planning

- ID: 853
- Canonical URL: https://synthcore.org/idea/853/algebraically-smoothed-relu-for-differentiable-planning
- API JSON: https://synthcore.org/api/idea/853.json
- API Markdown: https://synthcore.org/api/idea/853.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.13289](https://arxiv.org/abs/2607.13289)
- Category: optimization
- Solves: stability, accuracy
- ML areas: world-model, rl, training-dynamics, loss
- Math tags: numerical-analysis, optimization, approximation-theory, dynamical-systems
- Ratings: usefulness 7/10; difficulty 3/10; novelty 4/10

## Idea description

When a neural network is placed inside a Newton, SQP, or interior-point optimization loop, replace its ReLUs only in the embedded inference graph by a smooth algebraic approximation. The approximation is uniformly close to ReLU but has well-defined first and second derivatives, improving Hessian-based action optimization without retraining or changing the learned weights.

## Mathematical statement

The paper replaces \(\sigma(z)=\max(0,z)\) with \(\widetilde\sigma_\varepsilon(z)=\frac12(z+\sqrt{z^2+\varepsilon})\), where \(z\in\mathbb R\) is a preactivation and \(\varepsilon>0\) is the smoothing parameter. The function is smooth, converges to ReLU as \(\varepsilon\to0\), and obeys the uniform bound \(|\widetilde\sigma_\varepsilon(z)-\sigma(z)|\le\sqrt{\varepsilon}/2\). Its derivatives are \(\widetilde\sigma_\varepsilon'(z)=\frac12(1+z/\sqrt{z^2+\varepsilon})\) and \(\widetilde\sigma_\varepsilon''(z)=\varepsilon/[2(z^2+\varepsilon)^{3/2}]\). Thus the value perturbation is bounded while curvature is explicitly available near the ReLU kink. Use the smoothed graph for differentiation and constrained optimization, while optionally retaining the original ReLU graph for ordinary predictions.

## Key formulas

- $$\sigma(z)=\max(0,z),\qquad \widetilde{\sigma}_\varepsilon(z)=\frac12\left(z+\sqrt{z^2+\varepsilon}\right).$$
- $$\left|\widetilde{\sigma}_\varepsilon(z)-\sigma(z)\right|\le\frac{\sqrt{\varepsilon}}{2}.$$
- $$\widetilde{\sigma}_\varepsilon'(z)=\frac12\left(1+\frac{z}{\sqrt{z^2+\varepsilon}}\right),\qquad \widetilde{\sigma}_\varepsilon''(z)=\frac{\varepsilon}{2(z^2+\varepsilon)^{3/2}}.$$
- $$\min_{a,\eta}\;J(a)+w_\eta\sum_{i=1}^{N_h}\eta_i\quad\text{s.t.}\quad \widehat{g}_i(a)\le c_i,\;\widehat{g}_i(a)+\eta_i\ge \ell_i,\;\eta_i\ge0.$$

## Implementation notes

Integrate at the neural-network forward pass used inside differentiable model-predictive control, trajectory optimization, constrained policy improvement, or any solver consuming network Jacobians and Hessians. Keep all trained weights unchanged and add a solver-only activation mode: every ReLU in the planning copy is evaluated as `0.5*(z + sqrt(z*z + eps))`; ordinary training and unconstrained evaluation can continue using ReLU. Pseudocode is `def smooth_relu(z,eps): return 0.5*(z+sqrt(z*z+eps))`; then for each layer use `z=linear(z)` followed by `z=smooth_relu(z,eps)`, compute `prediction=net_smooth(state,action_sequence)`, define the control objective and constraints, and call an SQP or interior-point solver through automatic differentiation. The exact paper quantities are the activation and its uniform approximation bound; solver behavior must be measured empirically using gradient norms, Hessian condition numbers, line-search failures, iterations, final objective, and constraint violations. Start with standardized activations and sweep \(\varepsilon\in\{10^{-8},10^{-6},10^{-4},10^{-2}\}\). Note that the curvature peak is \(\widetilde\sigma''(0)=1/(2\sqrt\varepsilon)\), so overly small values can cause stiffness. Use a 2-layer ReLU dynamics surrogate on Pendulum or constrained CartPole, optimize an 8-step action sequence, and compare raw ReLU, this smoother, softplus, and tanh. Success means fewer failed solves and lower constraint violation at equal or better task cost, with negligible prediction drift.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built experiment.py implementing algebraically smoothed ReLU, analytic derivatives, exact scaling sweeps, and a fixed-weight constrained action-optimization comparison. The mathematical mechanism was confirmed: value error matched sqrt(epsilon)/2, curvature matched 1/(2 sqrt(epsilon)), observed log-log slopes were 0.5 and -0.5, and scaled transition derivatives were epsilon-invariant. In the toy optimizer, epsilon=1e-4 and 1e-2 matched or slightly improved ReLU loss, while epsilon<=1e-6 showed stiffness, so the practical gain is parameter-dependent rather than universal.

### Mechanism check

- Verdict: Built experiment.py implementing algebraically smoothed ReLU, analytic derivatives, exact scaling sweeps, and a fixed-weight constrained action-optimization comparison. The mathematical mechanism was confirmed: value error matched sqrt(epsilon)/2, curvature matched 1/(2 sqrt(epsilon)), observed log-log slopes were 0.5 and -0.5, and scaled transition derivatives were epsilon-invariant. In the toy optimizer, epsilon=1e-4 and 1e-2 matched or slightly improved ReLU loss, while epsilon<=1e-6 showed stiffness, so the practical gain is parameter-dependent rather than universal.
- Confidence: 9/10
- Limitations: The practical comparison is a tiny synthetic two-action projected finite-difference optimizer, not a full SQP or interior-point MPC solver or a Pendulum/CartPole benchmark. It does not measure Hessian conditioning, line-search failures, FLOPs, GPU behavior, retraining effects, or multi-step dynamics; the constraint metric includes the initial trajectory and is therefore conservative.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 0/0
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/345/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/345/bench_report.json)
- [experiment.py](https://synthcore.org/code/345/experiment.py)
- [report.md](https://synthcore.org/code/345/report.md)
- [report_bench_2026-08-31T014343.md](https://synthcore.org/code/345/report_bench_2026-08-31T014343.md)
- [results.json](https://synthcore.org/code/345/results.json)
- [Download all files as ZIP](https://synthcore.org/download/345)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.