Central-Path ReLU Inequality Layer

(1) Integration point: add a constrained head after an MLP, transformer projection, or router logits. Let the unconstrained tensor be \(w=f_\theta(z)\); apply the elementwise central-path layer \(x=\varphi_{+,s}(w)\) wherever the model requires a strictly positive slack, nonnegative allocation, positive intensity, or nonnegative mixture weight. For a simplex output, use \(x_i=\varphi_{+,s}(w_i)\) followed by normalization \(p_i=x_i/(\sum_jx_j+\epsilon)\). Do not use this layer where exact zeros are essential during early training unless an explicit threshold is added only at inference. (2) Pseudocode: w = net(z); x = 0.5*(w + sqrt(w*w + 4*s)); rc = x*(x-w)-s; task_loss = L(x,y); loss = task_loss + lambda_c*mean(rc*rc); loss.backward(); update(theta); s = max(s_min, rho*s) with \(0<\rho<1\). The analytic derivative in backpropagation is \(0.5+0.5*w/sqrt(w*w+4*s)\). Start with \(s_0\) equal to \(0.01\) times the median squared magnitude of \(w\), then reduce it only when the task loss and gradient norm are stable; compare geometric annealing with fixed \(s\). (3) Computed from the mathematics: the forward map, derivative, positivity guarantee, and complementarity residual formula. Estimated empirically: the initial scale, annealing rate, task-dependent tradeoff \(\lambda_c\), and whether the uniform \(\sqrt{s}\) approximation error is visible on the learned output distribution. Monitor minimum \(x\), mean and maximum \(|r_c|\), gradient variance, and fraction of outputs below \(2\sqrt{s}\). (4) First experiment: train a small MLP on nonnegative regression, such as predicting Poisson rates from MNIST or synthetic \(y=\exp(a^Tz)+\) noise, comparing ReLU, softplus, exponential, and this layer at matched parameter count and FLOPs. Then test a constrained mixture head on CIFAR-10 with cross-entropy after normalization. The expected signal is fewer NaNs and lower gradient variance than exponential outputs, lower final constraint violation than an unconstrained linear head with penalty, and no worse validation loss than softplus. A useful falsification is that annealing \(s\) fails to improve task loss or produces worse calibration at equal compute.