△ Mechanism confirmed, baseline not beaten 2026

Positive-Regime Observable ReLU State Space

Implementation & benchmark of arXiv:2608.29650 — On the Number of Observation Nodes in Recurrent Neural Networks with Linear Threshold and ReLU Functions

Usefulness7/10

Difficulty5/10

Novelty6/10

Architecture Stability Accuracy Generalization Rnn Ssm Memory Regularization Control theory Dynamical systems Linear algebra Spectral theory

Source paper: On the Number of Observation Nodes in Recurrent Neural Networks with Linear Threshold and ReLU Functions arXiv:2608.29650 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Constrain recurrent preactivations to remain nonnegative so that ReLU acts as the identity along realized trajectories. The hidden dynamics then admit a classical linear observability matrix, allowing principled hidden-coordinate selection and conditioning control instead of relying on potentially destructive activation masks.

Formulas

xₜ₊₁ = [A xₜ + B uₜ + b]₊, A xₜ + B uₜ + b ≥ 0 componentwise ⇒ xₜ₊₁ = A xₜ + B uₜ + b

δxₜ₊₁ = A δxₜ, Q_T = [C; CA; CA²; …; CAᵀ]

rank(Q_T) = n, σ_min(Q_T) > 0

L_pos = Σₜ ||[−A xₜ − B uₜ − b]₊||₂², L_cond = −log det(Q_Tᵀ Q_T + εI)

Mathematical statement

Consider xₜ₊₁ = [A xₜ + B uₜ + b]₊, where xₜ ∈ Rⁿ is the state, uₜ is the input, A and B are matrices, and b is a bias. If A xₜ + B uₜ + b is componentwise nonnegative for all relevant states and inputs, then ReLU is inactive and the dynamics become affine linear. Perturbations satisfy δxₜ₊₁ = A δxₜ. With observations yₜ = C xₜ, the finite-horizon observability matrix is Q_T = [C; CA; …; CAᵀ]. The initial state is observable when rank(Q_T) = n, and robustness is measured by the smallest singular value of Q_T or the condition number of Q_TᵀQ_T. A trainable positivity penalty is the squared norm of the negative preactivation, which encourages the regime in which the paper's exact reduction to linear observability applies.

Implementation notes

Integration point: replace the hidden transition of a small recurrent state-space model with xₜ₊₁ = relu(A xₜ + B uₜ + b), and expose m selected hidden coordinates through yₜ = C xₜ or an auxiliary memory decoder. Add the positivity and observability losses to the task loss. Pseudocode: unroll the sequence and compute zₜ = A xₜ + B uₜ + b; accumulate the positivity penalty ||relu(−zₜ)||²; update xₜ₊₁ = relu(zₜ). Separately construct Q = [C; CA; …; CAᵀ] for a chosen horizon T, calculate QᵀQ + εI, and add its negative log determinant to the objective. Optionally constrain A with spectral normalization if bounded recurrent dynamics are also desired. The paper supplies the exact condition under which ReLU observability reduces to linear observability; the positivity rate and singular values are measured empirically. First experiment: use n = 16 on the copy-memory and adding-problem datasets, compare unconstrained ReLU, positive-penalty ReLU, and tanh RNNs with m = 4 observed coordinates. Evaluate horizons T = 1, 2, 4, 8, and 16. The prediction is that when the negative-preactivation fraction falls below 1%, reconstruction quality follows the linear criterion: rank deficiency produces σ_min(Q_T) ≈ 0 and high reconstruction error, while full rank yields decreasing error as T increases until the smallest singular value saturates. Report whether observed rank transitions match Q_T within numerical tolerance.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

The positive-regime ReLU mechanism was numerically confirmed. ReLU exactly matched affine dynamics with zero negative preactivations, full observability appeared at the predicted horizon T=5, and reconstruction error scaled inversely with sigma_min using an observed log-log slope of -0.993. The positive model had perturbation error 8.5e-11 versus 1.025 for signed ReLU, but no end-to-end task win was tested.

Agent confidence: 9/10

Baseline: Signed ReLU: negative-preactivation fraction 0.490; linearization error 1.025; rank 8; sigma_min(Q) 0.0603.
Idea: Positive-regime ReLU: negative-preactivation fraction 0; positivity penalty 0; linearization error 8.5e-11; rank 8; sigma_min(Q) 0.0276. Rank transition matched prediction at T=5, and inverse-singular-value scaling matched with slope -0.993.

Limitations: No learned copy-memory or adding task, decoder training, conditioning-loss optimization, spectral normalization, or end-to-end accuracy comparison was tested; experiments used controlled toy systems.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.29650) Sep 2, 2026

No significant difference vs. the tuned baseline (+2.2%, p=0.7015, wins 4 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0001
Idea mean: 0.0001
Effect (Δ): 0 (+2.2%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.7015 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in dynamics track was tested with 1,200 training and 400 test samples for 20 epochs. The observability determinant/conditioning loss and explicit coordinate-selection readout were not included in the end-to-end intervention; only the positivity penalty was tested. The baseline and idea used small three-configuration sweeps.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 9/10

Built and evaluated Positive-Regime Observable ReLU on the structurally matched controlled-pendulum dynamics track. The math check passed exactly, and trained-model behavior confirmed reduced negative preactivations and a smaller ReLU-affine gap. Nevertheless, the idea's test MSE was worse than baseline and the paired permutation test was non-significant.

Baseline: lr=0.01, 20 epochs; mean test MSE=0.0001016568, std=0.0000333668.
Comparison: paired delta=+0.0000022138, idea wins=4/8, permutation p=0.70145, verdict=no significant win.
Idea: lr=0.01, penalty=0.01, 20 epochs; mean test MSE=0.0001038706, std=0.0000340591.
Math check: Affine difference=0.0, negative fraction=0.0, perturbation residual=1.03e-15, pass=true.
Mechanism signature: Baseline negative-preactivation fraction=0.6035 and affine gap=0.24998; idea fraction=0.5734 and gap=0.16072; confirmed=true.

Benchmark result