✗ Failed on benchmark 2026

Finite-Horizon Hidden-State Observability Regularizer

Implementation & benchmark of arXiv:2608.29650 — On the Number of Observation Nodes in Recurrent Neural Networks with Linear Threshold and ReLU Functions

Usefulness8/10

Difficulty6/10

Novelty7/10

Dynamics Stability Accuracy Generalization Rnn Ssm Training dynamics Regularization Control theory Dynamical systems Linear algebra Spectral theory

Source paper: On the Number of Observation Nodes in Recurrent Neural Networks with Linear Threshold and ReLU Functions arXiv:2608.29650 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add an observability objective to an RNN so that a finite trajectory of selected hidden coordinates preserves information about the initial hidden state. The regularizer maximizes the smallest singular value or log determinant of the finite-horizon observation Jacobian, counteracting ReLU activation masks that erase hidden-state directions.

Formulas

xₜ₊₁ = [W xₜ + U uₜ + b]₊, yₜ = C xₜ

Dₜ = diag(1{W xₜ + U uₜ + b > 0}), P₀ = I, Pₜ₊₁ = Dₜ W Pₜ

J_T = [C P₀; C P₁; …; C P_T], G_T = J_Tᵀ J_T + εI

L_obs = −log det(G_T), m ≥ n/2 for the predicted global-observability threshold

Mathematical statement

Let the recurrent state be xₜ ∈ Rⁿ with dynamics xₜ₊₁ = [W xₜ + U uₜ + b]₊, where W and U are parameter matrices, uₜ is the input, b is a bias, and [·]₊ is componentwise ReLU. Let yₜ = C xₜ, where C ∈ Rᵐˣⁿ selects m observed coordinates. Define the finite-horizon observation map O_T(x₀) = [y₀; y₁; …; y_T]. Local observability requires the Jacobian J_T = ∂O_T/∂x₀ to have rank n. With Dₜ = diag(1{W xₜ + U uₜ + b > 0}), the state-transition Jacobian is P₀ = I and Pₜ₊₁ = Dₜ W Pₜ, so J_T is the vertical stack of C P₀ through C P_T. The Gramian G_T = J_Tᵀ J_T + εI is well-conditioned when its smallest eigenvalue is large. The paper predicts that unrestricted real-valued ReLU networks require at least m ≥ n/2 observation nodes for global finite-horizon observability.

Implementation notes

Integration point: use a vanilla ReLU RNN or a recurrent state-space model with transition xₜ₊₁ = [W xₜ + U uₜ + b]₊. Choose C as a fixed coordinate-selection matrix, or learn a soft selection matrix and discretize it after training. During each training batch, unroll a prefix of length T while retaining x₀. Pseudocode: set P = I; store J_blocks = [C]; for each step compute z = W x + U u + b, D = diag(z > 0), update x = relu(z), update P = D W P, and append C P to J_blocks. Concatenate the blocks into J, form G = JᵀJ + εI, and optimize task_loss + λ times negative_logdet(G). For large n, estimate the smallest eigenvalue with Lanczos rather than constructing the full Jacobian. The paper supplies the mask-product Jacobian and the m ≥ n/2 lower-bound mechanism; the implementation estimates finite-batch Jacobians and conditioning empirically. First experiment: use an n = 16 ReLU RNN on permuted sequential MNIST or the copy-memory task, compare m = 2, 4, 7, 8, and 12, with and without the regularizer. Train an auxiliary decoder from y₀:T to reconstruct x₀. The quantitative prediction is a transition near m = 8 = n/2: below eight observed coordinates, increasing T should leave substantial hidden-state collisions, whereas at or above eight the smallest singular value and reconstruction accuracy should increase sharply. Measure whether the empirical transition lies within one observation node of n/2.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built the finite-horizon ReLU observability Jacobian, Gramian metrics, exact autograd check, dimension sweep, horizon sweep, and a small log-det optimization control. The mask-product formula is numerically correct; full-rank onset occurred at m=9 (predicted counting boundary m=8) and T=4 for m=4 (predicted T=3), both within one of the theoretical thresholds. The regularizer raised logdet from -25.53 to -17.04 at m=9, but the smallest singular value remained near zero and numerical rank did not improve, so it is a promising conditioning signal rather than a demonstrated robust observability win.

Agent confidence: 7/10

Baseline: m=9 fixed control: logdet -25.53, smin 6.86e-10, rank 15
Idea: m=9 log-det optimization: logdet -17.04, smin 2.56e-8, rank 14; autograd Jacobian max error 9.54e-7; observed boundaries m=9 and T=4

Limitations: No sequential MNIST/copy-memory task, auxiliary decoder, FLOP/speed study, multi-seed statistics, learned observation matrix, or comparison against a trained vanilla RNN was tested. The optimization comparison is a small synthetic parameter-only control, and log-det can improve through singular-value scaling without eliminating rank deficiency.

How to run: python3 observability_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.29650) Sep 1, 2026

No significant difference vs. the tuned baseline (+0.0%, p=0.4578, wins 3 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0199
Idea mean: 0.0199
Effect (Δ): 0 (+0.0%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.4578 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.1%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

The final run used a reduced equal budget after the initial configuration exceeded the runtime limit: 3 epochs, 200 training and 200 test examples, and a short T=2 regularization horizon. The regularizer was applied to one representative trajectory only on the final minibatch of each epoch. Longer horizons, trajectory averaging, auxiliary initial-state reconstruction, learned observation matrices, and other registered tracks were not tested.

How to run: python3 observability_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

The matched dynamics benchmark tested the observability-regularized rnn_small against the same architecture and tuned baseline across 8 paired seeds. The idea had test MSE 0.0199446993 versus baseline 0.0199376683, paired delta +0.0000070310 and permutation p=0.45785, so it produced no significant improvement. The trained-model mechanism signature confirmed the counting-rank prediction, but this did not translate into a task-metric win.

Baseline: best lr=0.006; sweep means: lr 0.0015 -> 0.3385223672, lr 0.003 -> 0.0249988447, lr 0.006 -> 0.0168350947; full 8-seed mean MSE 0.0199376683
Comparison: delta_mean=+0.00000703096, p_value=0.45785, idea_wins=3/8, verdict=no measurable effect
Idea: idea sweep means: lr 0.0015 -> 0.3223300446, lr 0.003 -> 0.0323592271, lr 0.006 -> 0.0199446993; best full 8-seed mean MSE 0.0199446993

Benchmark result