# Finite-Horizon Hidden-State Observability Regularizer - ID: 2843 - Canonical URL: https://synthcore.org/idea/2843/finite-horizon-hidden-state-observability-regularizer - API JSON: https://synthcore.org/api/idea/2843.json - API Markdown: https://synthcore.org/api/idea/2843.md - Verification status: failed_benchmark - Source: [arXiv:2608.29650](https://arxiv.org/abs/2608.29650) - Category: dynamics - Solves: stability, accuracy, generalization - ML areas: rnn, ssm, training-dynamics, regularization - Math tags: control-theory, dynamical-systems, linear-algebra, spectral-theory - Ratings: usefulness 8/10; difficulty 6/10; novelty 7/10 ## Idea description Add an observability objective to an RNN so that a finite trajectory of selected hidden coordinates preserves information about the initial hidden state. The regularizer maximizes the smallest singular value or log determinant of the finite-horizon observation Jacobian, counteracting ReLU activation masks that erase hidden-state directions. ## Mathematical statement Let the recurrent state be xₜ ∈ Rⁿ with dynamics xₜ₊₁ = [W xₜ + U uₜ + b]₊, where W and U are parameter matrices, uₜ is the input, b is a bias, and [·]₊ is componentwise ReLU. Let yₜ = C xₜ, where C ∈ Rᵐˣⁿ selects m observed coordinates. Define the finite-horizon observation map O_T(x₀) = [y₀; y₁; …; y_T]. Local observability requires the Jacobian J_T = ∂O_T/∂x₀ to have rank n. With Dₜ = diag(1{W xₜ + U uₜ + b > 0}), the state-transition Jacobian is P₀ = I and Pₜ₊₁ = Dₜ W Pₜ, so J_T is the vertical stack of C P₀ through C P_T. The Gramian G_T = J_Tᵀ J_T + εI is well-conditioned when its smallest eigenvalue is large. The paper predicts that unrestricted real-valued ReLU networks require at least m ≥ n/2 observation nodes for global finite-horizon observability. ## Key formulas - xₜ₊₁ = [W xₜ + U uₜ + b]₊, yₜ = C xₜ - Dₜ = diag(1{W xₜ + U uₜ + b > 0}), P₀ = I, Pₜ₊₁ = Dₜ W Pₜ - J_T = [C P₀; C P₁; …; C P_T], G_T = J_Tᵀ J_T + εI - L_obs = −log det(G_T), m ≥ n/2 for the predicted global-observability threshold ## Implementation notes Integration point: use a vanilla ReLU RNN or a recurrent state-space model with transition xₜ₊₁ = [W xₜ + U uₜ + b]₊. Choose C as a fixed coordinate-selection matrix, or learn a soft selection matrix and discretize it after training. During each training batch, unroll a prefix of length T while retaining x₀. Pseudocode: set P = I; store J_blocks = [C]; for each step compute z = W x + U u + b, D = diag(z > 0), update x = relu(z), update P = D W P, and append C P to J_blocks. Concatenate the blocks into J, form G = JᵀJ + εI, and optimize task_loss + λ times negative_logdet(G). For large n, estimate the smallest eigenvalue with Lanczos rather than constructing the full Jacobian. The paper supplies the mask-product Jacobian and the m ≥ n/2 lower-bound mechanism; the implementation estimates finite-batch Jacobians and conditioning empirically. First experiment: use an n = 16 ReLU RNN on permuted sequential MNIST or the copy-memory task, compare m = 2, 4, 7, 8, and 12, with and without the regularizer. Train an auxiliary decoder from y₀:T to reconstruct x₀. The quantitative prediction is a transition near m = 8 = n/2: below eight observed coordinates, increasing T should leave substantial hidden-state collisions, whereas at or above eight the smallest singular value and reconstruction accuracy should increase sharply. Measure whether the empirical transition lies within one observation node of n/2. ## Verification - Status: failed_benchmark - Mechanism evidence: yes - Mechanism confirmed: no - Practical verdict: no_effect - Verdict: Built the finite-horizon ReLU observability Jacobian, Gramian metrics, exact autograd check, dimension sweep, horizon sweep, and a small log-det optimization control. The mask-product formula is numerically correct; full-rank onset occurred at m=9 (predicted counting boundary m=8) and T=4 for m=4 (predicted T=3), both within one of the theoretical thresholds. The regularizer raised logdet from -25.53 to -17.04 at m=9, but the smallest singular value remained near zero and numerical rank did not improve, so it is a promising conditioning signal rather than a demonstrated robust observability win. ### Mechanism check - Verdict: Built the finite-horizon ReLU observability Jacobian, Gramian metrics, exact autograd check, dimension sweep, horizon sweep, and a small log-det optimization control. The mask-product formula is numerically correct; full-rank onset occurred at m=9 (predicted counting boundary m=8) and T=4 for m=4 (predicted T=3), both within one of the theoretical thresholds. The regularizer raised logdet from -25.53 to -17.04 at m=9, but the smallest singular value remained near zero and numerical rank did not improve, so it is a promising conditioning signal rather than a demonstrated robust observability win. - Confidence: 7/10 - Limitations: No sequential MNIST/copy-memory task, auxiliary decoder, FLOP/speed study, multi-seed statistics, learned observation matrix, or comparison against a trained vanilla RNN was tested. The optimization comparison is a small synthetic parameter-only control, and log-det can improve through singular-value scaling without eliminating rank deficiency. ### Practical benchmark - Paired seeds: 8 - Baseline mean: 0 - Idea mean: 0 - p-value: 0.45785 - Paired wins: 3/8 - Benchmark verdict: no measurable effect ## Artifacts - [bench_report.json](https://synthcore.org/code/1068/bench_report.json) - [observability_bench.py](https://synthcore.org/code/1068/observability_bench.py) - [observability_experiment.py](https://synthcore.org/code/1068/observability_experiment.py) - [report.md](https://synthcore.org/code/1068/report.md) - [report_bench_2026-09-01T235430.md](https://synthcore.org/code/1068/report_bench_2026-09-01T235430.md) - [results.json](https://synthcore.org/code/1068/results.json) - [Download all files as ZIP](https://synthcore.org/download/1068) ## Disclaimer AI-generated research hypothesis, automatically tested. Not peer-reviewed.