# Positive-Regime Observable ReLU State Space

- ID: 2844
- Canonical URL: https://synthcore.org/idea/2844/positive-regime-observable-relu-state-space
- API JSON: https://synthcore.org/api/idea/2844.json
- API Markdown: https://synthcore.org/api/idea/2844.md
- Verification status: unverified
- Source: [arXiv:2608.29650](https://arxiv.org/abs/2608.29650)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, memory, regularization
- Math tags: control-theory, dynamical-systems, linear-algebra, spectral-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Constrain recurrent preactivations to remain nonnegative so that ReLU acts as the identity along realized trajectories. The hidden dynamics then admit a classical linear observability matrix, allowing principled hidden-coordinate selection and conditioning control instead of relying on potentially destructive activation masks.

## Mathematical statement

Consider xₜ₊₁ = [A xₜ + B uₜ + b]₊, where xₜ ∈ Rⁿ is the state, uₜ is the input, A and B are matrices, and b is a bias. If A xₜ + B uₜ + b is componentwise nonnegative for all relevant states and inputs, then ReLU is inactive and the dynamics become affine linear. Perturbations satisfy δxₜ₊₁ = A δxₜ. With observations yₜ = C xₜ, the finite-horizon observability matrix is Q_T = [C; CA; …; CAᵀ]. The initial state is observable when rank(Q_T) = n, and robustness is measured by the smallest singular value of Q_T or the condition number of Q_TᵀQ_T. A trainable positivity penalty is the squared norm of the negative preactivation, which encourages the regime in which the paper's exact reduction to linear observability applies.

## Key formulas

- xₜ₊₁ = [A xₜ + B uₜ + b]₊,    A xₜ + B uₜ + b ≥ 0 componentwise ⇒ xₜ₊₁ = A xₜ + B uₜ + b
- δxₜ₊₁ = A δxₜ,    Q_T = [C; CA; CA²; …; CAᵀ]
- rank(Q_T) = n,    σ_min(Q_T) > 0
- L_pos = Σₜ ||[−A xₜ − B uₜ − b]₊||₂²,    L_cond = −log det(Q_Tᵀ Q_T + εI)

## Implementation notes

Integration point: replace the hidden transition of a small recurrent state-space model with xₜ₊₁ = relu(A xₜ + B uₜ + b), and expose m selected hidden coordinates through yₜ = C xₜ or an auxiliary memory decoder. Add the positivity and observability losses to the task loss. Pseudocode: unroll the sequence and compute zₜ = A xₜ + B uₜ + b; accumulate the positivity penalty ||relu(−zₜ)||²; update xₜ₊₁ = relu(zₜ). Separately construct Q = [C; CA; …; CAᵀ] for a chosen horizon T, calculate QᵀQ + εI, and add its negative log determinant to the objective. Optionally constrain A with spectral normalization if bounded recurrent dynamics are also desired. The paper supplies the exact condition under which ReLU observability reduces to linear observability; the positivity rate and singular values are measured empirically. First experiment: use n = 16 on the copy-memory and adding-problem datasets, compare unconstrained ReLU, positive-penalty ReLU, and tanh RNNs with m = 4 observed coordinates. Evaluate horizons T = 1, 2, 4, 8, and 16. The prediction is that when the negative-preactivation fraction falls below 1%, reconstruction quality follows the linear criterion: rank deficiency produces σ_min(Q_T) ≈ 0 and high reconstruction error, while full rank yields decreasing error as T increases until the smallest singular value saturates. Report whether observed rank transitions match Q_T within numerical tolerance.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.