On the Number of Observation Nodes in Recurrent Neural Networks with Linear Threshold and ReLU Functions
arXiv:2608.29650
2026
Dynamics
2 ideas extracted · analyzed Sep 1, 2026
What the math gives to ML
The paper gives a transferable finite-horizon observability mechanism for recurrent networks. Its strongest result is that general real-valued ReLU dynamics can lose state information through activation masks, yielding a global lower bound of m ≥ n/2 observed hidden coordinates, while nonnegative ReLU trajectories reduce exactly to linear-system observability. A practical neural-network transfer is to regularize the finite-horizon observation Jacobian so that hidden-state information is preserved, and to use positivity constraints when a classical linear observability certificate is desirable.
Ideas from this paper
✗ Failed on benchmark
2026
Add an observability objective to an RNN so that a finite trajectory of selected hidden coordinates preserves information about the initial hidden state. The regularizer maximizes the smallest singular value or log determinant of the finite-horizon observation Jacobian, counteracting ReLU activation masks that erase hidden-state directions.
Useful8/10
Difficulty6/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Constrain recurrent preactivations to remain nonnegative so that ReLU acts as the identity along realized trajectories. The hidden dynamics then admit a classical linear observability matrix, allowing principled hidden-coordinate selection and conditioning control instead of relying on potentially destructive activation masks.
Useful7/10
Difficulty5/10
Novelty6/10