# Fisher-Observable Latent State Training

- ID: 2387
- Canonical URL: https://synthcore.org/idea/2387/fisher-observable-latent-state-training
- API JSON: https://synthcore.org/api/idea/2387.json
- API Markdown: https://synthcore.org/api/idea/2387.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.16135](https://arxiv.org/abs/2608.16135)
- Category: regularization
- Solves: stability, accuracy, sample-efficiency
- ML areas: ssm, rnn, world-model, regularization
- Math tags: statistics, linear-algebra, control-theory, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Add an observability regularizer to a recurrent state-space model or world model so that short sequences of predicted multimodal observations identify the latent state. The regularizer penalizes poorly conditioned Fisher information, preventing the model from storing important state variables in directions that its available observations cannot distinguish.

## Mathematical statement

Let the latent dynamics be $z_{t+1}=f_{\theta}(z_t,u_t)+\epsilon_t$, with latent state $z_t\in\mathbb{R}^d$, parameter vector $\theta$, input $u_t$, and process-noise covariance $Q_t$. Let the model produce observations $y_t=h_{\theta}(z_t)$ with observation-noise covariance $R_t\succ0$. For a rollout of length $K$, define the sensitivity matrix $J_t=\partial y_t/\partial z_0$, including the dynamics chain rule $J_t=(\partial h_\theta/\partial z_t)(\partial z_t/\partial z_0)$. The local Fisher information for the initial latent state is $\mathcal I_0=\sum_{t=0}^{K}J_t^{\mathsf T}R_t^{-1}J_t+P_0^{-1}$, where $P_0$ is the latent-state prior covariance. Maximizing $\log\det(\mathcal I_0+\epsilon I)$ increases joint observability; penalizing the condition number or the smallest eigenvalue prevents nearly unobservable directions. The transferable mechanism is that adding a measurement channel with a non-collinear Jacobian, such as photometric intensity with range sensitivity, increases Fisher information and improves estimator convergence.

## Key formulas

- $$\mathcal I_0=\sum_{t=0}^{K}J_t^{\mathsf T}R_t^{-1}J_t+P_0^{-1},\qquad J_t=\frac{\partial h_\theta(z_t)}{\partial z_t}\frac{\partial z_t}{\partial z_0}.$$
- $$\mathcal L_{\mathrm{obs}}=-\log\det(\mathcal I_0+\epsilon I)+\beta\,\frac{\lambda_{\max}(\mathcal I_0)}{\lambda_{\min}(\mathcal I_0)+\epsilon},$$
- $$\mathcal L=\mathcal L_{\mathrm{task}}+\alpha\mathcal L_{\mathrm{obs}},\qquad \lambda_{\min}(\mathcal I_0)\uparrow\ \text{and}\ \kappa(\mathcal I_0)=\frac{\lambda_{\max}(\mathcal I_0)}{\lambda_{\min}(\mathcal I_0)}\downarrow\ \text{when the auxiliary channel is informative}.$$

## Implementation notes

1. Integration point: use a deterministic or variational RSSM, GRU state-space model, or transformer world model with latent state $z_t$. The decoder should predict two channels: a standard observation channel and an auxiliary photometric scalar or vector. The auxiliary channel may be a real sensor modality, an image-derived brightness statistic, or a deliberately constructed proxy whose sensitivity depends on hidden scale, depth, energy, or object configuration. 2. Pseudocode: unroll $z_{t+1}=f_\theta(z_t,u_t)$ for $K=10$--$30$ steps; compute predicted observations $\hat y_t=h_\theta(z_t)$; obtain $J_t=\partial\hat y_t/\partial z_0$ using automatic differentiation or forward-mode Jacobian products; accumulate $I=\sum_tJ_t^TR_t^{-1}J_t+P_0^{-1}$; add $-\alpha\log\det(I+10^{-5}I_d)$ and optionally a clipped condition-number penalty to the task loss. Detach the state rollout only if memory is limiting, but retain derivatives through the observation Jacobian. 3. Computed from the paper's mechanism: Fisher accumulation, inverse noise weighting, and comparison of bearing-only versus bearing-plus-light-curve information. Estimated empirically: $R_t$, $P_0$, $\alpha$, and the latent dimension; estimate $R_t$ from validation residual covariance and normalize each observation channel before computing the Fisher matrix. 4. First cheap experiment: train a GRU latent predictor on a synthetic 2D constant-velocity tracking dataset with target state $(x,y,v_x,v_y)$, bearing $\phi=\operatorname{atan2}(y,x)$), and brightness $b=c/(x^2+y^2)+\nu_b$, while the baseline receives bearing only. Compare prediction loss and latent-state recovery against the same model without the regularizer. The quantitative prediction is that bearing-only training will show one or more small Fisher eigenvalues and high condition number, whereas adding brightness plus the regularizer should increase $\lambda_{\min}(\mathcal I_0)$ by at least 2x and reduce 20-step state error; the measured transition should occur when brightness noise becomes large enough that $\|R_b^{-1/2}J_b\|$ is below the bearing sensitivity.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a differentiable Fisher-information utility and synthetic constant-velocity bearing/brightness experiment in fisher_observable.py and run_experiment.py. The mechanism manifested: brightness lifted Fisher rank from 3 to 4 at K=1, reduced K=4 condition number from 2105 to 249, and improved noisy initial-state RMSE from 0.129 to 0.058. The Jacobian chain rule matched central differences to 3.1e-11; inverse-noise scaling was qualitatively confirmed, although the smallest-eigenvalue ratio was 3.30 versus the ideal 4.0.

### Mechanism check

- Verdict: Built a differentiable Fisher-information utility and synthetic constant-velocity bearing/brightness experiment in fisher_observable.py and run_experiment.py. The mechanism manifested: brightness lifted Fisher rank from 3 to 4 at K=1, reduced K=4 condition number from 2105 to 249, and improved noisy initial-state RMSE from 0.129 to 0.058. The Jacobian chain rule matched central differences to 3.1e-11; inverse-noise scaling was qualitatively confirmed, although the smallest-eigenvalue ratio was 3.30 versus the ideal 4.0.
- Confidence: 8/10
- Limitations: This MVP verifies the Fisher-observability mechanism and auxiliary-channel effect, but does not train a learned GRU/RSSM or demonstrate that the observability regularizer improves learned-model task loss. The transition sweep is deterministic and uses an idealized constant-velocity system, fixed noise levels, and one recovery trial rather than a multi-seed statistical study.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.5164
- Paired wins: 3/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/936/bench_report.json)
- [fisher_observable.py](https://synthcore.org/code/936/fisher_observable.py)
- [report.md](https://synthcore.org/code/936/report.md)
- [report_bench_2026-09-01T163504.md](https://synthcore.org/code/936/report_bench_2026-09-01T163504.md)
- [results.json](https://synthcore.org/code/936/results.json)
- [run_experiment.py](https://synthcore.org/code/936/run_experiment.py)
- [stage2_bench.py](https://synthcore.org/code/936/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/936)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.