✗ Failed on benchmark 2026

Fisher-Observable Latent State Training

Implementation & benchmark of arXiv:2608.16135 — Improving Observability of Relative Orbit Estimation Using Bearing Measurements and Light Curves

Usefulness7/10

Difficulty5/10

Novelty6/10

Regularization Stability Accuracy Sample efficiency Ssm Rnn World model Regularization Statistics Linear algebra Control theory Dynamical systems

Source paper: Improving Observability of Relative Orbit Estimation Using Bearing Measurements and Light Curves arXiv:2608.16135 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add an observability regularizer to a recurrent state-space model or world model so that short sequences of predicted multimodal observations identify the latent state. The regularizer penalizes poorly conditioned Fisher information, preventing the model from storing important state variables in directions that its available observations cannot distinguish.

Formulas

$$\mathcal I_0=\sum_{t=0}^{K}J_t^{\mathsf T}R_t^{-1}J_t+P_0^{-1},\qquad J_t=\frac{\partial h_\theta(z_t)}{\partial z_t}\frac{\partial z_t}{\partial z_0}.$$

$$\mathcal L_{\mathrm{obs}}=-\log\det(\mathcal I_0+\epsilon I)+\beta\,\frac{\lambda_{\max}(\mathcal I_0)}{\lambda_{\min}(\mathcal I_0)+\epsilon},$$

$$\mathcal L=\mathcal L_{\mathrm{task}}+\alpha\mathcal L_{\mathrm{obs}},\qquad \lambda_{\min}(\mathcal I_0)\uparrow\ \text{and}\ \kappa(\mathcal I_0)=\frac{\lambda_{\max}(\mathcal I_0)}{\lambda_{\min}(\mathcal I_0)}\downarrow\ \text{when the auxiliary channel is informative}.$$

Mathematical statement

Let the latent dynamics be $z_{t+1}=f_{\theta}(z_t,u_t)+\epsilon_t$, with latent state $z_t\in\mathbb{R}^d$, parameter vector $\theta$, input $u_t$, and process-noise covariance $Q_t$. Let the model produce observations $y_t=h_{\theta}(z_t)$ with observation-noise covariance $R_t\succ0$. For a rollout of length $K$, define the sensitivity matrix $J_t=\partial y_t/\partial z_0$, including the dynamics chain rule $J_t=(\partial h_\theta/\partial z_t)(\partial z_t/\partial z_0)$. The local Fisher information for the initial latent state is $\mathcal I_0=\sum_{t=0}^{K}J_t^{\mathsf T}R_t^{-1}J_t+P_0^{-1}$, where $P_0$ is the latent-state prior covariance. Maximizing $\log\det(\mathcal I_0+\epsilon I)$ increases joint observability; penalizing the condition number or the smallest eigenvalue prevents nearly unobservable directions. The transferable mechanism is that adding a measurement channel with a non-collinear Jacobian, such as photometric intensity with range sensitivity, increases Fisher information and improves estimator convergence.

Implementation notes

1. Integration point: use a deterministic or variational RSSM, GRU state-space model, or transformer world model with latent state $z_t$. The decoder should predict two channels: a standard observation channel and an auxiliary photometric scalar or vector. The auxiliary channel may be a real sensor modality, an image-derived brightness statistic, or a deliberately constructed proxy whose sensitivity depends on hidden scale, depth, energy, or object configuration. 2. Pseudocode: unroll $z_{t+1}=f_\theta(z_t,u_t)$ for $K=10$--$30$ steps; compute predicted observations $\hat y_t=h_\theta(z_t)$; obtain $J_t=\partial\hat y_t/\partial z_0$ using automatic differentiation or forward-mode Jacobian products; accumulate $I=\sum_tJ_t^TR_t^{-1}J_t+P_0^{-1}$; add $-\alpha\log\det(I+10^{-5}I_d)$ and optionally a clipped condition-number penalty to the task loss. Detach the state rollout only if memory is limiting, but retain derivatives through the observation Jacobian. 3. Computed from the paper's mechanism: Fisher accumulation, inverse noise weighting, and comparison of bearing-only versus bearing-plus-light-curve information. Estimated empirically: $R_t$, $P_0$, $\alpha$, and the latent dimension; estimate $R_t$ from validation residual covariance and normalize each observation channel before computing the Fisher matrix. 4. First cheap experiment: train a GRU latent predictor on a synthetic 2D constant-velocity tracking dataset with target state $(x,y,v_x,v_y)$, bearing $\phi=\operatorname{atan2}(y,x)$), and brightness $b=c/(x^2+y^2)+\nu_b$, while the baseline receives bearing only. Compare prediction loss and latent-state recovery against the same model without the regularizer. The quantitative prediction is that bearing-only training will show one or more small Fisher eigenvalues and high condition number, whereas adding brightness plus the regularizer should increase $\lambda_{\min}(\mathcal I_0)$ by at least 2x and reduce 20-step state error; the measured transition should occur when brightness noise becomes large enough that $\|R_b^{-1/2}J_b\|$ is below the bearing sensitivity.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a differentiable Fisher-information utility and synthetic constant-velocity bearing/brightness experiment in fisher_observable.py and run_experiment.py. The mechanism manifested: brightness lifted Fisher rank from 3 to 4 at K=1, reduced K=4 condition number from 2105 to 249, and improved noisy initial-state RMSE from 0.129 to 0.058. The Jacobian chain rule matched central differences to 3.1e-11; inverse-noise scaling was qualitatively confirmed, although the smallest-eigenvalue ratio was 3.30 versus the ideal 4.0.

Agent confidence: 8/10

Baseline: Bearing-only: rank 3 at K=8 and K=4 condition number 2104.83; state-recovery RMSE 0.12864.
Idea: Bearing-plus-brightness: full rank by K=1, K=4 condition number 248.73, lambda_min=34.41 at brightness sigma 0.01, state-recovery RMSE 0.05760; lambda_min ratio for sigma 0.01 versus 0.02 was 3.30 versus predicted 4.0.

Limitations: This MVP verifies the Fisher-observability mechanism and auxiliary-channel effect, but does not train a learned GRU/RSSM or demonstrate that the observability regularizer improves learned-model task loss. The transition sweep is deterministic and uses an idealized constant-velocity system, fixed noise levels, and one recovery trial rather than a multi-seed statistical study.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.16135) Sep 1, 2026

No significant difference vs. the tuned baseline (+12.6%, p=0.5164, wins 3 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0198
Idea mean: 0.0223
Effect (Δ): 0.0025 (+12.6%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.5164 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±42.8%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

The MVP used 4 epochs and 240 training/240 test samples after the full higher-order Jacobian version exceeded the runtime budget. The regularizer is a two-sample Fisher proxy over hidden trajectory sensitivity to the first two input state coordinates, not a full RSSM latent-state Fisher matrix; larger datasets, longer training, alternate auxiliary channels, and broader hyperparameter searches were not tested.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented and ran the Fisher observability regularizer on the registered dynamics track with the matched rnn_small architecture. The trained-model signature confirmed a small observability improvement, but the best idea test MSE was worse than baseline and the paired permutation test was non-significant.

Baseline: Mean test MSE 0.0198293367 over 8 seeds; tuned baseline LR=0.01 from sweep [0.001, 0.003, 0.01].
Idea: Best idea alpha=0.0001 at LR=0.01; mean test MSE 0.0223195173 over 8 seeds. Paired delta=+0.0024901805, p=0.5164, 3/8 wins; mechanism signature confirmed but task metric worsened.

Benchmark result