# Masked Observability Preconditioner

- ID: 2026
- Canonical URL: https://synthcore.org/idea/2026/masked-observability-preconditioner
- API JSON: https://synthcore.org/api/idea/2026.json
- API Markdown: https://synthcore.org/api/idea/2026.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.09030](https://arxiv.org/abs/2608.09030)
- Category: dynamics
- Solves: stability, generalization
- ML areas: optimizer, training-dynamics, mlp, transformer
- Math tags: control-theory, dynamical-systems, linear-algebra, optimization
- Ratings: usefulness 8/10; difficulty 6/10; novelty 6/10

## Idea description

Replace the ordinary gradient step by an update preconditioned by parameter directions actually excited by the observed part of the input. In a neural network, approximate this geometry with a masked Jacobian Gramian and damp directions with low observability, preventing arbitrary drift of parameters associated with missing features.

## Mathematical statement

For parameters \(\theta\in\mathbb{R}^{p}\), input \(x_t\), binary feature mask \(M_t\), and masked input \(\tilde{x}_t=M_tx_t\), use the preconditioned update \(\theta_{t+1}=\theta_t-\eta B_t^{-1}g_t\), where \(g_t=\nabla_\theta\ell_t(f_\theta(\tilde{x}_t),y_t)\), \(\eta>0\) is the learning rate, and \(B_t\succ0\) defines update geometry. Let \(J_t=\partial f_\theta(M_tx_t)/\partial\theta\) and \(G_t=J_t^\top J_t\). Define \(B_t=\lambda I+\bar G_t\), where \(\lambda>0\) and \(\bar G_t\) is an exponentially averaged masked Gramian. The relevant recurrent-excitation condition is \(\sum_{i=t}^{t+W-1}G_i\succeq\alpha I\) on the identifiable parameter subspace, for window length \(W\) and excitation level \(\alpha>0\). Observable directions can contract under this condition, while regularization bounds unexcited directions.

## Key formulas

- $$\theta_{t+1}=\theta_t-\eta B_t^{-1}g_t,\qquad g_t=\nabla_\theta\ell_t\big(f_\theta(M_tx_t),y_t\big).$$
- $$J_t=\frac{\partial f_\theta(M_tx_t)}{\partial\theta},\qquad G_t=J_t^\top J_t,\qquad \bar G_t=\beta\bar G_{t-1}+(1-\beta)G_t,$$
- $$B_t=\lambda I+\bar G_t,\qquad \sum_{i=t}^{t+W-1}G_i\succeq\alpha I\quad\Longrightarrow\quad\text{persistent corrective excitation on identifiable directions}.$$
- $$\|B_t^{-1}g_t\|\leq\frac{\|g_t\|}{\lambda},\qquad \eta<\frac{2\lambda_{\min}(B_t)}{\lambda_{\max}(H_t)},$$

## Implementation notes

Integrate this at the optimizer step for a small MLP or transformer block, initially using a diagonal or low-rank approximation because a full parameter Gramian is infeasible. At each minibatch, apply the feature mask before the forward pass, compute the loss gradient \(g_t\), and estimate masked sensitivity. The cheapest estimator is diagonal: accumulate \(v_t=\beta v_{t-1}+(1-\beta)\operatorname{diag}(J_t^\top J_t)\), set \(B_t=\operatorname{diag}(\lambda+v_t)\), and update \(\theta\leftarrow\theta-\eta g_t/(\lambda+v_t)\). A stronger version uses rank-8 Nyström or K-FAC approximations. Pseudocode is: mask inputs; run forward and loss; backpropagate \(g\); estimate \(G\) using Jacobian-vector products; update \(\bar G\); solve \(Bu=g\); update \(\theta\leftarrow\theta-\eta u\). The paper supplies the observability interpretation, recurrent-excitation requirement, and residual-to-state logic; \(\alpha\), \(W\), \(\lambda\), and the effective Hessian bound must be estimated empirically. First test a two-layer MLP on MNIST with 50-95% structured feature masking, comparing SGD, Adam, and this method. Log parameter drift in persistently masked coordinates, loss spikes after mask changes, and the largest stable learning rate. The prediction is that update variance in weakly observed directions is reduced by approximately \(\lambda/(\lambda+v_j)\), while error in non-excited directions remains bounded instead of growing.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a diagonal masked-observability preconditioner and reproducible toy/MLP experiment in experiment.py. The mechanism manifested: the norm bound held (maximum observed ratio 0.848 <= 1), predicted contraction factors matched measured values to numerical precision, and observed stability boundaries were within about 1% of 2(lambda+q)/q. Excitation-dependent noise drift also fell strongly (mean theta^2 499.0 at q=0, 44.5 at q=1, 13.7 at q=4); however, the MLP did not beat Adam on validation MSE.

### Mechanism check

- Verdict: Built a diagonal masked-observability preconditioner and reproducible toy/MLP experiment in experiment.py. The mechanism manifested: the norm bound held (maximum observed ratio 0.848 <= 1), predicted contraction factors matched measured values to numerical precision, and observed stability boundaries were within about 1% of 2(lambda+q)/q. Excitation-dependent noise drift also fell strongly (mean theta^2 499.0 at q=0, 44.5 at q=1, 13.7 at q=4); however, the MLP did not beat Adam on validation MSE.
- Confidence: 9/10
- Limitations: Only a small synthetic regression MLP was tested, using a diagonal sensitivity proxy rather than exact Jacobian Gramians or a full neural-network training benchmark. No MNIST, structured-mask generalization study, multiple random seeds, FLOP/speed profiling, rank-8 approximation, or persistent-coordinate drift decomposition was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.60105
- Paired wins: 3/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_mop.py](https://synthcore.org/code/766/bench_mop.py)
- [bench_report.json](https://synthcore.org/code/766/bench_report.json)
- [experiment.py](https://synthcore.org/code/766/experiment.py)
- [report.md](https://synthcore.org/code/766/report.md)
- [report_bench_2026-09-01T101059.md](https://synthcore.org/code/766/report_bench_2026-09-01T101059.md)
- [results.json](https://synthcore.org/code/766/results.json)
- [Download all files as ZIP](https://synthcore.org/download/766)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.