△ Mechanism confirmed, baseline not beaten 2026

Masked Observability Preconditioner

Implementation & benchmark of arXiv:2608.09030 — Closing the loop in learning with missing data

Usefulness8/10

Difficulty6/10

Novelty6/10

Dynamics Stability Generalization Optimizer Training dynamics Mlp Transformer Control theory Dynamical systems Linear algebra Optimization

Source paper: Closing the loop in learning with missing data arXiv:2608.09030 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the ordinary gradient step by an update preconditioned by parameter directions actually excited by the observed part of the input. In a neural network, approximate this geometry with a masked Jacobian Gramian and damp directions with low observability, preventing arbitrary drift of parameters associated with missing features.

Formulas

$$\theta_{t+1}=\theta_t-\eta B_t^{-1}g_t,\qquad g_t=\nabla_\theta\ell_t\big(f_\theta(M_tx_t),y_t\big).$$

$$J_t=\frac{\partial f_\theta(M_tx_t)}{\partial\theta},\qquad G_t=J_t^\top J_t,\qquad \bar G_t=\beta\bar G_{t-1}+(1-\beta)G_t,$$

$$B_t=\lambda I+\bar G_t,\qquad \sum_{i=t}^{t+W-1}G_i\succeq\alpha I\quad\Longrightarrow\quad\text{persistent corrective excitation on identifiable directions}.$$

$$\|B_t^{-1}g_t\|\leq\frac{\|g_t\|}{\lambda},\qquad \eta<\frac{2\lambda_{\min}(B_t)}{\lambda_{\max}(H_t)},$$

Mathematical statement

For parameters $\theta\in\mathbb{R}^{p}$, input $x_t$, binary feature mask $M_t$, and masked input $\tilde{x}_t=M_tx_t$, use the preconditioned update $\theta_{t+1}=\theta_t-\eta B_t^{-1}g_t$, where $g_t=\nabla_\theta\ell_t(f_\theta(\tilde{x}_t),y_t)$, $\eta>0$ is the learning rate, and $B_t\succ0$ defines update geometry. Let $J_t=\partial f_\theta(M_tx_t)/\partial\theta$ and $G_t=J_t^\top J_t$. Define $B_t=\lambda I+\bar G_t$, where $\lambda>0$ and $\bar G_t$ is an exponentially averaged masked Gramian. The relevant recurrent-excitation condition is $\sum_{i=t}^{t+W-1}G_i\succeq\alpha I$ on the identifiable parameter subspace, for window length $W$ and excitation level $\alpha>0$. Observable directions can contract under this condition, while regularization bounds unexcited directions.

Implementation notes

Integrate this at the optimizer step for a small MLP or transformer block, initially using a diagonal or low-rank approximation because a full parameter Gramian is infeasible. At each minibatch, apply the feature mask before the forward pass, compute the loss gradient $g_t$, and estimate masked sensitivity. The cheapest estimator is diagonal: accumulate $v_t=\beta v_{t-1}+(1-\beta)\operatorname{diag}(J_t^\top J_t)$, set $B_t=\operatorname{diag}(\lambda+v_t)$, and update $\theta\leftarrow\theta-\eta g_t/(\lambda+v_t)$. A stronger version uses rank-8 Nyström or K-FAC approximations. Pseudocode is: mask inputs; run forward and loss; backpropagate $g$; estimate $G$ using Jacobian-vector products; update $\bar G$; solve $Bu=g$; update $\theta\leftarrow\theta-\eta u$. The paper supplies the observability interpretation, recurrent-excitation requirement, and residual-to-state logic; $\alpha$, $W$, $\lambda$, and the effective Hessian bound must be estimated empirically. First test a two-layer MLP on MNIST with 50-95% structured feature masking, comparing SGD, Adam, and this method. Log parameter drift in persistently masked coordinates, loss spikes after mask changes, and the largest stable learning rate. The prediction is that update variance in weakly observed directions is reduced by approximately $\lambda/(\lambda+v_j)$, while error in non-excited directions remains bounded instead of growing.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a diagonal masked-observability preconditioner and reproducible toy/MLP experiment in experiment.py. The mechanism manifested: the norm bound held (maximum observed ratio 0.848 <= 1), predicted contraction factors matched measured values to numerical precision, and observed stability boundaries were within about 1% of 2(lambda+q)/q. Excitation-dependent noise drift also fell strongly (mean theta^2 499.0 at q=0, 44.5 at q=1, 13.7 at q=4); however, the MLP did not beat Adam on validation MSE.

Agent confidence: 9/10

Baseline: SGD: validation MSE 0.08090, parameter drift 1.332; Adam: validation MSE 0.06873, parameter drift 4.006.
Idea: Masked diagonal preconditioner: validation MSE 0.07988, parameter drift 1.324; it slightly beat SGD and substantially reduced drift versus Adam, but was worse than Adam in MSE. Toy bound ratio=0.848, contraction predicted/measured=[0.9619/0.9619, 0.8667/0.8667, 0.6000/0.6000, 0.3600/0.3600]. Stability predicted/observed=[42.0/42.30, 12.0/12.05, 4.0/3.98, 2.5/2.45].

Limitations: Only a small synthetic regression MLP was tested, using a diagonal sensitivity proxy rather than exact Jacobian Gramians or a full neural-network training benchmark. No MNIST, structured-mask generalization study, multiple random seeds, FLOP/speed profiling, rank-8 approximation, or persistent-coordinate drift decomposition was evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.09030) Sep 2, 2026

Ahead of the baseline (−15.6%) but not statistically significant (p=0.601, wins 3 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 53.962
Idea mean: 45.5441
Effect (Δ): -8.4179 (−15.6%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.601 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 0 configs

Limitations:

Only the structurally matched tabular track was tested. The implementation uses a diagonal per-sample Jacobian Gramian rather than a full or low-rank Gramian; the run used 400 training samples, 100 test samples, 12 epochs, batch size 128, and CPU execution. The idea candidate list included a duplicated selected setting, although all tried learning rates were included in the baseline sweep union.

How to run: python3 bench_mop.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the diagonal Masked Observability Preconditioner as a masked-Jacobian optimizer update on the structurally matched tabular Friedman regression benchmark with the shared mlp_tiny architecture. The trained-model mechanism prediction was confirmed: predicted versus observed update factors were 0.85495 versus 0.85444, a 0.06% relative error. The idea's test MSE was lower on average, but the paired permutation test was not significant, so this is not a benchmark win.

Baseline: Adam, best lr=0.002 and weight_decay=0.0001; mean test MSE 53.96195 over 8 seeds.
Comparison: delta_mean=-8.41789 MSE, 3/8 paired wins, permutation p=0.60105, verdict=no significant win
Idea: Masked diagonal preconditioner, lr=0.002 and weight_decay=0.0001; mean test MSE 45.54407 over 8 seeds.

Benchmark result