# Noise-Adaptive Instantaneous Information Regularization

- ID: 1486
- Canonical URL: https://synthcore.org/idea/1486/noise-adaptive-instantaneous-information-regularization
- API JSON: https://synthcore.org/api/idea/1486.json
- API Markdown: https://synthcore.org/api/idea/1486.md
- Verification status: unverified
- Source: [arXiv:2607.27299](https://arxiv.org/abs/2607.27299)
- Category: regularization
- Solves: stability, generalization, sample-efficiency
- ML areas: rnn, ssm, world-model, rl, regularization
- Math tags: information-theory, stochastic-processes, statistics, dynamical-systems, control-theory
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Train a recurrent or state-space neural model with an information regularizer that uses trajectory-dependent predictive information at low observation noise but switches toward instantaneous mutual information as sensor noise increases. The switch is driven by an online estimate of the relative reliability of transfer entropy and instantaneous dependence, rather than by a fixed hyperparameter. This should prevent noisy histories from forcing the latent state to memorize unreliable temporal details while preserving state variables that remain correlated with observations and control-relevant targets.

## Mathematical statement

Let $z_t$ be the neural latent state, $y_t$ the clean sensor variable, $u_t$ the control, and $\tilde y_t=y_t+\varepsilon_t$ the measured observation, with independent measurement noise $\varepsilon_t\sim\mathcal N(0,\sigma_\varepsilon^2I)$. Estimate the trajectory-dependent signal as transfer entropy $\widehat T_{y\to z}=\widehat I(z_{t+1};y_{\leq t}\mid z_{\leq t},u_{\leq t})$ and the instantaneous signal as $\widehat I_{\mathrm{inst}}=\widehat I(z_t;\tilde y_t)$ or $\widehat I(z_t;u_t)$ when control relevance is the objective. Define $r=\widehat T_{y\to z}/(\widehat I_{\mathrm{inst}}+\delta)$, where $\delta>0$ prevents division by zero. The adaptive mixture is $R=(1-\alpha)\widehat T_{y\to z}+\alpha\widehat I_{\mathrm{inst}}$, with $\alpha=\sigma((r_c-r)/\tau)$. Here $r_c$ is a calibrated crossover ratio, $\tau$ is a softness parameter, and $\sigma$ is the logistic function. The paper’s mechanism predicts that increasing measurement noise reduces the trajectory statistic more rapidly than the instantaneous statistic, causing $r$ to cross $r_c$ at a finite noise level.

## Key formulas

- $$\tilde y_t=y_t+\varepsilon_t,\qquad \varepsilon_t\sim\mathcal N(0,\sigma_\varepsilon^2 I),$$
- $$T_{y\to z}=I(z_{t+1};y_{\leq t}\mid z_{\leq t},u_{\leq t}),\qquad I_{\mathrm{inst}}=I(z_t;\tilde y_t),$$
- $$R_t=(1-\alpha_t)\widehat T_{y\to z}+\alpha_t\widehat I_{\mathrm{inst}},\qquad \alpha_t=\sigma\!\left(\frac{r_c-\widehat T_{y\to z}/(\widehat I_{\mathrm{inst}}+\delta)}{\tau}\right),$$
- $$\mathcal L=\mathcal L_{\mathrm{task}}-\lambda R_t,\qquad r(\sigma_\varepsilon^2)=\frac{\widehat T_{y\to z}(\sigma_\varepsilon^2)}{\widehat I_{\mathrm{inst}}(\sigma_\varepsilon^2)+\delta}.$$

## Implementation notes

Integrate the method into an RNN, GRU, neural state-space model, or recurrent policy after the latent update $z_{t+1}=f_\theta(z_t,\tilde y_t,u_t)$. Use minibatches of short sequences. Compute the task loss, such as next-observation prediction, reward prediction, or policy loss. In parallel, estimate the two information terms with differentiable variational bounds: use a conditional density-ratio or contrastive estimator for $I(z_{t+1};y_{\leq t}\mid z_{\leq t},u_{\leq t})$, and an InfoNCE or Gaussian cross-covariance estimator for $I(z_t;\tilde y_t)$. Maintain exponential moving averages of both estimates and their variances; use stop-gradient copies of $z$ when updating the estimators to reduce feedback instability. Every $K$ optimizer steps calculate $r=\widehat T/(\widehat I_{\mathrm{inst}}+\delta)$ and update $\alpha$ using the logistic rule. Optimize $\mathcal L=\mathcal L_{\mathrm{task}}-\lambda R_t$. The stochastic Langevin feedback model and qualitative crossover are inherited from the paper; the finite-sample estimators, latent regularizer, and logistic gate are adaptations. Estimate $r_c$ from a validation noise sweep rather than assuming it theoretically. The first cheap experiment should use a GRU on noisy Lorenz-63 or cart-pole observations for next-state prediction. Compare no regularizer, fixed transfer-entropy regularization, fixed instantaneous-MI regularization, and the adaptive method while sweeping $\sigma_\varepsilon^2$ over one decade. The quantitative prediction is that $\widehat T(\sigma_\varepsilon^2)$ declines more steeply than $\widehat I_{\mathrm{inst}}(\sigma_\varepsilon^2)$, producing a reproducible crossing satisfying $r(\sigma_*^2)=r_c$. The gate should transition from $\alpha\approx0$ below $\sigma_*^2$ to $\alpha\approx1$ above it. Test whether the measured crossover varies by less than 20% across seeds and whether adaptive regularization avoids the high-noise degradation of fixed trajectory regularization.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.