# Empirical Preimage-Entropy Regularization

- ID: 2984
- Canonical URL: https://synthcore.org/idea/2984/empirical-preimage-entropy-regularization
- API JSON: https://synthcore.org/api/idea/2984.json
- API Markdown: https://synthcore.org/api/idea/2984.md
- Verification status: unverified
- Source: [arXiv:2609.00655](https://arxiv.org/abs/2609.00655)
- Category: dynamics
- Solves: stability, generalization, accuracy
- ML areas: rnn, ssm, world-model, regularization
- Math tags: dynamical-systems, information-theory, statistics, optimization
- Ratings: usefulness 6/10; difficulty 7/10; novelty 9/10

## Idea description

Apply the paper's empirical preimage-entropy construction to a learned recurrent transition map, penalizing excessive distinguishable hidden-state histories that produce the same current state while preserving multiple histories when the task requires genuine multimodality. Unlike a raw inverse-Jacobian penalty, the regularizer is computed only among inverse trajectories having similar empirical state distributions, so it distinguishes useful multimodal memory from uncontrolled branch explosion.

## Mathematical statement

Let $F_\theta:\mathcal H\to\mathcal H$ be a continuous learned hidden-state map on compact or numerically bounded state space $(\mathcal H,d)$, let $F_\theta^{-n}(y)$ be the set of states $x$ satisfying $F_\theta^n(x)=y$, and let $\hat\mu_n(x)=n^{-1}\sum_{t=0}^{n-1}\delta_{F_\theta^t(x)}$ be the empirical measure of an inverse trajectory. For a target state $y$, tolerance $\varepsilon$, and weak-metric neighborhood $U(\mu)$ around an invariant measure $\mu$, define $S_n(y,\varepsilon,U(\mu))$ as the largest subset of $F_\theta^{-n}(y)$ whose empirical measures lie in $U(\mu)$ and whose length-$n$ trajectories are pairwise $(n,\varepsilon)$-separated, meaning $\max_{0\le t<n}d(F_\theta^t(x),F_\theta^t(x'))>\varepsilon$. The empirical preimage entropy is $h^*_{p,\mu}(F_\theta)=\lim_{U\downarrow\{\mu\}}\lim_{\varepsilon\downarrow0}\limsup_{n\to\infty}n^{-1}\log\sup_y S_n(y,\varepsilon,U(\mu))$. The paper's key result is $h_p(F_\theta)=\sup_{\mu\in\mathcal M_{F_\theta}}h^*_{p,\mu}(F_\theta)$, with an analogous statement for the spanning/multiplicity entropy $h_m$. Use a finite-horizon soft estimate of $\log K_n/n$, where $K_n$ is the number of separated inverse trajectories found by a particle-and-optimization search, as the neural regularizer.

## Key formulas

- $$h_p(F_\theta)=\sup_{\mu\in\mathcal M_{F_\theta}}h^*_{p,\mu}(F_\theta),\qquad h_m(F_\theta)=\sup_{\mu\in\mathcal M_{F_\theta}}h^*_{m,\mu}(F_\theta).$$
- $$\hat\mu_n(x)=\frac1n\sum_{t=0}^{n-1}\delta_{F_\theta^t(x)},\qquad K_n(y;\varepsilon,U)=\max\left\{|A|:A\subset F_\theta^{-n}(y),\ \hat\mu_n(x)\in U,\ \max_{t<n}d(F_\theta^t(x),F_\theta^t(x'))>\varepsilon\ \forall x\neq x'\right\}.$$
- $$\widehat h_{\mathrm{pre}}(\theta)=\frac{1}{n}\log\left(1+\sum_{c=1}^{C}K_n(y;\varepsilon,U_c)\right),\qquad L(\theta)=L_{\mathrm{task}}(\theta)+\lambda\,[\widehat h_{\mathrm{pre}}(\theta)-\tau]_+^2,$$
- $$K_n(y;\varepsilon,U_c)\approx\max\left\{K:\ d_n(x_i,x_j)>\varepsilon,\ \rho(\hat\mu_n(x_i),\mu_c)<r\right\},\qquad d_n(x,x')=\max_{0\le t<n}d(F_\theta^t(x),F_\theta^t(x')).$$

## Implementation notes

1. Integration point: use this on an RNN, neural state-space model, or deterministic world model with transition $h_{t+1}=F_\theta(h_t,u_t)$; start with fixed input $u_t=0$ or a recorded input sequence and regularize only the hidden-state transition. At each training checkpoint, sample target states $y$ from replayed trajectories and search for approximate inverse histories $x_0$ satisfying $F_\theta^n(x_0;u_{0:n-1})\approx y$. 2. Pseudocode: sample $y$, draw $M$ initial candidates $x_0^j$; optimize each candidate with Adam on $\|F_\theta^n(x_0^j)-y\|^2$; discard residuals above tolerance; roll out every surviving candidate and compute $\hat\mu_n(x_0^j)$; cluster empirical measures into bins $U_c$ using Wasserstein distance or vectors of state moments; within each bin greedily retain candidates separated by $d_n>\varepsilon$; set $K_c$ to the retained count and add $\lambda[ n^{-1}\log(1+\sum_cK_c)-\tau]_+^2$ to the task loss. Stop gradients through the discrete count for the first MVP, or replace the count by a differentiable soft packing score $\sum_{i<j}\sigma((d_n(x_i,x_j)-\varepsilon)/T)$. 3. Computed from the paper: empirical-measure conditioning, separated inverse fibers, exponential rate $n^{-1}\log K_n$, and the supremum-over-measures interpretation. Estimated empirically: inverse roots, invariant measures, separation counts, and finite-horizon entropy. 4. First cheap experiment: train a tanh GRU on sequential MNIST or a synthetic delayed-bit task, comparing baseline training, a raw inverse-residual penalty, and this regularizer. Use $n=8$ or $16$, $M=64$, $C=4$, and sweep $\lambda$. The quantitative prediction is that plots of $n^{-1}\log K_n(y;\varepsilon,U_c)$ versus $n$ will show approximately linear growth within empirical-measure bins, with slope depending strongly on $c$; the global slope should be within roughly 20% of the largest bin slope. Increasing $\lambda$ should reduce the maximal slope toward the target $\tau$ without eliminating task-required branches. A second falsifiable signature is a sharp degradation or saturation when $\varepsilon$ exceeds the typical hidden-state separation scale, revealing the finite-resolution analogue of the entropy limit.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.