{
 "artifacts": null,
 "category": "dynamics",
 "description": "Apply the paper's empirical preimage-entropy construction to a learned recurrent transition map, penalizing excessive distinguishable hidden-state histories that produce the same current state while preserving multiple histories when the task requires genuine multimodality. Unlike a raw inverse-Jacobian penalty, the regularizer is computed only among inverse trajectories having similar empirical state distributions, so it distinguishes useful multimodal memory from uncontrolled branch explosion.",
 "formulas_latex": [
  "$$h_p(F_\\theta)=\\sup_{\\mu\\in\\mathcal M_{F_\\theta}}h^*_{p,\\mu}(F_\\theta),\\qquad h_m(F_\\theta)=\\sup_{\\mu\\in\\mathcal M_{F_\\theta}}h^*_{m,\\mu}(F_\\theta).$$",
  "$$\\hat\\mu_n(x)=\\frac1n\\sum_{t=0}^{n-1}\\delta_{F_\\theta^t(x)},\\qquad K_n(y;\\varepsilon,U)=\\max\\left\\{|A|:A\\subset F_\\theta^{-n}(y),\\ \\hat\\mu_n(x)\\in U,\\ \\max_{t\u003cn}d(F_\\theta^t(x),F_\\theta^t(x'))\u003e\\varepsilon\\ \\forall x\\neq x'\\right\\}.$$",
  "$$\\widehat h_{\\mathrm{pre}}(\\theta)=\\frac{1}{n}\\log\\left(1+\\sum_{c=1}^{C}K_n(y;\\varepsilon,U_c)\\right),\\qquad L(\\theta)=L_{\\mathrm{task}}(\\theta)+\\lambda\\,[\\widehat h_{\\mathrm{pre}}(\\theta)-\\tau]_+^2,$$",
  "$$K_n(y;\\varepsilon,U_c)\\approx\\max\\left\\{K:\\ d_n(x_i,x_j)\u003e\\varepsilon,\\ \\rho(\\hat\\mu_n(x_i),\\mu_c)\u003cr\\right\\},\\qquad d_n(x,x')=\\max_{0\\le t\u003cn}d(F_\\theta^t(x),F_\\theta^t(x')).$$"
 ],
 "id": 2984,
 "implementation": "1. Integration point: use this on an RNN, neural state-space model, or deterministic world model with transition $h_{t+1}=F_\\theta(h_t,u_t)$; start with fixed input $u_t=0$ or a recorded input sequence and regularize only the hidden-state transition. At each training checkpoint, sample target states $y$ from replayed trajectories and search for approximate inverse histories $x_0$ satisfying $F_\\theta^n(x_0;u_{0:n-1})\\approx y$. 2. Pseudocode: sample $y$, draw $M$ initial candidates $x_0^j$; optimize each candidate with Adam on $\\|F_\\theta^n(x_0^j)-y\\|^2$; discard residuals above tolerance; roll out every surviving candidate and compute $\\hat\\mu_n(x_0^j)$; cluster empirical measures into bins $U_c$ using Wasserstein distance or vectors of state moments; within each bin greedily retain candidates separated by $d_n\u003e\\varepsilon$; set $K_c$ to the retained count and add $\\lambda[ n^{-1}\\log(1+\\sum_cK_c)-\\tau]_+^2$ to the task loss. Stop gradients through the discrete count for the first MVP, or replace the count by a differentiable soft packing score $\\sum_{i\u003cj}\\sigma((d_n(x_i,x_j)-\\varepsilon)/T)$. 3. Computed from the paper: empirical-measure conditioning, separated inverse fibers, exponential rate $n^{-1}\\log K_n$, and the supremum-over-measures interpretation. Estimated empirically: inverse roots, invariant measures, separation counts, and finite-horizon entropy. 4. First cheap experiment: train a tanh GRU on sequential MNIST or a synthetic delayed-bit task, comparing baseline training, a raw inverse-residual penalty, and this regularizer. Use $n=8$ or $16$, $M=64$, $C=4$, and sweep $\\lambda$. The quantitative prediction is that plots of $n^{-1}\\log K_n(y;\\varepsilon,U_c)$ versus $n$ will show approximately linear growth within empirical-measure bins, with slope depending strongly on $c$; the global slope should be within roughly 20% of the largest bin slope. Increasing $\\lambda$ should reduce the maximal slope toward the target $\\tau$ without eliminating task-required branches. A second falsifiable signature is a sharp degradation or saturation when $\\varepsilon$ exceeds the typical hidden-state separation scale, revealing the finite-resolution analogue of the entropy limit.",
 "math_summary": "Let $F_\\theta:\\mathcal H\\to\\mathcal H$ be a continuous learned hidden-state map on compact or numerically bounded state space $(\\mathcal H,d)$, let $F_\\theta^{-n}(y)$ be the set of states $x$ satisfying $F_\\theta^n(x)=y$, and let $\\hat\\mu_n(x)=n^{-1}\\sum_{t=0}^{n-1}\\delta_{F_\\theta^t(x)}$ be the empirical measure of an inverse trajectory. For a target state $y$, tolerance $\\varepsilon$, and weak-metric neighborhood $U(\\mu)$ around an invariant measure $\\mu$, define $S_n(y,\\varepsilon,U(\\mu))$ as the largest subset of $F_\\theta^{-n}(y)$ whose empirical measures lie in $U(\\mu)$ and whose length-$n$ trajectories are pairwise $(n,\\varepsilon)$-separated, meaning $\\max_{0\\le t\u003cn}d(F_\\theta^t(x),F_\\theta^t(x'))\u003e\\varepsilon$. The empirical preimage entropy is $h^*_{p,\\mu}(F_\\theta)=\\lim_{U\\downarrow\\{\\mu\\}}\\lim_{\\varepsilon\\downarrow0}\\limsup_{n\\to\\infty}n^{-1}\\log\\sup_y S_n(y,\\varepsilon,U(\\mu))$. The paper's key result is $h_p(F_\\theta)=\\sup_{\\mu\\in\\mathcal M_{F_\\theta}}h^*_{p,\\mu}(F_\\theta)$, with an analogous statement for the spanning/multiplicity entropy $h_m$. Use a finite-horizon soft estimate of $\\log K_n/n$, where $K_n$ is the number of separated inverse trajectories found by a particle-and-optimization search, as the neural regularizer.",
 "math_tags": [
  "dynamical-systems",
  "information-theory",
  "statistics",
  "optimization"
 ],
 "ml_areas": [
  "rnn",
  "ssm",
  "world-model",
  "regularization"
 ],
 "paper": {
  "arxiv_id": "2609.00655",
  "arxiv_url": "https://arxiv.org/abs/2609.00655",
  "summary_what_math_gives_to_ml": "The paper provides a variational principle for the complexity of inverse trajectories in non-invertible dynamical systems: global preimage entropy equals the supremum of empirical preimage entropy over invariant empirical-measure classes. The transferable mechanism is to measure and control how many distinguishable hidden-state histories map to the same present state, while conditioning those histories on comparable long-run visitation statistics rather than counting all inverse branches indiscriminately. This suggests a preimage-entropy regularizer or diagnostic for recurrent state-space models and learned world models, with a falsifiable prediction that inverse-branch growth rates decompose by empirical state-distribution class and are maximized by one or more specific attractor-like visitation measures.",
  "title": "Empirical variational principles for preimage entropies",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 7,
  "novelty": 9,
  "usefulness": 6
 },
 "solves": [
  "stability",
  "generalization",
  "accuracy"
 ],
 "title": "Empirical Preimage-Entropy Regularization",
 "url": "https://synthcore.org/idea/2984/empirical-preimage-entropy-regularization",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}