Measure-Lifted Entropy Amplifier
Implementation & benchmark of arXiv:2608.09702 — Entropies of compact subsets and supported measures
Source paper: Entropies of compact subsets and supported measures arXiv:2608.09702 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace a deterministic latent state with a probability measure over latent states, represented by particles or weighted prototypes. Apply the learned latent transition to every particle, so one base trajectory map induces a dynamics on distributions; use an entropy-preservation or entropy-growth regularizer to prevent collapse of the ensemble. The mechanism predicts that any positive base-state trajectory entropy can generate unbounded distinguishability in the ideal measure space through different mixture weights and supports.
Formulas
Mathematical statement
Let $(K,T)$ be a compact dynamical system, where $K$ is a compact latent-state set and $T:K\to K$ is the learned transition. Let $\mathcal{M}(K)$ be the Borel probability measures supported on $K$, and define the induced map $T_*:\mathcal{M}(K)\to\mathcal{M}(K)$ by $T_*\mu(A)=\mu(T^{-1}(A))$. For a metric $d$, let $s_n(E,\varepsilon)$ be the largest cardinality of an $(n,\varepsilon)$-separated subset of $E$, where $d_n(x,y)=\max_{0\leq t<n}d(T^t x,T^t y)$. The upper-capacity entropy is $h_{\mathrm{UC}}(T,E)=\lim_{\varepsilon\to0}\limsup_{n\to\infty}n^{-1}\log s_n(E,\varepsilon)$. The paper proves, for nonempty compact $K$, $h_{\mathrm{UC}}(T,K)>0$ if and only if $h_{\mathrm{UC}}(T_*,\mathcal{M}(K))=+\infty$. In a neural approximation, use empirical measures $\hat\mu=\sum_{i=1}^{m}w_i\delta_{z_i}$, propagate $z_i'=T_\theta(z_i)$, preserve or separately evolve the weights, and estimate finite-particle separated-set growth.
Implementation notes
1. Integration point: use this as the latent transition in a probabilistic RNN, state-space model, or world model. For each input context, maintain m latent particles z_i and normalized weights w_i. The deterministic transition network T_theta receives the current observation and maps every particle forward; decode with a weighted mean or a mixture decoder. Avoid resampling in the first experiment because resampling can destroy the direct pushforward structure. 2. Pseudocode: initialize particles z_i and weights w_i=1/m; for t=1,...,H compute z_i <- T_theta(z_i,x_t), y_hat_t <- sum_i w_i D_phi(z_i), and the task loss. Periodically sample pairs of particle ensembles, compute their Wasserstein-1 distance over the whole rollout, and estimate separated-set growth using a greedy epsilon-net. Add a hinge penalty if the estimated base entropy h_hat_K falls below target h_0, and penalize measure-ensemble collapse when h_hat_mu falls below H. 3. Computed from the paper: the pushforward construction and the zero-versus-infinite entropy implication. Estimated empirically: separated-set counts, Wasserstein distances, h_hat_K, and h_hat_mu; infinity is operationalized as continued growth with particle count and horizon. 4. First cheap experiment: train a 2D GRU world model on a switching nonlinear map or sequential MNIST, comparing deterministic, Gaussian-latent, and 32-particle measure-lifted models at similar parameter counts. Sweep the diversity coefficient and measure greedy (H,epsilon)-separated counts for epsilon=0.1 over H=10,20,40. The prediction is a sharp boundary: when h_hat_K is statistically zero, measure entropy remains bounded; once h_hat_K is positive, log separated-measure counts continue increasing with particle count instead of saturating, with a positive per-time slope. Fit h_hat_K from deterministic trajectories and test whether positive h_hat_K predicts nonzero measure entropy across random seeds.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.