Resolution-Gated Dual Masking
Implementation & benchmark of arXiv:2609.03074 — Finite-Sample Limits of Entropy-Based Structure Identification in Discretized Nonlinear Systems
Source paper: Finite-Sample Limits of Entropy-Based Structure Identification in Discretized Nonlinear Systems arXiv:2609.03074 ⓘ · analyzed Sep 4, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Add a discrete structure-selection gate before a neural predictor, maintaining separate masks for explanatory structure and predictive performance. Use entropy reduction only when the discretization resolution is finer than the observed stochasticity; otherwise use a validation-calibrated predictive mask or retain both masks through a mixture-of-experts gate.
Formulas
Mathematical statement
Let $X^j_t$ be input variable $j$ at time $t$, $Y_t$ the target, and $M$ a candidate mask containing selected lagged variables. For discretized variables with alphabet sizes $q_j$, score explanatory structure by conditional entropy reduction $S_H(j)=H(Y\mid M)-H(Y\mid M,X^j)$, where $H$ is empirical Shannon entropy. Score prediction by validation risk reduction $S_R(j)=\widehat{R}(M)-\widehat{R}(M\cup\{j\})$, with squared risk $\widehat{R}=n^{-1}\sum_{r=1}^n(y_r-\hat y_r)^2$. Define an operational resolution-stochasticity ratio $\kappa=\widehat H(Y\mid M)/H_{\mathrm{res}}$, where $H_{\mathrm{res}}=\log q_Y$ for a $q_Y$-state output, or more conservatively the entropy of the within-bin residual distribution. Low $\kappa$ means the output uncertainty is resolved by the bins; high $\kappa$ means stochasticity is comparable to or larger than the available resolution. Select the entropy mask only when its bootstrap signal-to-noise ratio $\gamma=\max_j S_H(j)/(\operatorname{sd}_{b}[S_H^{(b)}(j)]+\epsilon)$ exceeds a threshold and $\kappa<\kappa_c$. Otherwise select by $S_R$. The mechanism predicts that causal-mask recovery degrades sharply near and above $\kappa_c$, while prediction regret from using the entropy mask grows with the number of input combinations and decreases with sample size.
Implementation notes
1. Integration point: use this as a front-end selector for a tabular MLP, an RNN/SSM input window, or a transformer whose candidate tokens are variable-lag pairs. During each selection refresh, discretize each continuous feature using fixed quantile bins, and discretize the target or residual target into $q_Y$ bins solely for structure discovery; the neural predictor still receives the original continuous values. 2. Pseudocode: construct candidate mask $M$; compute plug-in counts for $\widehat H(Y|M)$ and $\widehat H(Y|M,j)$; compute $S_H(j)$; generate $B$ bootstrap resamples and estimate $\gamma$; train or briefly evaluate the network for each candidate addition to estimate $S_R(j)$; calculate $\kappa$; choose $M_H$ if $\kappa<\kappa_c$ and $\gamma>\gamma_c$, else choose $M_R$; train the final network on the chosen mask. A safer variant concatenates both masks and learns a gate $g=\sigma(a(\kappa_c-\kappa))$, using $g f_{M_H}+(1-g)f_{M_R}$. 3. Computed versus estimated: alphabet sizes, entropy formulas, and the mask rule are computed exactly from counts; $\kappa_c$, $\gamma_c$, and prediction risk are estimated on held-out data. Start with $\kappa_c=1$ and $\gamma_c=2$, then sweep them. 4. First cheap experiment: use a synthetic two-state Markov system with one true binary lag, several distractor lags, and controllable flip noise $Y_t=X_{t-1}\oplus N_t$, where $N_t\sim\operatorname{Bernoulli}(p)$. Compare entropy-only, MSE-only, and gated selection with a small MLP. Sweep bin resolution and $p$ over 0 to 0.5, using sequence lengths 256, 1024, and 4096. The predicted signature is a crossover near the point where conditional noise entropy approaches the output discretization entropy: entropy-mask recovery should fall sharply there and should not recover merely by increasing $n$ at fixed excessive noise. Below the crossover, the entropy mask should match the true lag more often; above it, MSE should have lower test risk. Measure whether the entropy-mask regret $\Delta_R$ increases with the number of distractor combinations and decreases with sample size.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.