△ Mechanism confirmed, baseline not beaten 2026

Resolution-Gated Dual Masking

Implementation & benchmark of arXiv:2609.03074 — Finite-Sample Limits of Entropy-Based Structure Identification in Discretized Nonlinear Systems

Usefulness7/10

Difficulty4/10

Novelty7/10

Regularization Generalization Accuracy Stability Training dynamics Optimizer Transformer Rnn Information theory Statistics Dynamical systems Optimization

Source paper: Finite-Sample Limits of Entropy-Based Structure Identification in Discretized Nonlinear Systems arXiv:2609.03074 ⓘ · analyzed Sep 4, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a discrete structure-selection gate before a neural predictor, maintaining separate masks for explanatory structure and predictive performance. Use entropy reduction only when the discretization resolution is finer than the observed stochasticity; otherwise use a validation-calibrated predictive mask or retain both masks through a mixture-of-experts gate.

Formulas

$$S_H(j)=\widehat H(Y\mid M)-\widehat H(Y\mid M,X^j),\qquad \widehat H(Y\mid Z)=-\sum_z\widehat p(z)\sum_y\widehat p(y\mid z)\log\widehat p(y\mid z).$$

$$S_R(j)=\widehat R(M)-\widehat R(M\cup\{j\}),\qquad \widehat R(M)=\frac{1}{n}\sum_{r=1}^{n}\left(y_r-f_\theta(x_{M,r})\right)^2.$

$$\kappa=\frac{\widehat H(Y\mid M)}{H_{\mathrm{res}}},\qquad \gamma=\frac{\max_j S_H(j)}{\operatorname{sd}_{b=1}^{B}\!\left[S_H^{(b)}(j^*)\right]+\epsilon},\qquad M=\begin{cases}M_H,&\kappa<\kappa_c\ \text{and}\ \gamma>\gamma_c,\\M_R,&\text{otherwise.}\end{cases}$$

$$\Delta_R=\widehat R(M_H)-\widehat R(M_R),\qquad \text{prediction: }\Delta_R\text{ increases with }\prod_{j\in M_H}q_j\text{ and decreases approximately as }n^{-1/2}\text{ or faster.}$$

Mathematical statement

Let $X^j_t$ be input variable $j$ at time $t$, $Y_t$ the target, and $M$ a candidate mask containing selected lagged variables. For discretized variables with alphabet sizes $q_j$, score explanatory structure by conditional entropy reduction $S_H(j)=H(Y\mid M)-H(Y\mid M,X^j)$, where $H$ is empirical Shannon entropy. Score prediction by validation risk reduction $S_R(j)=\widehat{R}(M)-\widehat{R}(M\cup\{j\})$, with squared risk $\widehat{R}=n^{-1}\sum_{r=1}^n(y_r-\hat y_r)^2$. Define an operational resolution-stochasticity ratio $\kappa=\widehat H(Y\mid M)/H_{\mathrm{res}}$, where $H_{\mathrm{res}}=\log q_Y$ for a $q_Y$-state output, or more conservatively the entropy of the within-bin residual distribution. Low $\kappa$ means the output uncertainty is resolved by the bins; high $\kappa$ means stochasticity is comparable to or larger than the available resolution. Select the entropy mask only when its bootstrap signal-to-noise ratio $\gamma=\max_j S_H(j)/(\operatorname{sd}_{b}[S_H^{(b)}(j)]+\epsilon)$ exceeds a threshold and $\kappa<\kappa_c$. Otherwise select by $S_R$. The mechanism predicts that causal-mask recovery degrades sharply near and above $\kappa_c$, while prediction regret from using the entropy mask grows with the number of input combinations and decreases with sample size.

Implementation notes

1. Integration point: use this as a front-end selector for a tabular MLP, an RNN/SSM input window, or a transformer whose candidate tokens are variable-lag pairs. During each selection refresh, discretize each continuous feature using fixed quantile bins, and discretize the target or residual target into $q_Y$ bins solely for structure discovery; the neural predictor still receives the original continuous values. 2. Pseudocode: construct candidate mask $M$; compute plug-in counts for $\widehat H(Y|M)$ and $\widehat H(Y|M,j)$; compute $S_H(j)$; generate $B$ bootstrap resamples and estimate $\gamma$; train or briefly evaluate the network for each candidate addition to estimate $S_R(j)$; calculate $\kappa$; choose $M_H$ if $\kappa<\kappa_c$ and $\gamma>\gamma_c$, else choose $M_R$; train the final network on the chosen mask. A safer variant concatenates both masks and learns a gate $g=\sigma(a(\kappa_c-\kappa))$, using $g f_{M_H}+(1-g)f_{M_R}$. 3. Computed versus estimated: alphabet sizes, entropy formulas, and the mask rule are computed exactly from counts; $\kappa_c$, $\gamma_c$, and prediction risk are estimated on held-out data. Start with $\kappa_c=1$ and $\gamma_c=2$, then sweep them. 4. First cheap experiment: use a synthetic two-state Markov system with one true binary lag, several distractor lags, and controllable flip noise $Y_t=X_{t-1}\oplus N_t$, where $N_t\sim\operatorname{Bernoulli}(p)$. Compare entropy-only, MSE-only, and gated selection with a small MLP. Sweep bin resolution and $p$ over 0 to 0.5, using sequence lengths 256, 1024, and 4096. The predicted signature is a crossover near the point where conditional noise entropy approaches the output discretization entropy: entropy-mask recovery should fall sharply there and should not recover merely by increasing $n$ at fixed excessive noise. Below the crossover, the entropy mask should match the true lag more often; above it, MSE should have lower test risk. Measure whether the entropy-mask regret $\Delta_R$ increases with the number of distractor combinations and decreases with sample size.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built Resolution-Gated Dual Masking with plug-in conditional entropy, bootstrap gamma, validation-risk scoring, and the kappa/gamma selection rule. The binary-channel check matched the analytical information gain, and entropy-mask recovery degraded near p=0.4–0.49 while risk selection was generally more reliable; the gate consequently matched the risk baseline in the difficult regime. This validates the claimed crossover/safety behavior, but no prediction improvement over ordinary risk-based selection was observed in this toy experiment.

Agent confidence: 8/10

Baseline: Risk-only selection: recovery at p=0.4 was 0.89, 0.1% MSE lower than entropy-only at n=256 averaged across distractor settings; at p=0.49 and n=256 recovery was 0.22 with MSE 0.247.
Idea: Gated selection: recovery at p=0.4 was 0.89 and at p=0.49 was 0.22 for n=256; it matched risk selection because kappa was approximately 1 and/or gamma fell below threshold. Entropy recovery declined from 1.00 at p<=0.25 to 0.17 at p=0.49 for n=256, while the analytical entropy-gain check agreed within sampling error.

Limitations: The predictor is an empirical conditional-mean lookup table rather than an MLP, and continuous quantile discretization, residual-resolution entropy, learned mixture-of-experts gating, FLOP/speed measurements, and threshold sweeps were not tested. The current toy setup uses an empty base mask, so kappa remains close to one; broader sequential mask-refresh experiments are needed to assess gains beyond safe fallback behavior.

How to run: python3 resolution_gated_masking.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.03074) Sep 4, 2026

Ahead of the baseline (−0.6%) but not statistically significant (p=0.5014, wins 4 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.0532
Idea mean: 0
Effect (Δ): -0.0391 (−0.6%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.5014 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±1.9%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 6 configs

Limitations:

Only the structurally matched registered tabular track was tested. Learned mixture-of-experts gating, broader threshold sweeps, speed/FLOP measurements, and larger datasets were not tested.

How to run: cd /home/maxwelhelp/all/math2nn/experiments/exp3124_resolution-gated-dual-masking && /home/maxwelhelp/main/bin/python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 9/10

Resolution-Gated Dual Masking was implemented and evaluated on the registered Friedman#1 tabular track with the matched mlp_tiny architecture and identical train_model path. The idea obtained lower mean MSE (7.01413 vs 7.05324), but the paired permutation test was non-significant (delta=-0.03911, p=0.50145), so this is not a demonstrated win. The trained-model mechanism signature confirmed kappa=1.0 and fallback away from the entropy mask on all 8 seeds.

Baseline: Best tuned baseline MSE=7.0532397628, lr=0.006, weight_decay=0.0001, epochs=12.
Idea: Best gated-mask MSE=7.0141316056, paired delta=-0.0391081572, 4/8 wins, permutation p=0.50145.

Benchmark result