✓✓ Beats tuned baseline 2026

Strongly-Rayleigh Forest Dropout

Implementation & benchmark of arXiv:2609.01672 — All-Minors Matrix-Tree Theory for Superport Networks: Completed Quotient-Incidence Determinants and Conductance-Weighted Subdivision Extensions

Usefulness7/10

Difficulty7/10

Novelty8/10

Regularization Generalization Accuracy Stability Moe routing Graph nn Regularization Pruning Graph theory Combinatorics Probability Linear algebra

Source paper: All-Minors Matrix-Tree Theory for Superport Networks: Completed Quotient-Incidence Determinants and Conductance-Weighted Subdivision Extensions arXiv:2609.01672 ⓘ · analyzed Sep 3, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace independent feature, expert, or edge dropout by sampling subsets from a forest-induced strongly Rayleigh distribution. The distribution is multiaffine and negatively dependent, so selecting one computational route suppresses redundant correlated routes without the positive correlations created by independent or blockwise heuristics. Use the sampled subset as a structured mask during training and compare it against Bernoulli dropout at equal expected sparsity.

Formulas

$$p(z)=\sum_{F\in\mathcal F}w(F)\prod_{e\in F}z_e,\qquad w(F)=\prod_{e\in F}c_e,\quad c_e>0.$$

$$\Pr(S)=\frac{w(S)}{p(\mathbf 1)},\qquad \Pr(A\cap B)\leq \Pr(A)\Pr(B)\quad\text{for disjoint increasing events }A,B.$$

$$Z(A)Z(B)\geq Z(A\cup B)Z(A\cap B).$$

Mathematical statement

For a graph G=(V,E) with positive edge weights c_e, let F be the relevant spanning-forest family and w(F)=product_{e in F} c_e. The paper's unsigned principal forest numerators assemble into a multiaffine real-stable polynomial p(z)=sum_{F in F} w(F) product_{e in F} z_e, where z_e is an indeterminate for edge e. Normalizing at z_e=1 gives the subset law Pr(S)=w(S)/p(1), with S the selected edge or feature coordinates. Real stability implies the law is strongly Rayleigh, hence negatively associated: for disjoint increasing events A and B, Pr(A intersection B) is at most Pr(A)Pr(B). The paper also gives the Hadamard-Fischer/log-submodular inequality Z(A)Z(B) >= Z(A union B)Z(A intersection B), where Z(A) is the unsigned numerator associated with coordinate set A. In a neural network, map graph edges to candidate experts, attention heads, channels, or message-passing routes, and use c_e=exp(theta_e) or a positive softplus parameter.

Implementation notes

Integrate this at the mask-generation step of a sparse MoE layer, graph neural network, or structured channel-pruning module. First choose a small dependency graph whose edges represent candidate routes; for an MoE, use vertices as experts and edges as admissible co-selection links, while for a GNN use physical graph edges directly. Maintain positive logits theta_e and set c_e=softplus(theta_e)+epsilon. At each training step, construct the multiaffine generating polynomial p or its determinantal representation from the corresponding weighted Laplacian. For a small MVP, enumerate all admissible forests S, compute w(S)=product(c_e), normalize the weights, and sample one subset with the desired cardinality. For larger graphs, use a fixed-size strongly Rayleigh sampler based on sequential conditional probabilities: at each step estimate the marginal of every remaining edge by evaluating p with that variable set to one versus zero, sample one edge, then condition the polynomial by deleting or contracting the chosen edge. Apply the binary mask to expert outputs, channels, or message edges and rescale by empirical inclusion probabilities. The mathematical quantities are the positive weights, forest polynomial, and negative-dependence law; sampling approximations and cardinality control are engineering choices. Start with a 4- or 8-expert MoE on CIFAR-100 or a small Transformer on WikiText-2, matching Bernoulli dropout's expected active experts and FLOPs. Measure validation loss, duplicate expert co-activation, gradient variance, and load balance. Success means equal-or-better accuracy with lower co-activation correlation and no increase in training instability; a useful target is a 1-2 point validation improvement or the same accuracy at 20% fewer active routes.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built an exact K4 spanning-tree forest sampler and a small masked-regression comparison against Bernoulli dropout at equal expected active features (3 of 6). The math check reproduced multiaffinity, zero log-submodularity violations, and nonpositive pairwise covariances (minimum -0.0625); the forest model also had lower mean validation MSE (0.2283 vs 0.2589 across 3 seeds) and lower run-to-run variation. This is a promising small-scale signal, not conclusive evidence of a general neural-network win.

Agent confidence: 7/10

Baseline: Bernoulli p=0.5: validation MSE 0.2589 +/- 0.0831 over 3 runs; expected 3 active features
Idea: Uniform spanning-tree forest mask: validation MSE 0.2283 +/- 0.0239 over 3 runs; exactly 3 active features; minimum measured pairwise inclusion covariance -0.0625

Limitations: Only a tiny synthetic regression task, K4 uniform forests, 6 input routes, 500 optimization steps, and 3 random seeds were tested. No learned positive edge weights, larger graphs, MoE/GNN architecture, FLOP benchmark, gradient-variance measurement, or direct empirical co-activation statistic was included.

How to run: python3 forest_dropout_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.01672) Sep 3, 2026

Beats the tuned baseline by 24.3% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 25.4337
Idea mean: 19.2583
Effect (Δ): -6.1754 (−24.3%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±5.1%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 9 configs

Limitations:

Only the tabular Friedman#1 track was tested; no vision, sequence, dynamics, MoE, graph-message-passing, learned edge weights, speed/FLOP measurement, or larger forest sampler was evaluated. The negative-dependence portion of the mechanism signature is measured from the sampled masks used with trained models but is distributional rather than a learned-model output correlation; the trained-model behavioral quantity additionally recorded was output variance under forest masks.

How to run: python3 stage2_forest_dropout.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 3, 2026 latest Worked ✓

Agent confidence: 8/10

Implemented Strongly-Rayleigh forest dropout as exact uniformly weighted K5 spanning-tree feature masks for the Friedman#1 tabular benchmark, using the same mlp_tiny architecture and training budget as Bernoulli dropout. The math check found 125 trees, mean inclusion 0.4, nonpositive pairwise covariances, and zero log-submodularity violations. On eight paired seeds, the idea achieved MSE 19.2583 versus 25.4337 for the tuned baseline, delta -6.1754 with permutation p=0.0081, so the benchmark verdict is a significant win.

Baseline: Bernoulli feature dropout, tuned p in {0.30,0.40,0.50} and lr in {0.0015,0.003,0.006}; best p=0.50, lr=0.0015; 8-seed test MSE 25.4337 +/- 2.8304
Delta mean: -6.175431489944458
Idea: K5 spanning-tree forest masks with exactly 4/10 active features, lr=0.0015; 8-seed test MSE 19.2583 +/- 1.7862
Idea wins: 8/8
Math: 125 spanning trees; mean inclusion 0.4; max pair covariance approximately 0; min pair covariance -0.04; 0 log-submodularity violations
Permutation p value: 0.0081

Benchmark result