# Strongly-Rayleigh Forest Dropout

- ID: 3020
- Canonical URL: https://synthcore.org/idea/3020/strongly-rayleigh-forest-dropout
- API JSON: https://synthcore.org/api/idea/3020.json
- API Markdown: https://synthcore.org/api/idea/3020.md
- Verification status: beats_baseline
- Source: [arXiv:2609.01672](https://arxiv.org/abs/2609.01672)
- Category: regularization
- Solves: generalization, accuracy, stability
- ML areas: moe-routing, graph-nn, regularization, pruning
- Math tags: graph-theory, combinatorics, probability, linear-algebra
- Ratings: usefulness 7/10; difficulty 7/10; novelty 8/10

## Idea description

Replace independent feature, expert, or edge dropout by sampling subsets from a forest-induced strongly Rayleigh distribution. The distribution is multiaffine and negatively dependent, so selecting one computational route suppresses redundant correlated routes without the positive correlations created by independent or blockwise heuristics. Use the sampled subset as a structured mask during training and compare it against Bernoulli dropout at equal expected sparsity.

## Mathematical statement

For a graph G=(V,E) with positive edge weights c_e, let F be the relevant spanning-forest family and w(F)=product_{e in F} c_e. The paper's unsigned principal forest numerators assemble into a multiaffine real-stable polynomial p(z)=sum_{F in F} w(F) product_{e in F} z_e, where z_e is an indeterminate for edge e. Normalizing at z_e=1 gives the subset law Pr(S)=w(S)/p(1), with S the selected edge or feature coordinates. Real stability implies the law is strongly Rayleigh, hence negatively associated: for disjoint increasing events A and B, Pr(A intersection B) is at most Pr(A)Pr(B). The paper also gives the Hadamard-Fischer/log-submodular inequality Z(A)Z(B) >= Z(A union B)Z(A intersection B), where Z(A) is the unsigned numerator associated with coordinate set A. In a neural network, map graph edges to candidate experts, attention heads, channels, or message-passing routes, and use c_e=exp(theta_e) or a positive softplus parameter.

## Key formulas

- $$p(z)=\sum_{F\in\mathcal F}w(F)\prod_{e\in F}z_e,\qquad w(F)=\prod_{e\in F}c_e,\quad c_e>0.$$
- $$\Pr(S)=\frac{w(S)}{p(\mathbf 1)},\qquad \Pr(A\cap B)\leq \Pr(A)\Pr(B)\quad\text{for disjoint increasing events }A,B.$$
- $$Z(A)Z(B)\geq Z(A\cup B)Z(A\cap B).$$

## Implementation notes

Integrate this at the mask-generation step of a sparse MoE layer, graph neural network, or structured channel-pruning module. First choose a small dependency graph whose edges represent candidate routes; for an MoE, use vertices as experts and edges as admissible co-selection links, while for a GNN use physical graph edges directly. Maintain positive logits theta_e and set c_e=softplus(theta_e)+epsilon. At each training step, construct the multiaffine generating polynomial p or its determinantal representation from the corresponding weighted Laplacian. For a small MVP, enumerate all admissible forests S, compute w(S)=product(c_e), normalize the weights, and sample one subset with the desired cardinality. For larger graphs, use a fixed-size strongly Rayleigh sampler based on sequential conditional probabilities: at each step estimate the marginal of every remaining edge by evaluating p with that variable set to one versus zero, sample one edge, then condition the polynomial by deleting or contracting the chosen edge. Apply the binary mask to expert outputs, channels, or message edges and rescale by empirical inclusion probabilities. The mathematical quantities are the positive weights, forest polynomial, and negative-dependence law; sampling approximations and cardinality control are engineering choices. Start with a 4- or 8-expert MoE on CIFAR-100 or a small Transformer on WikiText-2, matching Bernoulli dropout's expected active experts and FLOPs. Measure validation loss, duplicate expert co-activation, gradient variance, and load balance. Success means equal-or-better accuracy with lower co-activation correlation and no increase in training instability; a useful target is a 1-2 point validation improvement or the same accuracy at 20% fewer active routes.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built an exact K4 spanning-tree forest sampler and a small masked-regression comparison against Bernoulli dropout at equal expected active features (3 of 6). The math check reproduced multiaffinity, zero log-submodularity violations, and nonpositive pairwise covariances (minimum -0.0625); the forest model also had lower mean validation MSE (0.2283 vs 0.2589 across 3 seeds) and lower run-to-run variation. This is a promising small-scale signal, not conclusive evidence of a general neural-network win.

### Mechanism check

- Verdict: Built an exact K4 spanning-tree forest sampler and a small masked-regression comparison against Bernoulli dropout at equal expected active features (3 of 6). The math check reproduced multiaffinity, zero log-submodularity violations, and nonpositive pairwise covariances (minimum -0.0625); the forest model also had lower mean validation MSE (0.2283 vs 0.2589 across 3 seeds) and lower run-to-run variation. This is a promising small-scale signal, not conclusive evidence of a general neural-network win.
- Confidence: 7/10
- Limitations: Only a tiny synthetic regression task, K4 uniform forests, 6 input routes, 500 optimization steps, and 3 random seeds were tested. No learned positive edge weights, larger graphs, MoE/GNN architecture, FLOP benchmark, gradient-variance measurement, or direct empirical co-activation statistic was included.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/1216/bench_report.json)
- [forest_dropout_experiment.py](https://synthcore.org/code/1216/forest_dropout_experiment.py)
- [report.md](https://synthcore.org/code/1216/report.md)
- [report_bench_2026-09-03T121422.md](https://synthcore.org/code/1216/report_bench_2026-09-03T121422.md)
- [stage2_forest_dropout.py](https://synthcore.org/code/1216/stage2_forest_dropout.py)
- [Download all files as ZIP](https://synthcore.org/download/1216)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.