✗ Failed on benchmark 2026

Correlation-Exponent-Safe Weight Initialization

Implementation & benchmark of arXiv:2608.23944 — Bulk Phase Transition and Edge Behavior in Temporally Correlated Random Matrices

Usefulness7/10

Difficulty4/10

Novelty7/10

Architecture Stability Generalization Initialization Mlp Transformer Ssm Random matrix Spectral theory Statistics Statistical mechanics

Source paper: Bulk Phase Transition and Edge Behavior in Temporally Correlated Random Matrices arXiv:2608.23944 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Initialize each row of a neural weight matrix as a stationary correlated Gaussian process instead of using independent entries, but constrain its correlation tail to remain on the finite-fourth-moment side of the transition. This creates controllable structured spectra while avoiding the heavy-edge regime predicted for correlations slower than $t^{-1/2}$.

Formulas

$$S_{ij}=\frac{\sigma}{\sqrt{N}}\eta_i(j),\qquad \operatorname{Cov}(S_{ab},S_{cd})=\frac{\sigma^2}{N}\delta_{\min(a,b),\min(c,d)}c_{|\max(a,b)-\max(c,d)|}.$$

$$M_2(N)=\sum_{t=1}^{N}c_t^2\asymp\begin{cases}N^{1-2\gamma},&\gamma<\tfrac12,\\ \log N,&\gamma=\tfrac12,\\ 1,&\gamma>\tfrac12,\end{cases}\qquad c_t\asymp t^{-\gamma}.$$

$$\eta_i(j)=\rho\eta_i(j-1)+\sqrt{1-\rho^2}\,\varepsilon_i(j),\qquad |\rho|<1.$$

Mathematical statement

For a symmetric matrix $S$, the paper uses $S_{ij}=\sigma\eta_i(j)/\sqrt{N}$, where each row process $\eta_i(j)$ is Gaussian and independent across rows. Its within-row covariance is $\operatorname{Cov}(S_{ab},S_{cd})=(\sigma^2/N)\delta_{\min(a,b),\min(c,d)}c_{|\max(a,b)-\max(c,d)|}$. For a power-law correlation $c_t\asymp t^{-\gamma}$, the correlated fourth-moment contribution scales as $\sum_{t=1}^{N}c_t^2$, which is finite for $\gamma>1/2$, logarithmic at $\gamma=1/2$, and proportional to $N^{1-2\gamma}$ for $\gamma<1/2$. The proposed neural initialization uses the same row-wise construction and treats this fourth-moment scaling as a spectral-risk certificate.

Implementation notes

1. Integration point: replace iid Gaussian initialization of an MLP, attention projection, convolutional channel matrix, or SSM transition matrix by row-wise correlated initialization. For every output row $i$, generate a Gaussian sequence $z_i(1{:}d)$ with covariance $c_{|j-k|}$, then set $W_{ij}=\sigma z_i(j)/\sqrt{d}$. Start with an AR(1) covariance, generated recursively by $z_j=\rho z_{j-1}+\sqrt{1-\rho^2}\epsilon_j$, and then test truncated power-law covariances using a stationary Gaussian sampler. 2. Pseudocode: sample independent rows; compute the empirical row autocovariance $\hat c_t$; form $\hat M_2=\sum_t\hat c_t^2$; reject or rescale the initialization if $\hat M_2$ exceeds a chosen budget; apply standard variance normalization afterward. 3. Computed from the paper: the covariance construction and the $\gamma=1/2$ fourth-moment boundary. Estimated empirically: the largest singular value, empirical fourth spectral moment $N^{-1}\operatorname{tr}[(WW^\top)^2]$, and activation/Jacobian growth. 4. First cheap experiment: compare iid, AR(1) with $\rho\in\{0.5,0.9,0.99\}$, and power-law correlations with $\gamma\in\{0.3,0.5,0.7\}$ in a width-512 MLP on MNIST and a 2-layer residual CIFAR-10 model. Track singular values at initialization and after 100 updates. The quantitative prediction is that the normalized fourth spectral moment grows approximately as $N^{1-2\gamma}$ below $\gamma=1/2$, as $\log N$ near the boundary, and remains bounded above it; the largest-singular-value instability should appear sharply for the $\gamma<1/2$ family.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built row-wise stationary correlated Gaussian initialization with iid, AR(1), and power-law Toeplitz covariance options, plus fourth-moment scaling, spectral, and 100-step MLP checks. The core predictions were observed: gamma=0.3 gave log-log M2 slope 0.485 versus predicted 0.4, gamma=0.5 gave an approximately linear-in-log-width slope 0.995, and gamma=0.7 showed only slow finite-size growth (slope 0.088) consistent with bounded asymptotics. The unsafe gamma=0.3 spectrum grew from max singular value 5.30 to 8.91 and m4 from 13.90 to 28.74 over widths 64 to 256, while safe gamma=0.7 grew from 3.58 to 4.80 and 4.89 to 5.59; this is a clear spectral-risk signal. In the tiny MLP, safe gamma=0.7 reached loss 0.141 versus iid 0.182 and unsafe gamma=0.3 0.161 after 100 updates, but this training improvement is not established as robust.

Agent confidence: 8/10

Baseline: iid: max_sv/m4 at width 256 = 1.980/2.028; MLP loss step 100 = 0.1822
Idea: power gamma=0.7: max_sv/m4 at width 256 = 4.804/5.591; MLP loss step 100 = 0.1407. Unsafe gamma=0.3: 8.906/28.743 and loss 0.1610. Scaling slopes: gamma=.3/.5/.7 = .485/.231/.088, with boundary M2-vs-log-width slope .995; predicted subcritical exponent=.4 and boundary logarithmic growth.

Limitations: Only synthetic data and a small one-hidden-layer MLP were tested; no MNIST, CIFAR-10, attention, convolution, or 512-wide 100-update benchmark was run. The finite width range makes the gamma=0.7 bounded conclusion asymptotic rather than definitive, and the power-law sampler uses finite Toeplitz covariance with small jitter.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.23944) Sep 2, 2026

Ahead of the baseline (−13.7%) but not statistically significant (p=0.0751, wins 6 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 15.3926
Idea mean: 13.2881
Effect (Δ): -2.1045 (−13.7%; negative = idea better)
Wins: 6 / 8 paired seeds
p-value: 0.0751 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±14.2%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 9 configs

Limitations:

Only the structurally matched built-in tabular track and mlp_tiny model were tested. The idea used AR(1) correlation values rho in {0.5,0.9,0.99}; truncated power-law covariance families, vision/CIFAR, sequence, dynamics, wider MLPs, and longer training were not tested. The mechanism signature used trained weights but showed substantial finite-training deviation from the analytic infinite-width AR energy.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Built correlation-safe row-wise AR(1) initialization for the bench MLP and evaluated it against the same architecture and training loop. On the tabular benchmark, the tuned idea reached mean test MSE 13.2881 versus baseline 15.3926, but the paired permutation p-value was 0.07505, so this is a promising but non-significant improvement and does not qualify as a win. The trained-model mechanism signature was not confirmed: observed autocorrelation energy was 7.36 times the infinite-width AR prediction.

Baseline: mean test MSE 15.392620921134949; per-seed [13.949790954589844,16.821916580200195,13.37869644165039,14.50743293762207,11.848803520202637,16.447656631469727,16.45771598815918,19.728954315185547]; best_cfg lr=0.006, rho=0.5, wd=0.0
Idea: mean test MSE 13.288133263587952; per-seed [12.68891716003418,12.484254837036133,15.12032413482666,14.082161903381348,12.240248680114746,12.460922241210938,10.672504425048828,16.55573272705078]; best_cfg lr=0.006, rho=0.5, wd=0.0

Benchmark result