Top Singular Value in Sum-Products of Random Matrices
arXiv:2607.04047
2026
Dynamics
2 ideas extracted · analyzed Aug 29, 2026
What the math gives to ML
The paper identifies a random-energy-model phase transition governing the largest singular value of a sum of deep random matrix products. The controlling parameter is not simply the depth-to-width ratio, but the effective inverse temperature beta = sqrt(2(N-1)/(n log m)), which compares path depth with the logarithm of the number of summed branches. This gives a concrete design rule for deep multi-branch networks: keep beta below the critical value sqrt(2) when stable aggregation is desired, because above it the largest path can dominate the entire sum. The same criterion can become an online diagnostic or regularizer for activation and gradient concentration in residual, multi-branch, and mixture-of-experts-like networks.
Ideas from this paper
✓ Mechanism works
2026
Use the paper's inverse-temperature parameter to initialize networks containing m parallel depth-N branches. Choose branch count, depth, or an explicit aggregation scale so that beta = sqrt(2(N-1)/(n log m)) stays below the critical value sqrt(2), preventing the largest random branch from dominating the aggregate. This is applicable to residual multi-branch MLPs and other architectures whose block Jacobian is a sum of products.
Useful7/10
Difficulty4/10
Novelty7/10
Unverified
Re-invented
2026
Add a training-time diagnostic and optional regularizer that detects whether a multi-branch block has entered the paper's low-temperature, winner-take-all regime. Estimate concentration from actual branch log-gains and penalize extreme dominance when the observed system behaves as though beta is at least sqrt(2), preserving diverse paths instead of allowing one branch to determine the block Jacobian.
Useful6/10
Difficulty5/10
Novelty6/10