# Critical-Gain Covariance Controller

- ID: 1628
- Canonical URL: https://synthcore.org/idea/1628/critical-gain-covariance-controller
- API JSON: https://synthcore.org/api/idea/1628.json
- API Markdown: https://synthcore.org/api/idea/1628.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.00306](https://arxiv.org/abs/2608.00306)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: dynamical-systems, spectral-theory, linear-algebra, stochastic-processes
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Track the covariance of a small recurrent population state and regulate its effective gain before finite-size fluctuations diverge. The controller uses the covariance Jacobian eigenvalues from the paper, making the distance to criticality an explicit adaptive regularization signal for recurrent or state-space neural networks.

## Mathematical statement

In the paper's two-population reduction, \(c=(c_{EE},c_{II},c_{EI})^\top\) contains excitatory variance, inhibitory variance, and cross-covariance; \(k_E\) and \(k_I\) are effective excitatory and inhibitory gains, \(\Delta=k_E-k_I\), and \(m_0\), \(T\), \(N_{exc}\), \(N_{inh}\) define the baseline rate, bin duration, and population sizes. The covariance dynamics are an affine linear system \(\dot c=Jc+D\), with the displayed \(J\) and \(D\). Its eigenvalues are \(\lambda_3=-2\), \(\lambda_4=2(\Delta-1)\), and \(\lambda_5=\Delta-2\). The fixed point satisfies \(Jc^{FP}=-D\), and the paper reports fluctuations growing near \(\Delta=1\), with order \((1-\Delta)^{-3}\) scaling. We transfer the eigenvalue margin \(1-\Delta\) as a trainable stability statistic, not as a claim that arbitrary neural networks exactly obey the two-population model.

## Key formulas

- $$\frac{d}{dt}\begin{pmatrix}c_{EE}\\c_{II}\\c_{EI}\end{pmatrix}=\underbrace{\begin{pmatrix}2k_{E}-2&0&2k_{I}\\0&2k_{I}-2&2k_{E}\\k_{E}&k_{I}&k_{E}+k_{I}-2\end{pmatrix}}_{J}\begin{pmatrix}c_{EE}\\c_{II}\\c_{EI}\end{pmatrix}+\underbrace{\begin{pmatrix}\frac{m_{0}(1/T-m_{0})}{N_{exc}}\\\frac{m_{0}(1/T-m_{0})}{N_{inh}}\\0\end{pmatrix}}_{D}$$
- $$\lambda_{3}=-2,\qquad \lambda_{4}=2(\Delta-1),\qquad \lambda_{5}=\Delta-2,\qquad \Delta=k_E-k_I$$
- $$Jc^{FP}=-D,\qquad c^{FP}=-J^{-1}D$$
- $$\mathcal L_{\mathrm{crit}}=\alpha\,\big[\max(0,\Delta-(1-\varepsilon))\big]^2+\beta\,\log\!\left(1+\frac{\widehat{\operatorname{tr}C}}{\tau}\right)$$

## Implementation notes

Integrate the controller into a recurrent block or linear state-space layer after the recurrent weight matrix has been applied and before the nonlinearity. Partition hidden channels into two groups, interpreted as E and I, and parameterize recurrent weights as \(W=\begin{bmatrix}W_{EE}&W_{EI}\\W_{IE}&W_{II}\end{bmatrix}\). Estimate effective gains \(k_E,k_I\) every training step or every 100 steps using one power iteration per block: \(k_E\approx\|W_{EE}v\|_2/\|v\|_2\) and \(k_I\approx\|W_{II}u\|_2/\|u\|_2\), with nonnegative gains enforced by softplus if desired. Compute \(\Delta=k_E-k_I\), and add the criticality penalty \(\alpha[\max(0,\Delta-(1-\varepsilon))]^2\). In parallel, estimate the empirical hidden covariance \(\widehat C=(H-\bar H)^\top(H-\bar H)/(B-1)\) over batch and time samples, and penalize excessive trace or largest eigenvalue. The paper's exact \(J,D\), eigenvalues, and fixed-point relation define the two-population diagnostic; the general-network covariance is estimated empirically. For a more faithful controller, integrate \(c_{t+1}=c_t+\delta(Jc_t+D)\) and compare it with observed three-component covariance. Start with a 1-2 layer GRU on sequential MNIST or WikiText-2 against an unconstrained GRU and spectral normalization. Log \(\Delta\), covariance trace, gradient norm, and long-horizon loss. Success is elimination of exploding hidden covariance and improved loss over long sequences with less performance degradation than hard spectral clipping.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built the exact covariance Jacobian diagnostic, fixed-point sweep, and a small CUDA RNN comparison with the proposed gain/covariance penalty. The stated Delta eigenvalue formula failed with maximum absolute error 3.2; symbolic and numerical checks instead confirmed eigenvalues {-2, 2(kE+kI-1), kE+kI-2} to 4.0e-12, with the stability boundary at kE+kI=1. The measured decay-rate scaling matched the corrected prediction closely and fixed-point log-log scaling was -0.945 versus -1.0, but the proposed controller did not reduce hidden covariance and only modestly improved long loss (0.7053 vs 0.7456).

### Mechanism check

- Verdict: Built the exact covariance Jacobian diagnostic, fixed-point sweep, and a small CUDA RNN comparison with the proposed gain/covariance penalty. The stated Delta eigenvalue formula failed with maximum absolute error 3.2; symbolic and numerical checks instead confirmed eigenvalues {-2, 2(kE+kI-1), kE+kI-2} to 4.0e-12, with the stability boundary at kE+kI=1. The measured decay-rate scaling matched the corrected prediction closely and fixed-point log-log scaling was -0.945 versus -1.0, but the proposed controller did not reduce hidden covariance and only modestly improved long loss (0.7053 vs 0.7456).
- Confidence: 10/10
- Limitations: Only a tiny synthetic next-symbol RNN was tested, not sequential MNIST, language modeling, GRU/SSM architectures, spectral normalization, or multi-seed statistical comparisons. The supplied paper formula was not reconciled against a broader model variant because the displayed J directly contradicts the stated Delta eigenvalues.

## Artifacts

- [experiment.py](https://synthcore.org/code/607/experiment.py)
- [report.md](https://synthcore.org/code/607/report.md)
- [results.json](https://synthcore.org/code/607/results.json)
- [Download all files as ZIP](https://synthcore.org/download/607)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.