Two-sided conditioned DFA

Integrate this at the hidden-layer weight-update site of an MLP or small CNN trained with direct feedback alignment. For every minibatch and layer, compute the usual DFA error \(\delta=(Be)\odot\phi'(a)\) and raw batch gradient \(G=\frac{1}{m}\sum_{i=1}^m\delta_i h_i^\top\), where \(m\) is batch size. Maintain exponential moving averages \(C_A\leftarrow\beta C_A+(1-\beta)\frac{1}{m}\sum_i h_i h_i^\top\) and \(C_E\leftarrow\beta C_E+(1-\beta)\frac{1}{m}\sum_i\delta_i\delta_i^\top\). Every update, or every K updates for speed, solve linear systems using Cholesky factors for \(P_A=(C_A+\lambda_A I)^{-1}\) and \(P_E=(C_E+\lambda_E I)^{-1}\); do not explicitly materialize inverses. Apply \(\Delta W=-\eta P_EGP_A\). Equivalently, transform each factor before accumulating the outer product: \(\tilde\delta_i=P_E\delta_i\), \(\tilde h_i=P_Ah_i\), then use \(\Delta W=-\eta m^{-1}\sum_i\tilde\delta_i\tilde h_i^\top\). The mathematics determines the second moments and inverse preconditioners; estimate them empirically. Initialize with \(\beta=0.99\), \(\lambda_A=10^{-3}\operatorname{tr}(C_A)/d_{\ell-1}\), and the more conservative \(\lambda_E=10^{-2}\operatorname{tr}(C_E)/d_\ell\), increasing \(\lambda_E\) if updates become unstable. First test a 2--3 hidden-layer tanh MLP on MNIST and Fashion-MNIST. Compare raw DFA, activity-only conditioning \((P_E=I)\), error-only conditioning \((P_A=I)\), and two-sided conditioning using matched learning-rate sweeps, feedback matrices, and parameter counts. Measure validation accuracy, loss versus optimizer steps, update norms, covariance condition numbers, wall-clock time, and divergence frequency. Success means reproducible accuracy or loss-descent gains over raw DFA and activity-only DFA at equal steps; failure means no gain after accounting for covariance-maintenance overhead.