# Two-sided conditioned DFA

- ID: 1115
- Canonical URL: https://synthcore.org/idea/1115/two-sided-conditioned-dfa
- API JSON: https://synthcore.org/api/idea/1115.json
- API Markdown: https://synthcore.org/api/idea/1115.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.18574](https://arxiv.org/abs/2607.18574)
- Category: optimizer
- Solves: accuracy, stability, sample-efficiency
- ML areas: optimizer, training, mlp, cnn
- Math tags: linear-algebra, optimization, statistics
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Replace the raw DFA outer-product update with a damped left-right preconditioned update that whitens both presynaptic activity directions and local-error directions. The activity factor removes nuisance-dominated input anisotropy, while the error factor equalizes postsynaptic credit coordinates; separate damping prevents noisy error covariances from destabilizing training.

## Mathematical statement

For layer \(\ell\), let \(h_{\ell-1}\in\mathbb{R}^{d_{\ell-1}}\) be the presynaptic activation, \(a_\ell\in\mathbb{R}^{d_\ell}\) the preactivation, \(\phi'\) the elementwise derivative, \(e\in\mathbb{R}^{d_o}\) the output error, and \(B_\ell\in\mathbb{R}^{d_\ell\times d_o}\) a fixed DFA feedback matrix. The paper defines \(\delta_\ell^{\mathrm{DFA}}=(B_\ell e)\odot\phi'(a_\ell)\) and \(G_\ell^{\mathrm{DFA}}=\delta_\ell^{\mathrm{DFA}}h_{\ell-1}^{\top}\), where \(\odot\) denotes elementwise multiplication. It defines uncentered second moments \(C_{A,\ell}=\mathbb{E}[h_{\ell-1}h_{\ell-1}^{\top}]\) and \(C_{E,\ell}=\mathbb{E}[\delta_\ell^{\mathrm{DFA}}(\delta_\ell^{\mathrm{DFA}})^{\top}]\). Use damped inverses \(P_{A,\ell}=(C_{A,\ell}+\lambda_A I_{d_{\ell-1}})^{-1}\) and \(P_{E,\ell}=(C_{E,\ell}+\lambda_E I_{d_\ell})^{-1}\), where \(\lambda_A,\lambda_E>0\), and update \(W_\ell\) with \(W_\ell\leftarrow W_\ell-\eta P_{E,\ell}G_\ell^{\mathrm{DFA}}P_{A,\ell}\). Left multiplication corrects error-coordinate anisotropy and right multiplication corrects activity-coordinate anisotropy. The factorization \(P_E(\delta h^\top)P_A=(P_E\delta)(P_Ah)^\top\) preserves a local outer-product implementation, while damping limits amplification from small covariance eigenvalues.

## Key formulas

- $$\delta_{\ell}^{\mathrm{DFA}}=(B_{\ell}e)\odot\phi^{\prime}(a_{\ell}),\qquad G_{\ell}^{\mathrm{DFA}}=\delta_{\ell}^{\mathrm{DFA}}h_{\ell-1}^{\top}.$$
- $$C_{A,\ell}=\mathbb{E}[h_{\ell-1}h_{\ell-1}^{\top}],\qquad C_{E,\ell}=\mathbb{E}[\delta_{\ell}^{\mathrm{DFA}}(\delta_{\ell}^{\mathrm{DFA}})^{\top}].$$
- $$P_{A,\ell}=(C_{A,\ell}+\lambda_A I)^{-1},\qquad P_{E,\ell}=(C_{E,\ell}+\lambda_E I)^{-1},\qquad W_\ell\leftarrow W_\ell-\eta P_{E,\ell}G_{\ell}^{\mathrm{DFA}}P_{A,\ell}.$$
- $$P_{E,\ell}G_{\ell}^{\mathrm{DFA}}P_{A,\ell}=\bigl(P_{E,\ell}\delta_{\ell}^{\mathrm{DFA}}\bigr)\bigl(P_{A,\ell}h_{\ell-1}\bigr)^{\top}.$$

## Implementation notes

Integrate this at the hidden-layer weight-update site of an MLP or small CNN trained with direct feedback alignment. For every minibatch and layer, compute the usual DFA error \(\delta=(Be)\odot\phi'(a)\) and raw batch gradient \(G=\frac{1}{m}\sum_{i=1}^m\delta_i h_i^\top\), where \(m\) is batch size. Maintain exponential moving averages \(C_A\leftarrow\beta C_A+(1-\beta)\frac{1}{m}\sum_i h_i h_i^\top\) and \(C_E\leftarrow\beta C_E+(1-\beta)\frac{1}{m}\sum_i\delta_i\delta_i^\top\). Every update, or every K updates for speed, solve linear systems using Cholesky factors for \(P_A=(C_A+\lambda_A I)^{-1}\) and \(P_E=(C_E+\lambda_E I)^{-1}\); do not explicitly materialize inverses. Apply \(\Delta W=-\eta P_EGP_A\). Equivalently, transform each factor before accumulating the outer product: \(\tilde\delta_i=P_E\delta_i\), \(\tilde h_i=P_Ah_i\), then use \(\Delta W=-\eta m^{-1}\sum_i\tilde\delta_i\tilde h_i^\top\). The mathematics determines the second moments and inverse preconditioners; estimate them empirically. Initialize with \(\beta=0.99\), \(\lambda_A=10^{-3}\operatorname{tr}(C_A)/d_{\ell-1}\), and the more conservative \(\lambda_E=10^{-2}\operatorname{tr}(C_E)/d_\ell\), increasing \(\lambda_E\) if updates become unstable. First test a 2--3 hidden-layer tanh MLP on MNIST and Fashion-MNIST. Compare raw DFA, activity-only conditioning \((P_E=I)\), error-only conditioning \((P_A=I)\), and two-sided conditioning using matched learning-rate sweeps, feedback matrices, and parameter counts. Measure validation accuracy, loss versus optimizer steps, update norms, covariance condition numbers, wall-clock time, and divergence frequency. Success means reproducible accuracy or loss-descent gains over raw DFA and activity-only DFA at equal steps; failure means no gain after accounting for covariance-maintenance overhead.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a two-sided conditioned DFA MVP with EMA activity/error covariances, damped linear solves, mechanism checks, and matched raw/activity-only/error-only/two-sided DFA training runs. The required mechanism manifested: outer-product factorization error was 1.22e-15, the anisotropy sweep exactly matched predicted transformed-covariance ratios, scalar two-sided gain matched 1/((cE+lambdaE)(cA+lambdaA)) to numerical precision, and an independent 200k-sample covariance check had maximum error 0.0068. In training, two-sided DFA improved over raw DFA (83.6% vs 78.5%), but activity-only was better (86.9%) and conditioning incurred substantial overhead, so this is mechanism-confirmed rather than a demonstrated overall optimization win.

### Mechanism check

- Verdict: Built a two-sided conditioned DFA MVP with EMA activity/error covariances, damped linear solves, mechanism checks, and matched raw/activity-only/error-only/two-sided DFA training runs. The required mechanism manifested: outer-product factorization error was 1.22e-15, the anisotropy sweep exactly matched predicted transformed-covariance ratios, scalar two-sided gain matched 1/((cE+lambdaE)(cA+lambdaA)) to numerical precision, and an independent 200k-sample covariance check had maximum error 0.0068. In training, two-sided DFA improved over raw DFA (83.6% vs 78.5%), but activity-only was better (86.9%) and conditioning incurred substantial overhead, so this is mechanism-confirmed rather than a demonstrated overall optimization win.
- Confidence: 8/10
- Limitations: Only a small synthetic anisotropic 3-class MLP was tested, not MNIST, Fashion-MNIST, or CNNs. Learning-rate sweeps, K-step preconditioner refreshes, Cholesky implementations, FLOP-normalized comparisons, and broader damping/beta studies were not tested; wall-clock measurements were CPU-only and reflect repeated dense linear solves.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/445/bench_report.json)
- [experiment.py](https://synthcore.org/code/445/experiment.py)
- [report.md](https://synthcore.org/code/445/report.md)
- [report_bench_2026-08-31T123046.md](https://synthcore.org/code/445/report_bench_2026-08-31T123046.md)
- [results.json](https://synthcore.org/code/445/results.json)
- [stage2_bench.py](https://synthcore.org/code/445/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/445)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.