# Free-Loss Jacobian Spectral Target

- ID: 447
- Canonical URL: https://synthcore.org/idea/447/free-loss-jacobian-spectral-target
- API JSON: https://synthcore.org/api/idea/447.json
- API Markdown: https://synthcore.org/api/idea/447.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.05693](https://arxiv.org/abs/2607.05693)
- Category: regularization
- Solves: stability, accuracy, generalization
- ML areas: training-dynamics, initialization, regularization, transformer
- Math tags: probability, spectral-theory, random-matrix, linear-algebra
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Regularize the end-to-end Jacobian singular-value distribution of a deep network toward the explicit free small-loss law generated by independently mixed projection-like layers. The target controls several gradient-spectrum moments, including the predicted fraction of nearly preserved directions, instead of controlling only the average gradient norm.

## Mathematical statement

For independent Haar unitary matrices S_i in U(N) and a rank-(1-c)N orthogonal projection P, the paper studies B_L=(PS_L)(PS_{L-1})\cdots(PS_1), whose squared singular-value distribution converges for fixed L to 
u_c^{\boxtimes L}, where 
u_c=(1-c)\delta_1+c\delta_0, \delta_x is a point mass at x, and \boxtimes is free multiplicative convolution. In the weak-loss scaling c=1/N and L/N    o    au, the limiting law \mu_    au has S-transform S_{\mu_    au}(z)=\exp(    au/(1+z)). Its p-th moment is m_p(    au)=\frac{1}{p}[w^{p-1}](1+w)^p\exp(-p    au/(1+w)), where [w^{p-1}] extracts the coefficient of w^{p-1}. Here p is a positive integer,     au is effective depth divided by width, and m_p is the expected p-th moment of squared singular values. The first moments are m_1=e^{-    au} and m_2=e^{-2    au}(1+    au). The law also has an atom \mu_    au(\{1\})=(1-    au)_+, where (x)_+=\max(x,0). Use these moments as targets for the eigenvalues of a neural-network Jacobian Gram matrix J^    op J, with J the input-output Jacobian and N the probe dimension or hidden width. The free-law assumptions are exact only for independent isotropic mixing and weak loss, so this is an experimentally testable initialization or regularization hypothesis rather than a direct theorem about trained networks.

## Key formulas

- $$B_L=(PS_L)(PS_{L-1})\cdots(PS_1),\qquad \nu_c=(1-c)\delta_1+c\delta_0,$$
- $$S_{\mu_\tau}(z)=\exp\left(\frac{\tau}{1+z}\right),\qquad \mu_\tau(\{1\})=(1-\tau)_+,$$
- $$m_p(\tau)=\frac{1}{p}[w^{p-1}](1+w)^p\exp\left(-\frac{p\tau}{1+w}\right),\qquad m_1=e^{-\tau},\quad m_2=e^{-2\tau}(1+\tau),$$
- $$\mathcal L_{\mathrm{spec}}=\sum_{p=1}^{K}\alpha_p\left(\log(\widehat m_p+\epsilon)-\log(m_p(\tau)+\epsilon)\right)^2,\qquad \widehat m_p=\frac{1}{R}\sum_{r=1}^{R}v_r^\top(J^\top J)^p v_r.$$

## Implementation notes

Integrate the method into the training loop of a deep MLP, residual transformer, or state-space stack. Let J denote the Jacobian from a minibatch input representation x to its output representation y; never materialize J. Every T training steps, draw R normalized Gaussian or Rademacher probe vectors v_r with the same dimension as x. For each probe set u_{r,0}=v_r, then repeatedly compute u_{r,k+1}=J^    op(Ju_{r,k}) using a Jacobian-vector product followed by a vector-Jacobian product. Estimate each Gram moment by \widehat m_p=(1/R)\sum_r v_r^    op u_{r,p}, for p=1,...,K, with K=2 or 3. Select an effective parameter     au, initially L/N where L is the number of mixing blocks and N is their width, and sweep     au in {0.25,0.5,1,2}. Precompute m_p(    au) from the coefficient-extraction formula using polynomial expansion or automatic symbolic differentiation. Add \lambda_{spec}\mathcal L_{spec} to the task loss, stop-gradient through m_p, clip the auxiliary loss, and apply it only during the first 10-30% of training or every T steps. The mathematical quantities computed exactly are the target moments; \widehat m_p is estimated empirically by Hutchinson-style probes. First test a 12-layer width-256 MLP on CIFAR-10 or a 6-layer transformer on a small language-model corpus. Compare Xavier or orthogonal initialization, a scalar Jacobian-norm penalty, and this spectral penalty. Record early loss descent at equal FLOPs, gradient-norm variance, activation variance, final accuracy or perplexity, and singular-value histograms. Success requires closer tracking of m_1 and m_2, reduced exploding or vanishing gradients, and faster early optimization without worse final task quality.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a reproducible free-loss moment calculator, Haar projection-product verification, and MLP spectral-regularization experiment. The product check closely matched m1-m3 and reproduced the atom masses: empirical versus target moment errors were below 0.005 for tau in {0.25, 0.5, 1, 2}. In the MLP, spectral regularization reduced final log-moment RMSE from 4.15 to 2.82 and reduced both moments substantially toward target, but final task MSE worsened from 0.767 to 0.922 and optimization was slower; therefore this is a genuine spectral-control signal, not evidence of an end-to-end task-performance win.

### Mechanism check

- Verdict: Built a reproducible free-loss moment calculator, Haar projection-product verification, and MLP spectral-regularization experiment. The product check closely matched m1-m3 and reproduced the atom masses: empirical versus target moment errors were below 0.005 for tau in {0.25, 0.5, 1, 2}. In the MLP, spectral regularization reduced final log-moment RMSE from 4.15 to 2.82 and reduced both moments substantially toward target, but final task MSE worsened from 0.767 to 0.922 and optimization was slower; therefore this is a genuine spectral-control signal, not evidence of an end-to-end task-performance win.
- Confidence: 8/10
- Limitations: The MLP experiment used exact per-example Jacobians rather than memory-efficient JVP/VJP probes, only one seed and one tiny regression task, 120 steps, K=2, and no CIFAR/transformer benchmark. The implementation did not test GPU execution, multiple regularizer weights, equal-FLOP comparisons, gradient variance, or trained-network singular-value histograms.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 0/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/1305/bench_report.json)
- [experiment.py](https://synthcore.org/code/1305/experiment.py)
- [report.md](https://synthcore.org/code/1305/report.md)
- [report_bench_2026-09-04T132152.md](https://synthcore.org/code/1305/report_bench_2026-09-04T132152.md)
- [results.json](https://synthcore.org/code/1305/results.json)
- [stage2_bench.py](https://synthcore.org/code/1305/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1305)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.