# Learnable anisotropic Jacobian smoothing

- ID: 245
- Canonical URL: https://synthcore.org/idea/245/learnable-anisotropic-jacobian-smoothing
- API JSON: https://synthcore.org/api/idea/245.json
- API Markdown: https://synthcore.org/api/idea/245.md
- Verification status: beats_baseline
- Source: [arXiv:2607.03692](https://arxiv.org/abs/2607.03692)
- Category: regularization
- Solves: generalization, accuracy, stability
- ML areas: mlp, loss, regularization
- Math tags: functional-analysis, geometry, linear-algebra, spectral-theory
- Ratings: usefulness 7/10; difficulty 6/10; novelty 6/10

## Idea description

Replace isotropic input-Jacobian regularization with a positive semidefinite, input-dependent metric learned jointly with the network. The metric uses diagonal scaling to suppress sensitivity in nuisance directions and a structured orthogonal rotation to discover combinations of input coordinates in which smoothness is task-useful.

## Mathematical statement

The paper defines the data-space inner product \(\langle f_1,f_2\rangle_0=\int f_1(\mathbf{x})f_2(\mathbf{x})p(\mathbf{x})\,d\mathbf{x}\), where \(p\) is the input density, and the ordinary Dirichlet bilinear form \(\langle f_1,f_2\rangle_{\mathrm{de}}=\int\sum_{a=1}^{d}\frac{\partial f_1}{\partial x_a}\frac{\partial f_2}{\partial x_a}p(\mathbf{x})\,d\mathbf{x}\), where \(d\) is input dimension. Adapt this to the modified energy \(\mathcal{E}_{A}(z)=\mathbb{E}_{x\sim p}\|J_z(x)A(x)^{\mathsf T}\|_F^2\), where \(z:\mathbb{R}^d\to\mathbb{R}^k\) is the neural coordinate map and \(J_z\in\mathbb{R}^{k\times d}\) is its input Jacobian. Set \(A(x)=\Lambda(x)U(x)\), with \(\Lambda(x)=\operatorname{diag}(\exp s_1(x),\ldots,\exp s_d(x))\) positive and \(U(x)\) orthogonal. Then \(\|J_zA^{\mathsf T}\|_F^2=\operatorname{tr}(J_zA^{\mathsf T}AJ_z^{\mathsf T})\), so the penalty is a positive semidefinite directional smoothness cost. Parameterize \(U\) as \(U=\prod_{r=1}^{R}G_{i_rj_r}(\phi_r)\), where each Givens factor is identity except for the \((i,j)\) block \(\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix}\).

## Key formulas

- $$\langle f_{1},f_{2}\rangle_{0}=\int f_{1}(\mathbf{x})f_{2}(\mathbf{x})p(\mathbf{x})d\mathbf{x}$$
- $$\langle f_{1},f_{2}\rangle_{\mathrm{de}}=\int\sum_{a=1}^{d}\frac{\partial f_{1}(\mathbf{x})}{\partial x_{a}}\frac{\partial f_{2}(\mathbf{x})}{\partial x_{a}}p(\mathbf{x})\,\mathrm{d}\mathbf{x}$$
- $$\mathcal{E}_{A}(z_\theta)=\mathbb{E}_{x\sim p}\left[\|J_{z_\theta}(x)A_\psi(x)^{\mathsf T}\|_{F}^{2}\right]=\mathbb{E}_{x\sim p}\operatorname{tr}\left(J_{z_\theta}(x)A_\psi(x)^{\mathsf T}A_\psi(x)J_{z_\theta}(x)^{\mathsf T}\right),\quad A_\psi(x)=\Lambda_\psi(x)U_\psi(x)$$
- $$U_\psi(x)=\prod_{r=1}^{R}G_{i_rj_r}(\phi_r(x)),\qquad G_{ij}(\phi)_{\{i,j\},\{i,j\}}=\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix}$$

## Implementation notes

Integrate this at the representation layer immediately before the classifier, not at every layer initially. Let an MLP produce \(z=f_\theta(x)\in\mathbb{R}^k\), logits \(Wz+b\), and let a small metric head produce scale logits \(s(x)\in\mathbb{R}^d\) and \(R\) rotation angles \(\phi(x)\). Use \(\Lambda=\operatorname{diag}(\exp(\operatorname{clip}(s,-3,3)))\), construct \(U\) by multiplying predetermined sparse Givens rotations, and set \(A=\Lambda U\). A minibatch update is: (1) compute logits and cross-entropy \(L_{ce}\); (2) compute \(J_zA^T\) using forward-mode JVPs for each input direction, or reverse-mode vector-Jacobian products when \(d\) is small; (3) compute \(L_{dir}=B^{-1}\sum_{n=1}^B\|J_z(x_n)A(x_n)^T\|_F^2\); (4) optimize \(L=L_{ce}+\lambda_{dir}L_{dir}+\lambda_sB^{-1}\sum_n\|s(x_n)\|_2^2\). Initialize all scales and angles at zero, so the initial metric is identity. To prevent the metric from shrinking all scales toward zero, constrain the per-example mean scale with \(d^{-1}\sum_a s_a(x)=0\), or add \(\lambda_{vol}(d^{-1}\sum_a s_a)^2\). The paper supplies the Dirichlet structure; minibatch averaging estimates the expectation and ordinary backpropagation learns metric parameters. First use a 2D or 10D synthetic classification task containing one high-frequency informative direction and several noisy directions. Compare ordinary weight decay, isotropic Jacobian regularization, diagonal scaling, and diagonal-plus-Givens scaling at matched parameter count. Measure validation accuracy, input-gradient norm, calibration, and training stability. A successful result is higher validation accuracy at equal compute and reduced prediction sensitivity to perturbations in nuisance coordinates.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built a PyTorch MVP for learnable diagonal scaling plus a Givens rotation in an input-Jacobian regularizer. The mechanism checks passed: scale energy matched q² exactly, rotation energy matched 9cos²(phi)+sin²(phi) within 4.8e-7, and the PSD trace identity differed by only 1.22e-4 in float32. The toy classifier showed a weak accuracy increase (0.578 vs 0.564), but nuisance sensitivity increased, so the claimed stability/generalization win is not established.

### Mechanism check

- Verdict: Built a PyTorch MVP for learnable diagonal scaling plus a Givens rotation in an input-Jacobian regularizer. The mechanism checks passed: scale energy matched q² exactly, rotation energy matched 9cos²(phi)+sin²(phi) within 4.8e-7, and the PSD trace identity differed by only 1.22e-4 in float32. The toy classifier showed a weak accuracy increase (0.578 vs 0.564), but nuisance sensitivity increased, so the claimed stability/generalization win is not established.
- Confidence: 8/10
- Limitations: Only one small synthetic task and one seed were tested. Models were not parameter-count matched, and no repeated-seed statistics, calibration, or perturbation curves were measured.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [anisotropic_jacobian_experiment.py](https://synthcore.org/code/108/anisotropic_jacobian_experiment.py)
- [bench_anisotropic.py](https://synthcore.org/code/108/bench_anisotropic.py)
- [bench_report.json](https://synthcore.org/code/108/bench_report.json)
- [report.md](https://synthcore.org/code/108/report.md)
- [report_bench_2026-08-30T142652.md](https://synthcore.org/code/108/report_bench_2026-08-30T142652.md)
- [results.json](https://synthcore.org/code/108/results.json)
- [Download all files as ZIP](https://synthcore.org/download/108)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.