✓✓ Beats tuned baseline 2026

Learnable anisotropic Jacobian smoothing

Implementation & benchmark of arXiv:2607.03692 — PIEFS: Physics-Informed Eigenfunction Features with Learnable Scaling

Usefulness7/10

Difficulty6/10

Novelty6/10

Regularization Generalization Accuracy Stability Mlp Loss Regularization Functional analysis Geometry Linear algebra Spectral theory

Source paper: PIEFS: Physics-Informed Eigenfunction Features with Learnable Scaling arXiv:2607.03692 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace isotropic input-Jacobian regularization with a positive semidefinite, input-dependent metric learned jointly with the network. The metric uses diagonal scaling to suppress sensitivity in nuisance directions and a structured orthogonal rotation to discover combinations of input coordinates in which smoothness is task-useful.

Formulas

$$\langle f_{1},f_{2}\rangle_{0}=\int f_{1}(\mathbf{x})f_{2}(\mathbf{x})p(\mathbf{x})d\mathbf{x}$$

$$\langle f_{1},f_{2}\rangle_{\mathrm{de}}=\int\sum_{a=1}^{d}\frac{\partial f_{1}(\mathbf{x})}{\partial x_{a}}\frac{\partial f_{2}(\mathbf{x})}{\partial x_{a}}p(\mathbf{x})\,\mathrm{d}\mathbf{x}$$

$$\mathcal{E}_{A}(z_\theta)=\mathbb{E}_{x\sim p}\left[\|J_{z_\theta}(x)A_\psi(x)^{\mathsf T}\|_{F}^{2}\right]=\mathbb{E}_{x\sim p}\operatorname{tr}\left(J_{z_\theta}(x)A_\psi(x)^{\mathsf T}A_\psi(x)J_{z_\theta}(x)^{\mathsf T}\right),\quad A_\psi(x)=\Lambda_\psi(x)U_\psi(x)$$

$$U_\psi(x)=\prod_{r=1}^{R}G_{i_rj_r}(\phi_r(x)),\qquad G_{ij}(\phi)_{\{i,j\},\{i,j\}}=\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix}$$

Mathematical statement

The paper defines the data-space inner product $\langle f_1,f_2\rangle_0=\int f_1(\mathbf{x})f_2(\mathbf{x})p(\mathbf{x})\,d\mathbf{x}$, where $p$ is the input density, and the ordinary Dirichlet bilinear form $\langle f_1,f_2\rangle_{\mathrm{de}}=\int\sum_{a=1}^{d}\frac{\partial f_1}{\partial x_a}\frac{\partial f_2}{\partial x_a}p(\mathbf{x})\,d\mathbf{x}$, where $d$ is input dimension. Adapt this to the modified energy $\mathcal{E}_{A}(z)=\mathbb{E}_{x\sim p}\|J_z(x)A(x)^{\mathsf T}\|_F^2$, where $z:\mathbb{R}^d\to\mathbb{R}^k$ is the neural coordinate map and $J_z\in\mathbb{R}^{k\times d}$ is its input Jacobian. Set $A(x)=\Lambda(x)U(x)$, with $\Lambda(x)=\operatorname{diag}(\exp s_1(x),\ldots,\exp s_d(x))$ positive and $U(x)$ orthogonal. Then $\|J_zA^{\mathsf T}\|_F^2=\operatorname{tr}(J_zA^{\mathsf T}AJ_z^{\mathsf T})$, so the penalty is a positive semidefinite directional smoothness cost. Parameterize $U$ as $U=\prod_{r=1}^{R}G_{i_rj_r}(\phi_r)$, where each Givens factor is identity except for the $(i,j)$ block $\begin{bmatrix}\cos\phi&-\sin\phi\\\sin\phi&\cos\phi\end{bmatrix}$.

Implementation notes

Integrate this at the representation layer immediately before the classifier, not at every layer initially. Let an MLP produce $z=f_\theta(x)\in\mathbb{R}^k$, logits $Wz+b$, and let a small metric head produce scale logits $s(x)\in\mathbb{R}^d$ and $R$ rotation angles $\phi(x)$. Use $\Lambda=\operatorname{diag}(\exp(\operatorname{clip}(s,-3,3)))$, construct $U$ by multiplying predetermined sparse Givens rotations, and set $A=\Lambda U$. A minibatch update is: (1) compute logits and cross-entropy $L_{ce}$; (2) compute $J_zA^T$ using forward-mode JVPs for each input direction, or reverse-mode vector-Jacobian products when $d$ is small; (3) compute $L_{dir}=B^{-1}\sum_{n=1}^B\|J_z(x_n)A(x_n)^T\|_F^2$; (4) optimize $L=L_{ce}+\lambda_{dir}L_{dir}+\lambda_sB^{-1}\sum_n\|s(x_n)\|_2^2$. Initialize all scales and angles at zero, so the initial metric is identity. To prevent the metric from shrinking all scales toward zero, constrain the per-example mean scale with $d^{-1}\sum_a s_a(x)=0$, or add $\lambda_{vol}(d^{-1}\sum_a s_a)^2$. The paper supplies the Dirichlet structure; minibatch averaging estimates the expectation and ordinary backpropagation learns metric parameters. First use a 2D or 10D synthetic classification task containing one high-frequency informative direction and several noisy directions. Compare ordinary weight decay, isotropic Jacobian regularization, diagonal scaling, and diagonal-plus-Givens scaling at matched parameter count. Measure validation accuracy, input-gradient norm, calibration, and training stability. A successful result is higher validation accuracy at equal compute and reduced prediction sensitivity to perturbations in nuisance coordinates.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a PyTorch MVP for learnable diagonal scaling plus a Givens rotation in an input-Jacobian regularizer. The mechanism checks passed: scale energy matched q² exactly, rotation energy matched 9cos²(phi)+sin²(phi) within 4.8e-7, and the PSD trace identity differed by only 1.22e-4 in float32. The toy classifier showed a weak accuracy increase (0.578 vs 0.564), but nuisance sensitivity increased, so the claimed stability/generalization win is not established.

Agent confidence: 8/10

Baseline: Isotropic Jacobian regularization: accuracy 0.5640; informative gradient 0.0706; nuisance gradient 0.0356; final energy 0.21995; 442 parameters.
Idea: Learnable anisotropic regularization: accuracy 0.5780; informative gradient 0.1417; nuisance gradient 0.0925; final energy 0.27184; 607 parameters.
Mechanism: Scale max error 0.0; rotation max error 4.77e-7; minimum random energy 0.0496; trace identity max error 1.22e-4.

Limitations: Only one small synthetic task and one seed were tested. Models were not parameter-count matched, and no repeated-seed statistics, calibration, or perturbation curves were measured.

How to run: python3 anisotropic_jacobian_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.03692) Sep 2, 2026

Beats the tuned baseline by 19.1% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 18.2182
Idea mean: 14.7421
Effect (Δ): -3.4761 (−19.1%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±16.8%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 8 configs

Limitations:

Only the structurally appropriate built-in tabular track was tested. Vision, sequence, and dynamics were not tested; no calibration or perturbation curves were collected; and the small benchmark uses fixed Givens coordinate pairs.

How to run: python3 bench_anisotropic.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Worked ✓

Agent confidence: 7/10

The tabular MLP benchmark reports a significant win for learnable anisotropic Jacobian smoothing: mean MSE decreased from 18.218181610107422 to 14.742117166519165 across 8 paired seeds. The paired delta is -3.476064443588257 with 8/8 wins and permutation p=0.0081, so the bench verdict is idea better (significant).

Baseline mean mse: 18.218181610107422
Delta mean mse: -3.476064443588257
Idea mean mse: 14.742117166519165
P value: 0.0081

Benchmark result