# Hankel Moment Regularizer

- ID: 281
- Canonical URL: https://synthcore.org/idea/281/hankel-moment-regularizer
- API JSON: https://synthcore.org/api/idea/281.json
- API Markdown: https://synthcore.org/api/idea/281.md
- Verification status: mechanism_failed
- Source: [arXiv:2606.31647](https://arxiv.org/abs/2606.31647)
- Category: regularization
- Solves: stability, generalization, accuracy
- ML areas: regularization, training-dynamics, diffusion
- Math tags: total-positivity, moment-theory, convex-analysis, measure-theory
- Ratings: usefulness 6/10; difficulty 3/10; novelty 8/10

## Idea description

Regularize a neural network's response along an ordered variable by requiring its sampled values to form a positive Hankel moment sequence. This upgrades ordinary pairwise monotonicity or log-convexity penalties into simultaneous constraints on several higher-order interactions, while remaining differentiable and inexpensive for small Hankel order.

## Mathematical statement

The paper uses the Bernstein–Widder representation of a completely monotone function: $g(y)=\int_{[0,\infty)}e^{-sy}\,\mathrm{d}\mu(s)$, where $g:(0,\infty)\to\mathbb{R}_{+}$ and $\mu$ is a positive measure. Gamma averaging extracts an order exponent: $\frac{1}{\Gamma(a)}\int_{0}^{\infty}y^{a-1}e^{-y}g(y)\,\mathrm{d}y=\int_{[0,\infty)}(1+s)^{-a}\,\mathrm{d}\mu(s)$. Therefore, for an order-dependent positive response $f(a)$ that has this representation, the Hankel matrix $H^{(K)}(a)$ with entries $H_{ij}=f(a+i+j)$, $0\leq i,j<K$, is positive semidefinite because it is a Gram/moment matrix. Its principal minors satisfy $\det H^{(K)}(a)\geq0$; strict positivity occurs when the representing measure has sufficiently rich support. In the adaptation, $f(a)$ is a neural scalar response sampled at equally spaced order values, and the loss penalizes violations of $H\succeq\epsilon I$ rather than assuming the network is exactly a moment function.

## Key formulas

- $$g(y)=\int_{[0,\infty)}e^{-sy}\,\mathrm{d}\mu(s)$$
- $$\frac{1}{\Gamma(a)}\int_{0}^{\infty}y^{a-1}e^{-y}g(y)\,\mathrm{d}y=\int_{[0,\infty)}(1+s)^{-a}\,\mathrm{d}\mu(s)$$
- $$H^{(K)}_{ij}(a)=f(a+(i+j)\Delta),\qquad 0\leq i,j<K$$
- $$\mathcal{L}_{\mathrm{Hankel}}=\left\|\operatorname{ReLU}\!\left(\epsilon I-H^{(K)}(a)\right)\right\|_{F}^{2}-\lambda\log\det\!\left(H^{(K)}(a)+\epsilon I\right)$$

## Implementation notes

Integrate this at a scalar response head, not inside every hidden layer. Choose an ordered scalar $a$, such as diffusion timestep, attention temperature, augmentation severity, model depth proxy, or the index of a loss-scale schedule. For each minibatch, evaluate the same network response at $K+1$ or $2K$ nearby values $a+r\Delta$, producing positive scalars $f_r=\operatorname{softplus}(u_r)+\delta$, where $u_r$ is the raw response and $\delta>0$ avoids singular matrices. Construct $H_{ij}=f_{i+j}$ for $0\leq i,j<K$. Add either a PSD penalty $\|\operatorname{ReLU}(\epsilon I-H)\|_F^2$ or, when the matrix is numerically positive definite, the barrier $-\lambda\log\det(H+\epsilon I)$. Use $K=3$ initially; compute the eigendecomposition or Cholesky factor in float32 and stop gradients through no quantities. Pseudocode: evaluate $f[0:2K-1]$; set $H[i,j]=f[i+j]$; loss $=L_{task}+\alpha L_{Hankel}$; backpropagate. The paper supplies the moment representation and the determinant-positivity rationale; the empirical quantities are the sampled responses and their eigenvalues. First test on a small CIFAR-10 ResNet or a 125M-parameter language model with an order-dependent calibration/temperature head. Compare no regularizer, pairwise log-convexity, and Hankel $K=3$ at equal compute. Success means fewer oscillations in the response across $a$, improved held-out calibration or robustness, and no more than 1.2x training overhead.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a differentiable K=3 Hankel PSD regularizer and a fixed-seed toy response-fitting comparison against no regularization and pairwise log-convexity. The analytic mixture-of-exponentials check confirmed the valid moment sequence has nonnegative Hankel eigenvalues and that a perturbation creates a negative eigenvalue. In the learning experiment, Hankel regularization had worse test MSE than baseline (0.0001163 vs 0.0000871), did not reduce dense PSD violations (29.9% vs 29.9%), and cost 3.15x runtime, so the promised practical win was not observed.

### Mechanism check

- Verdict: Built a differentiable K=3 Hankel PSD regularizer and a fixed-seed toy response-fitting comparison against no regularization and pairwise log-convexity. The analytic mixture-of-exponentials check confirmed the valid moment sequence has nonnegative Hankel eigenvalues and that a perturbation creates a negative eigenvalue. In the learning experiment, Hankel regularization had worse test MSE than baseline (0.0001163 vs 0.0000871), did not reduce dense PSD violations (29.9% vs 29.9%), and cost 3.15x runtime, so the promised practical win was not observed.
- Confidence: 9/10
- Limitations: This was a small synthetic scalar-response experiment with 18 noisy training points, one seed, K=3, and 700 optimization steps; no CIFAR, diffusion, calibration, multi-seed, FLOP-normalized, or larger-scale neural-network evaluation was performed.

## Artifacts

- [hankel_experiment.py](https://synthcore.org/code/1251/hankel_experiment.py)
- [report.md](https://synthcore.org/code/1251/report.md)
- [results.json](https://synthcore.org/code/1251/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1251)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.