# Histogram-Preserving Variation Projection

- ID: 399
- Canonical URL: https://synthcore.org/idea/399/histogram-preserving-variation-projection
- API JSON: https://synthcore.org/api/idea/399.json
- API Markdown: https://synthcore.org/api/idea/399.md
- Verification status: unverified
- Source: [arXiv:2607.03450](https://arxiv.org/abs/2607.03450)
- Category: architecture
- Solves: stability, generalization
- ML areas: ssm, mlp, regularization
- Math tags: analysis, measure-theory, optimization
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Insert a rearrangement operation on scalar feature maps sampled along an ordered coordinate such as time, spatial position, or a neural-field input grid. The operation sorts values into non-increasing order, preserving the empirical histogram exactly while provably not increasing the Riesz fractional variation in the ideal one-dimensional continuous setting.

## Mathematical statement

For an interval I=[a,b], the Riesz (p,alpha)-variation is \(\mathcal{V}_{p}^{\alpha}(f;I)=\sup_{\mathcal{I}}(\sum_{I_j\in\mathcal{I}}({\rm osc}(f;I_j)/|I_j|^{\alpha})^p)^{1/p}\), where \(\mathcal{I}\) ranges over families of pairwise non-overlapping subintervals, \({\rm osc}(f;E)=\sup_{x\in E}f(x)-\inf_{x\in E}f(x)\), \(1\le p<\infty\), and \(0\le\alpha\le 1/p'\) with \(p'=p/(p-1)\) for \(p>1\) and \(1'=\infty\). If \(f^*\) is the non-increasing rearrangement of f, then the paper proves \(\mathcal{V}_{p}^{\alpha}(f^*;I)\leq\mathcal{V}_{p}^{\alpha}(f;I)\). For a sampled vector \(z\in\mathbb{R}^n\), use the discrete rearrangement \(z^*=\operatorname{sort}(z,\mathrm{descending})\), and estimate variation over contiguous index intervals by \(\widehat{\mathcal V}_{p}^{\alpha}(z)=\max_{\mathcal P}(\sum_{[l,r]\in\mathcal P}((\max_{i\in[l,r]}z_i-\min_{i\in[l,r]}z_i)/(r-l+1)^\alpha)^p)^{1/p}\). The continuous theorem motivates the projection, while the exact finite-sample guarantee must be checked empirically or established separately for the chosen discrete estimator.

## Key formulas

- $$\mathcal{V}_{p}^{\alpha}(f;I)=\sup_{\mathcal{I}}\left(\sum_{I_{j}\in\mathcal{I}}\left(\frac{{\rm osc}(f;I_{j})}{|I_{j}|^{\alpha}}\right)^{p}\right)^{1/p}$$
- $${\rm osc}(f;E)=\sup_{x\in E}f(x)-\inf_{x\in E}f(x)$$
- $$\mathcal{V}_{p}^{\alpha}(f^{*};I)\leq\mathcal{V}_{p}^{\alpha}(f;I),\qquad 1\leq p<\infty,\quad 0\leq\alpha\leq 1-\frac{1}{p}$$
- $$z^{*}=\operatorname{sort}(z,\mathrm{descending}),\qquad \widehat{\mathcal V}_{p}^{\alpha}(z)=\max_{\mathcal P}\left[\sum_{[l,r]\in\mathcal P}\left(\frac{\max_{i\in[l,r]}z_i-\min_{i\in[l,r]}z_i}{(r-l+1)^\alpha}\right)^p\right]^{1/p}$$

## Implementation notes

(1) Integration point: apply the operation to selected scalar channels of a tensor with a meaningful ordered axis, such as the time axis of an SSM or temporal MLP, or the coordinate axis of a one-dimensional neural field. For X with shape [batch, channels, length], compute a sorted version along the last axis and mix it into the original activation. (2) Pseudocode: `z=X[b,c,:]; order=argsort(z,descending=True); zstar=z[order]; Y[b,c,:]=(1-lambda)*z+lambda*zstar`, where lambda is ramped from 0 to 1 or applied only to a subset of channels. Hard sorting has discontinuous rank changes, so use stop-gradient through the permutation, a straight-through estimator, or a differentiable soft-sort relaxation. (3) Compute the paper's rearrangement exactly for each finite vector, but estimate the continuous variation with a discrete interval estimator. For short sequences enumerate all contiguous interval partitions using dynamic programming; for long sequences sample random multiscale partitions. Compare variation before and after the layer and verify that the empirical activation histogram is unchanged. (4) First experiment: train a small temporal MLP or SSM on synthetic noisy sequence classification and a real 1D benchmark, comparing no projection, hard sorting, and soft sorting at equal parameter count and FLOPs. Record validation accuracy, robustness to temporal noise, calibration, empirical fractional variation, and gradient norm variance. Success means lower variation and improved noise robustness without a substantial clean-accuracy loss; failure is positional-information destruction or unstable gradients.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.