# Tunable Haar-Moment Mixing Regularizer

- ID: 2681
- Canonical URL: https://synthcore.org/idea/2681/tunable-haar-moment-mixing-regularizer
- API JSON: https://synthcore.org/api/idea/2681.json
- API Markdown: https://synthcore.org/api/idea/2681.md
- Verification status: unverified
- Source: [arXiv:2608.21959](https://arxiv.org/abs/2608.21959)
- Category: regularization
- Solves: stability, generalization, sample-efficiency
- ML areas: rnn, ssm, diffusion, regularization
- Math tags: probability, statistics, representation-theory, dynamical-systems
- Ratings: usefulness 5/10; difficulty 4/10; novelty 7/10

## Idea description

Regularize hidden-state trajectories so that their temporal statistics match the moments of an isotropic Haar-distributed state up to order k, while deliberately leaving moments above k unconstrained. Use k as a controllable mixing knob: k=1 or 2 suppresses drift and anisotropic variance, whereas larger k imposes stronger distributional invariance and may remove useful temporal information.

## Mathematical statement

The paper defines the k-th temporal moment of an observable O along a trajectory |\psi(t)\rangle as \(\overline{O^k}=\lim_{T\to\infty}T^{-1}\int_0^Tdt\,\langle\psi(t)|O|\psi(t)\rangle^k=\operatorname{tr}[O^{\otimes k}\rho_\infty^{(k)}]\). Here O is an observable, \(\rho_\infty^{(k)}\) is the time-averaged k-copy state, and \(\rho_{\mathrm{Haar}}^{(k)}\) is the k-copy moment tensor of a Haar-random pure state. Under k-HSE, the temporal moment equals \(\operatorname{tr}[O^{\otimes k}\rho_{\mathrm{Haar}}^{(k)}]\), independent of the initial state. For a neural hidden state \(h_t\in\mathbb R^d\), use random unit probes u and scalar observables \(z_t=u^\top h_t\). Match empirical moments \(\hat m_r(u)=L^{-1}\sum_{t=1}^Lz_t^r\) to a reference isotropic distribution for \(1\le r\le k\). Generate reference unit vectors \(v=g/\|g\|_2\), with \(g\sim\mathcal N(0,I_d)\), and estimate \(m_r^{\mathrm{ref}}(u)=\mathbb E[(u^\top v)^r]\) by Monte Carlo. The penalty is \(\mathcal L_{\mathrm{HME}}=\lambda\sum_{r=1}^k\mathbb E_u[(\hat m_r(u)-m_r^{\mathrm{ref}}(u))^2]\). The important property is truncation at order k: lower-order statistics become initial-state-insensitive, while higher-order information is not explicitly destroyed.

## Key formulas

- $$\overline{O^{k}}=\lim_{T\to\infty}\frac{1}{T}\int_{0}^{T}\mathrm{d}t\,\langle\psi(t)|O|\psi(t)\rangle^{k}=\mathrm{tr}[O^{\otimes k}\rho_{\infty}^{(k)}].$$
- $$\overline{O^{k}}\xrightarrow{k\text{-HSE}}\mathrm{tr}[O^{\otimes k}\rho_{\mathrm{Haar}}^{(k)}].$$
- $$\hat m_r(u)=\frac{1}{L}\sum_{t=1}^{L}(u^{\top}h_t)^r,\qquad \mathcal L_{\mathrm{HME}}=\lambda\sum_{r=1}^{k}\mathbb E_{u\sim\mathcal N(0,I_d)}\left[\hat m_r(u)-m_r^{\mathrm{ref}}(u)\right]^2.$$$
- $$v^{(j)}=\frac{g^{(j)}}{\|g^{(j)}\|_2},\quad g^{(j)}\sim\mathcal N(0,I_d),\qquad m_r^{\mathrm{ref}}(u)\approx\frac{1}{J}\sum_{j=1}^{J}(u^{\top}v^{(j)})^r.$$$

## Implementation notes

Integrate the regularizer after the recurrent or state-space transition, rather than on raw inputs. For each training sequence, collect a contiguous hidden-state block H=(h_1,...,h_L) from the RNN or SSM. Either RMS-normalize H or use \(\tilde h_t=h_t/(\|h_t\|_2+\epsilon)\) if the penalty should be scale invariant. Sample q Gaussian probes u_i and normalize each to unit norm. Precompute J normalized Gaussian reference vectors v_j in the hidden dimension d; these approximate Haar-random directions on the real unit sphere. Compute reference moments \(m_r^{ref}(u_i)=J^{-1}\sum_j(u_i^\top v_j)^r\), empirical moments \(\hat m_r(u_i)=L^{-1}\sum_t(u_i^\top\tilde h_t)^r\), and add \(\lambda q^{-1}\sum_i\sum_{r=1}^k(\hat m_r(u_i)-m_r^{ref}(u_i))^2\) to the task loss. Pseudocode: `H=encoder(x); H=H/(norm(H,dim=-1)+eps); U=normalize(randn(q,d)); V=normalize(randn(J,d)); R=0; for r in 1..k: A=(H @ U.T)**r; B=(V @ U.T)**r; R += mean((A.mean(dim=time)-B.mean(dim=reference))**2); loss=task_loss+lambda*R; backward(loss)`. Start with k=2, then test k=3 and k=4; for a symmetric reference, odd target moments are zero and can be omitted in a first ablation. The paper provides the moment-matching principle, while probe and reference averages are estimated empirically. Use a small GRU and an S4-like SSM on sequential MNIST, permuted MNIST, and a synthetic long-range copy task, comparing against the unregularized model and RMSNorm plus weight decay. Log validation accuracy, hidden covariance condition number, gradient-norm variance, loss spikes, and long-horizon memory accuracy. Success is fewer unstable runs or lower validation loss at equal FLOPs without harming copy accuracy; failure is hidden-state collapse or increasing degradation as k and lambda grow.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.