# Bartlett-LKJ Correlated Head Noise

- ID: 1874
- Canonical URL: https://synthcore.org/idea/1874/bartlett-lkj-correlated-head-noise
- API JSON: https://synthcore.org/api/idea/1874.json
- API Markdown: https://synthcore.org/api/idea/1874.md
- Verification status: unverified
- Source: [arXiv:2608.06116](https://arxiv.org/abs/2608.06116)
- Category: regularization
- Solves: accuracy, sample-efficiency, stability
- ML areas: attention, regularization, training
- Math tags: probability, statistics, linear-algebra
- Ratings: usefulness 5/10; difficulty 4/10; novelty 7/10

## Idea description

Replace independent dropout or Gaussian perturbations across attention heads, ensemble members, or diffusion score replicas with a positive-semidefinite correlation matrix sampled from an LKJ distribution. The concentration parameter eta controls whether perturbations are nearly independent or strongly correlated in a controlled way, while the Bartlett construction guarantees a valid covariance without matrix rejection or projection.

## Mathematical statement

The LKJ distribution on the elliptope E_n={C in R^{n x n}: C=C^T, C is positive semidefinite, C_ii=1} has density p(C) proportional to det(C)^(eta-1), with eta>0. A direct row-normalized Bartlett sampler draws A_ij independently from N(0,1) for j<i, draws A_ii^2 from chi-square with degrees of freedom 2 eta+i-2, and normalizes each row B_i=A_i/||A_i||_2. The resulting C=B B^T is an LKJ_n(eta) correlation matrix. For head-wise perturbations epsilon with marginal scale sigma, use epsilon=sigma B z for z~N(0,I_n), giving conditional covariance sigma^2 C. The factor B avoids a separate eigendecomposition or matrix square root.

## Key formulas

- $$p_\eta(C) \propto \det(C)^{\eta-1}, \qquad C\in\mathcal{E}_n=\{C=C^\top,\ C\succeq 0,\ C_{ii}=1\}.$$
- $$A_{ij}\sim\mathcal{N}(0,1)\ (j<i),\qquad A_{ii}^2\sim\chi^2_{\,2\eta+i-2},\qquad B_{i,:}=\frac{A_{i,:}}{\|A_{i,:}\|_2},\qquad C=BB^\top.$$
- $$\varepsilon=\sigma Bz,\qquad z\sim\mathcal{N}(0,I_n),\qquad \operatorname{Cov}(\varepsilon\mid B)=\sigma^2 C.$$

## Implementation notes

Integrate this at the perturbation-generation point rather than inside the core attention matrix multiplication. For a transformer with H heads, set n=H and generate one lower-triangular Bartlett factor B per training step, microbatch, or epoch. Use eta as a fixed hyperparameter or a schedule. Pseudocode: input head outputs h[1:H], noise scale sigma, and eta; initialize A=0; for i=2,...,H, sample A[i,j]~Normal(0,1) for j<i and A[i,i]~sqrt(ChiSquare(2*eta+i-2)); set B[i]=A[i]/norm(A[i]); set B[1,1]=1; sample z~Normal(0,I_H); set eps=sigma*(B@z); replace independent head noise eps_i with eps_i, or use eps_i to scale head dropout logits. For multiple feature dimensions, draw z independently per token and feature group, reusing B across the sequence to create head-level rather than token-level correlation. The exact paper-derived computation is the Bartlett degrees-of-freedom rule and row normalization; no empirical eigendecomposition is needed. Record head-output correlation and gradient variance empirically. First experiment: compare a 6-head, 2-layer Transformer on CIFAR-10 or WikiText-2 using independent dropout versus LKJ-correlated dropout at eta in {0.3,1,3,10}, matched noise variance and FLOPs. Measure validation accuracy or perplexity, gradient variance, and robustness across seeds. Success means improved accuracy or calibration at equal cost, or lower seed-to-seed variance without optimization instability.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.