# Tangent Brownian symmetry breaking

- ID: 2409
- Canonical URL: https://synthcore.org/idea/2409/tangent-brownian-symmetry-breaking
- API JSON: https://synthcore.org/api/idea/2409.json
- API Markdown: https://synthcore.org/api/idea/2409.md
- Verification status: unverified
- Source: [arXiv:2608.16664](https://arxiv.org/abs/2608.16664)
- Category: dynamics
- Solves: stability, generalization, accuracy
- ML areas: transformer, training-dynamics, regularization
- Math tags: stochastic-processes, geometry, dynamical-systems, probability
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Replace ordinary isotropic residual noise in a normalized continuous-depth block with projected Brownian forcing on the unit sphere. Apply a shared random symmetric quadratic drift to all tokens, plus a small token-specific tangent perturbation; the shared term preserves structured antipodal dynamics while the independent term removes persistent symmetry and cluster degeneracy. This is intended as a controlled stochastic regularizer, not merely additive Gaussian noise.

## Mathematical statement

The paper studies the Stratonovich SDE dX_t = P_{X_t} ∂Q_t X_t + γ P_{X_t} ∂W_t on S^{n−1}, where X_t ∈ R^n has ||X_t||₂=1, P_X = I − XXᵀ is the orthogonal projection onto the tangent space T_X S^{n−1}, Q_t = (B_t+B_tᵀ)/√2 is a symmetric matrix Brownian motion formed from independent scalar Brownian motions B_t^{ij}, W_t is an independent R^n-valued Brownian motion, and γ ≥ 0 controls the symmetry-breaking forcing. The first term is multiplicative shared quadratic noise: for an increment dQ, the vector dQ X is projected onto the tangent space. The second term is independent isotropic tangent Brownian forcing. The geometric property doing the work is P_X X = 0, which keeps the continuous process on the sphere; the paper's qualitative result is that γ=0 retains partial synchronization caused by RQF symmetries, whereas any γ>0 eventually produces full synchronization after an initial anti-polar two-cluster phase. In a neural block, use the same construction with a shared symmetric Gaussian matrix A_k for all tokens and independent Gaussian vectors z_{k,i} for each token.

## Key formulas

- $$\mathrm{d}X_t=P_{X_t}\,\partial Q_tX_t+\gamma P_{X_t}\,\partial W_t,\qquad P_X=I-XX^T,$$
- $$Q_t=\frac{1}{\sqrt{2}}(B_t+B_t^T),$$
- $$x_{k+1}=\operatorname{Normalize}\!\left(x_k+hP_{x_k}A_kx_k+\gamma\sqrt{h}\,P_{x_k}z_k\right),\quad A_k=\frac{G_k+G_k^T}{\sqrt{2}},\quad G_{k,ab}\sim\mathcal N(0,1),\ z_k\sim\mathcal N(0,I),$$
- $$P_xv=v-x(x^Tv),\qquad \operatorname{Normalize}(u)=\frac{u}{\|u\|_2+\varepsilon}.$$

## Implementation notes

(1) Integration point: insert this stochastic update after each continuous-depth residual step or after the value/output projection in a transformer. First normalize every token embedding x_{k,i} ∈ R^d to unit norm. For a sequence of m tokens, use one shared symmetric matrix A_k ∈ R^{d×d} at depth step k, but an independent Gaussian vector z_{k,i} for each token. The shared matrix can be implemented as a low-rank symmetric operator to avoid an O(d²) cost: A_k x = U_k(U_k^T x)+V_k(V_k^T x), with U_k,V_k ∈ R^{d×r}; alternatively use a full matrix only in a small pilot experiment. (2) Pseudocode: sample G_k; set A_k=(G_k+G_k^T)/sqrt(2); for each token, compute p_i=A_k x_i−x_i(x_i^T A_k x_i), draw z_i~N(0,I), compute q_i=z_i−x_i(x_i^T z_i), set y_i=x_i+h p_i+gamma sqrt(h) q_i, and set x_i←y_i/(||y_i||₂+epsilon). Add the ordinary deterministic transformer drift either before this update or as h f_theta(x_i), giving y_i=x_i+h f_theta(x_i)+h p_i+gamma sqrt(h)q_i. (3) Mathematics versus estimates: the projection and symmetric-noise construction are exact; h is the Euler-Maruyama depth step, gamma is a tunable forcing amplitude, and r is an engineering approximation. Estimate synchronization and anti-polar behavior empirically using pairwise cosine similarities, the fraction of token pairs with cosine similarity below −0.9, and the number of clusters under spherical k-means. (4) First experiment: train a 6-layer small Transformer on CIFAR-10 patches or WikiText-2 with baseline residual dropout, isotropic Gaussian residual noise, and this tangent-noise block. Sweep gamma ∈ {1e−3, 3e−3, 1e−2, 3e−2} and h matched to the residual scaling. Measure training loss at equal FLOPs, validation accuracy or perplexity, representation cosine-spectrum entropy, gradient norm variance, and robustness to small input perturbations. The hypothesis is an initial rise in anti-polar pairs followed by their disappearance for gamma>0, with reduced representation degeneracy and either improved validation performance or more stable loss curves at the same noise budget.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.