Tangent Brownian symmetry breaking

(1) Integration point: insert this stochastic update after each continuous-depth residual step or after the value/output projection in a transformer. First normalize every token embedding x_{k,i} ∈ R^d to unit norm. For a sequence of m tokens, use one shared symmetric matrix A_k ∈ R^{d×d} at depth step k, but an independent Gaussian vector z_{k,i} for each token. The shared matrix can be implemented as a low-rank symmetric operator to avoid an O(d²) cost: A_k x = U_k(U_k^T x)+V_k(V_k^T x), with U_k,V_k ∈ R^{d×r}; alternatively use a full matrix only in a small pilot experiment. (2) Pseudocode: sample G_k; set A_k=(G_k+G_k^T)/sqrt(2); for each token, compute p_i=A_k x_i−x_i(x_i^T A_k x_i), draw z_i~N(0,I), compute q_i=z_i−x_i(x_i^T z_i), set y_i=x_i+h p_i+gamma sqrt(h) q_i, and set x_i←y_i/(||y_i||₂+epsilon). Add the ordinary deterministic transformer drift either before this update or as h f_theta(x_i), giving y_i=x_i+h f_theta(x_i)+h p_i+gamma sqrt(h)q_i. (3) Mathematics versus estimates: the projection and symmetric-noise construction are exact; h is the Euler-Maruyama depth step, gamma is a tunable forcing amplitude, and r is an engineering approximation. Estimate synchronization and anti-polar behavior empirically using pairwise cosine similarities, the fraction of token pairs with cosine similarity below −0.9, and the number of clusters under spherical k-means. (4) First experiment: train a 6-layer small Transformer on CIFAR-10 patches or WikiText-2 with baseline residual dropout, isotropic Gaussian residual noise, and this tangent-noise block. Sweep gamma ∈ {1e−3, 3e−3, 1e−2, 3e−2} and h matched to the residual scaling. Measure training loss at equal FLOPs, validation accuracy or perplexity, representation cosine-spectrum entropy, gradient norm variance, and robustness to small input perturbations. The hypothesis is an initial rise in anti-polar pairs followed by their disappearance for gamma>0, with reduced representation degeneracy and either improved validation performance or more stable loss curves at the same noise budget.