Consensus-Safe RoPE Residual Attention

1. Integration point: insert this update inside a deep residual attention stack or an attention-based recurrent state-space model. Apply RoPE only to queries and keys, leave values unrotated as in the paper, and normalize each token state before and after the attention residual. Replace the usual unconstrained residual $h^+=h+\eta\,\mathrm{Attn}(h)$ by the spherical update above, or use it in a stability-control branch while retaining the normal value projection. 2. Pseudocode: normalize token rows $x$; compute $q_i=R_{p_i}x_i$ and $k_i=R_{p_i}x_i$; form $S_{ij}=q_i^\top k_j$ and $A=\operatorname{softmax}(\beta S)$; compute $m_i=\sum_jA_{ij}x_j$ and $v_i=m_i-(x_i^\top m_i)x_i$; update $x_i\leftarrow\operatorname{normalize}(x_i+\eta v_i)$. Every $K$ steps, form $\Pi=\operatorname{diag}(Z)$, compute eigenvalues of $\Pi^{1/2}A\Pi^{-1/2}$, discard the consensus eigenvector, and clip $\eta\leftarrow0.9(2/\lambda_{\max})$. 3. Computed from the paper: the RoPE score construction, exact floor $\alpha=e^{-2\beta}/n$, reversibility transform, and Euler stability criterion. Estimated empirically: the local spectrum when value projections, nonlinearities, masking, or layer normalization are present; use power iteration or Lanczos instead of a full eigendecomposition for long sequences. 4. First cheap experiment: compare a 24-layer residual Transformer with standard RoPE against this controlled block on character-level Penn Treebank or Tiny Shakespeare, using sequence lengths 256 and 1024 and identical parameter counts. Measure loss, token-state angular diameter $D_t=\max_{ij}\arccos(x_i^\top x_j)$, gradient norm, and the observed divergence boundary while sweeping $\eta$. The quantitative prediction is that instability begins near $\eta_c=2/\lambda_{\max}(I-S)$, with the measured threshold within 20% for the toy spherical block; at fixed $\beta$, the worst-case inter-token mixing envelope should decay no slower than approximately $(1-e^{-2\beta})^t$ per idealized averaging step, while increasing $\beta$ should weaken this guaranteed contraction exponentially.