arXiv:2608.02071 2026 Architecture 1 ideas extracted · analyzed Aug 31, 2026
The paper supplies a useful geometric decomposition of a Transformer residual update into radial magnitude change and tangential direction change on the unit sphere. Its key transferable asset is that only the projected FFN field can alter residual direction, while radial FFN energy primarily changes token norm and may be redundant in pre-normalized or RMS-normalized Transformers. This suggests a minimally invasive projected-FFN architecture or regularizer that removes radial components while preserving the directional contribution responsible for representation movement. A practical diagnostic is to measure residual norm drift and angular diversity before deciding whether projection is beneficial in a given layer.
Project each FFN residual update onto the tangent space of the current token residual direction before adding it to the stream. This preserves the component that changes representation direction while suppressing norm-only motion, which may reduce residual-norm drift and aggregation-induced representation collapse.