{
 "artifacts": null,
 "category": "architecture",
 "description": "Project each FFN residual update onto the tangent space of the current token residual direction before adding it to the stream. This preserves the component that changes representation direction while suppressing norm-only motion, which may reduce residual-norm drift and aggregation-induced representation collapse.",
 "formulas_latex": [
  "$$x_i=r_i u_i,\\qquad r_i=\\|x_i\\|,\\qquad u_i=\\frac{x_i}{\\|x_i\\|}\\in\\mathbb{S}^{d-1}.$$",
  "$$\\dot{u}_{i}=\\frac{1}{s_{i}(t)}P_{u_{i}}\\big[A_{i}(U)+\\Phi_{i}(u_{i})\\big],\\qquad P_{u_i}=I-u_i u_i^{\\top}.$$",
  "$$\\Phi_i(u_i)=\\underbrace{\\langle u_i,\\Phi_i(u_i)\\rangle u_i}_{\\text{radial}}+\\underbrace{P_{u_i}\\Phi_i(u_i)}_{\\text{tangential}},\\qquad \\dot r_i=\\langle u_i,F_i\\rangle.$$",
  "$$\\widetilde{\\Phi}(x)=\\left(I-\\frac{xx^{\\top}}{\\|x\\|_2^2+\\varepsilon}\\right)\\Phi(x),\\qquad x_{\\mathrm{out}}=x+\\alpha\\,\\widetilde{\\Phi}(x).$$"
 ],
 "id": 1693,
 "implementation": "1. Integration point: modify the FFN residual branch in a pre-norm Transformer block, after the FFN produces y=FFN(RMSNorm(x)) and immediately before the residual addition. Treat each x[b,t,:] as one token residual vector. Leave attention unchanged for the first experiment, and use the same residual multiplier alpha as the baseline.\n\n2. Pseudocode:\n```text\ninput x[B,T,D]\nh = RMSNorm(x)\ny = FFN(h)\ndenom = sum(x*x, dim=-1, keepdim=True) + eps\nradial_coeff = sum(x*y, dim=-1, keepdim=True) / denom\nif mode == projected:\n    y_used = y - radial_coeff * x\nelif mode == soft_projected:\n    y_used = y - beta * radial_coeff * x\nelse:\n    y_used = y\nx_next = x + alpha * y_used\n```\nUse eps=1e-6 in float32, or scale eps relative to the mean squared activation. In the soft version, beta is fixed at 0.25, 0.5, or 1.0; beta=1 is the hard tangent projection. Also test a norm-preserving variant y_used=y_projected*||y||/(||y_projected||+eps), so directional projection is separated from a reduction in update magnitude.\n\n3. Computed directly from the paper's mathematics: the radial coefficient, projected FFN vector, and tangent-space residual. Log the normalized radial fraction rho=|x^T y|/(||x||||y||+eps), the post-projection orthogonality error |x^T y_used|/(||x||||y_used||+eps), residual norm ||x||, and angular displacement arccos(clamp(x^T x_next/(||x||||x_next||),-1,1)). These diagnostics distinguish whether any gain comes from removing radial motion, changing update scale, or regularizing directions.\n\n4. First cheap experiment: train a 6-layer, 8-head small Transformer on WikiText-103 or a 10k-example OpenWebText subset with baseline FFNs, hard projection, beta=0.5 projection, and norm-preserving projection. Match initialization, batch size, optimizer, parameter count, and training FLOPs. Compare validation perplexity and loss versus optimizer steps, residual-norm drift, angular diversity across tokens, gradient variance, and activation overflow frequency. Then apply the variants to GPT-2-small for one short fine-tuning run to test compatibility with pretrained weights. A positive signal is equal-or-better perplexity with lower norm drift or gradient variance and higher directional diversity; a negative result is a persistent loss increase after equal compute or severe degradation when projection is enabled in all layers.",
 "math_summary": "The paper writes each token residual as x_i=r_i u_i, where r_i=||x_i||\u003e0 is the residual magnitude and u_i=x_i/||x_i|| lies on the unit sphere S^{d-1}. For an ambient update F_i=A_i(U)+Phi_i(u_i), the direction dynamics are dot{u}_i=(1/s_i(t))P_{u_i}[A_i(U)+Phi_i(u_i)], where s_i(t)\u003e0 is the angular-speed scale and P_{u_i}=I-u_i u_i^T is the orthogonal projector onto the tangent space T_{u_i}S^{d-1}. The FFN field decomposes as Phi_i(u_i)=\u003cu_i,Phi_i(u_i)\u003eu_i+P_{u_i}Phi_i(u_i). The first term is radial and changes residual magnitude through dot{r}_i=\u003cu_i,F_i\u003e; the second is tangential and is the only FFN component that directly changes residual direction. The proposed network operation applies the adapted projector P_x=I-xx^T/(||x||^2+epsilon) to the FFN output y, with x the current residual token, y=FFN(Norm(x)), and epsilon\u003e0 for numerical stability. Because the finite-epsilon projector is only approximately orthogonal, measure x^T(P_x y) during experiments and optionally use the exact denominator ||x||^2 when activations are safely nonzero.",
 "math_tags": [
  "geometry",
  "dynamical-systems",
  "linear-algebra"
 ],
 "ml_areas": [
  "transformer",
  "mlp",
  "regularization",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.02071",
  "arxiv_url": "https://arxiv.org/abs/2608.02071",
  "summary_what_math_gives_to_ml": "The paper supplies a useful geometric decomposition of a Transformer residual update into radial magnitude change and tangential direction change on the unit sphere. Its key transferable asset is that only the projected FFN field can alter residual direction, while radial FFN energy primarily changes token norm and may be redundant in pre-normalized or RMS-normalized Transformers. This suggests a minimally invasive projected-FFN architecture or regularizer that removes radial components while preserving the directional contribution responsible for representation movement. A practical diagnostic is to measure residual norm drift and angular diversity before deciding whether projection is beneficial in a given layer.",
  "title": "Feed-Forward Steering in Transformer Residual Dynamics",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 6,
  "usefulness": 6
 },
 "solves": [
  "stability",
  "generalization",
  "accuracy"
 ],
 "title": "Tangential FFN Residuals",
 "url": "https://synthcore.org/idea/1693/tangential-ffn-residuals",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}