{
 "artifacts": null,
 "category": "architecture",
 "description": "Construct a symmetric token affinity matrix and approximate only its spectrally outlying token-mixing modes using a small set of sampled landmark columns. The resulting low-rank operator replaces an \\(O(n^2)\\) dense mixer by two skinny matrix multiplications, while the paper's residual guarantee predicts that large-magnitude global interaction modes are preserved.",
 "formulas_latex": [
  "$$\\|A\\|_\\infty\\leq 1,\\qquad s=\\frac{c\\log n}{\\epsilon^{4}\\delta}\\log^{3}\\frac{1}{\\epsilon\\delta}.$$",
  "$$|\\lambda-\\tilde\\lambda|\\leq\\epsilon n,\\qquad \\|Av-\\lambda v\\|_{2}\\leq\\epsilon n,\\qquad \\|v\\|_2=1,\\qquad |\\lambda|\\geq\\epsilon n.$$",
  "$$\\widehat A=CW^{\\dagger}C^{T},\\qquad C=A[:,S],\\quad W=A[S,S],\\quad \\widehat A X=CW^{\\dagger}(C^{T}X).$$",
  "$$\\widehat A=U_r\\operatorname{diag}(\\widehat\\lambda_1,\\ldots,\\widehat\\lambda_r)U_r^T,\\qquad r=\\#\\{k:|\\widehat\\lambda_k|\\geq\\tau\\},\\quad \\tau=\\epsilon n.$$"
 ],
 "id": 1647,
 "implementation": "Integrate this into a transformer as a token-mixing residual branch, either after the value projection or as a replacement for one dense self-attention block. For token matrix X with shape [batch,n,d], first compute a symmetric bounded affinity A implicitly: use normalized tokens Z=X/(||X||_2+10^-6), compute landmark affinities C between every token and s sampled landmarks, and compute W, the s-by-s landmark affinity matrix. Sample S uniformly without replacement during training; for cheaper inference, fix S using a deterministic hash or reservoir sampling. Pseudocode: (1) choose s=ceil(c log(n)/(epsilon^4 delta) log(1/(epsilon delta))^3), clipped to [r_min,n]; (2) compute C_ij=tanh(z_i^T z_{S_j}/sqrt(d)) and W_ij=tanh(z_{S_i}^T z_{S_j}/sqrt(d)); (3) symmetrize W=(W+W^T)/2; (4) compute W=U diag(mu) U^T and set P=U diag(1/mu_k if |mu_k|\u003e=tau_W, else 0) U^T; (5) for values V, output Y=X+alpha C P(C^T V), where alpha is a learned scalar. The paper's mathematical contribution is the sampled-column span and outlier threshold; Nyström is the concrete reconstruction used here because the extracted material does not expose Algorithm (1)'s full estimator. Estimate the approximation residual on validation batches by explicitly forming A for n\u003c=1024 and computing ||(A-Ahat)V||_F/||AV||_F. First test on a 6-layer transformer with 512 and 1024 tokens using CIFAR-100 patch tokens or Long Range Arena, comparing full attention, standard Nyström attention, and this module. Record accuracy, wall-clock time, peak memory, rank, and spectral residual. Success means at least 1.5x lower attention memory or latency with less than one percentage point accuracy loss, while increasing s should reduce the measured residual and preserve the dominant eigenmodes.",
 "math_summary": "The paper assumes a symmetric matrix \\(A\\in\\mathbb R^{n\\times n}\\) with entrywise bound \\(\\|A\\|_\\infty\\le 1\\), and for \\(\\epsilon,\\delta\\in(0,1)\\) samples \\(s=\\frac{c\\log n}{\\epsilon^4\\delta}\\log^3\\frac{1}{\\epsilon\\delta}\\) columns. With probability at least \\(1-\\delta\\), every eigenvalue \\(\\lambda\\) satisfying \\(|\\lambda|\\ge \\epsilon n\\) has an estimated eigenvalue \\(\\tilde\\lambda\\) and normalized vector \\(v\\) obeying \\(|\\lambda-\\tilde\\lambda|\\le \\epsilon n\\) and \\(\\|Av-\\lambda v\\|_2\\le \\epsilon n\\). There are at most \\(1/\\epsilon^2\\) such eigenvalues because \\(\\|A\\|_F\\le n\\). For an implementable neural version, let \\(X\\in\\mathbb R^{n\\times d}\\) be token features, define a bounded symmetric affinity \\(A_{ij}=\\tanh(x_i^Tx_j/\\sqrt d)\\) or a clipped Gaussian kernel, sample landmark indices \\(S\\) of size \\(s\\), form \\(C=A[:,S]\\) and \\(W=A[S,S]\\), and use the Nyström operator \\(\\widehat A=CW^\\dagger C^T\\). Its nonzero eigenvectors lie in \\(\\operatorname{span}(C)\\), matching the paper's column-span construction.",
 "math_tags": [
  "spectral-theory",
  "linear-algebra",
  "random-matrix",
  "numerical-analysis"
 ],
 "ml_areas": [
  "attention",
  "transformer",
  "inference-speedup",
  "memory"
 ],
 "paper": {
  "arxiv_id": "2608.00840",
  "arxiv_url": "https://arxiv.org/abs/2608.00840",
  "summary_what_math_gives_to_ml": "The paper gives a constructive route to recover outlying eigenspaces of a large bounded symmetric matrix from only a small uniformly sampled set of columns, with the approximation living in the span of those columns. The transferable asset is not merely low-rank approximation, but an explicit spectral criterion: components with eigenvalue magnitude at least \\(\\epsilon n\\) can be approximated with residual \\(O(\\epsilon n)\\) using only \\(\\tilde O(\\log n/\\epsilon^4)\\) columns. In neural networks, this suggests replacing dense symmetric token-mixing or kernel operations by a landmark-column spectral module, with the sampling budget selected from the desired residual tolerance and the retained rank selected by the outlier threshold.",
  "title": "Sublinear Time Eigenvector Approximation via Column Sampling",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 5,
  "usefulness": 6
 },
 "solves": [
  "speedup",
  "memory",
  "scalability"
 ],
 "title": "Outlier-Spectral Landmark Attention",
 "url": "https://synthcore.org/idea/1647/outlier-spectral-landmark-attention",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}