# Gaussian-resummed sparse interaction layer - ID: 3163 - Canonical URL: https://synthcore.org/idea/3163/gaussian-resummed-sparse-interaction-layer - API JSON: https://synthcore.org/api/idea/3163.json - API Markdown: https://synthcore.org/api/idea/3163.md - Verification status: unverified - Source: [arXiv:2609.04079](https://arxiv.org/abs/2609.04079) - Category: architecture - Solves: scalability, speedup, stability - ML areas: attention, graph-nn, world-model, loss - Math tags: probability, combinatorics, approximation-theory, numerical-analysis - Ratings: usefulness 5/10; difficulty 7/10; novelty 8/10 ## Idea description Replace a dense weak pairwise interaction over tokens or graph nodes by an analytically resummed Gaussian quadratic term plus explicitly evaluated corrections containing only subsets of size at most R. Choose R from the target approximation error and interaction scale instead of fixing the interaction order heuristically. This is intended for dense global attention, graph interactions, or latent-variable energy models with individually small couplings. ## Mathematical statement The paper considers Gaussian integrals with a possibly indefinite moment matrix J whose entries satisfy |J_ij|=O(1/n), performs linear cancellation and exact quadratic resummation, and makes the remaining coordinate factor vanish to order at least three. Its truncation guarantee is |Tail_{>R}| <= (CR/n)^{R/2}+e^{-cn}, where n is the number of interacting coordinates, R is the largest retained support size, C>0 depends on bounded scalar factors and the weak-coupling regime, and c>0 is a large-deviation rate. It also proves Z(J)=2^n det(I-J)^(-1/2)(1+O_{beta,kappa}(1/n)) for zero-diagonal symmetric J with max_{i,j}|J_ij|<=beta/n and lambda_max(J)<=1-kappa. In the neural adaptation, X is the input feature matrix, J(X) is a learned symmetric zero-diagonal coupling matrix, z is a scalar latent vector, and the quadratic Gaussian base is -1/2 log det(I-J). The non-Gaussian residual corrections c_S are indexed by subsets S of coordinates and are retained only for 1<=|S|<=R. The order-three cancellation is what yields the faster-than-geometric tail rather than a generic first-order truncation bound. ## Key formulas - $$Z(J)=2^{n}\det(I-J)^{-1/2}\left(1+O_{\beta,\kappa}(1/n)\right),\qquad \max_{i,j}|J_{ij}|\leq\frac{\beta}{n},\quad \lambda_{\max}(J)\leq1-\kappa.$$ - $$\left|\mathrm{Tail}_{>R}\right|\leq\left(\frac{CR}{n}\right)^{R/2}+e^{-cn}.$$ - $$R\log\!\left(\frac{en}{R}\right)=O\!\left(\log n+\log\frac{1}{\varepsilon}\right).$$ - $$\widehat{\log Z}(X)=-\frac12\log\det(I-J(X))+\sum_{\substack{S\subseteq[n]\\1\leq |S|\leq R}}c_S(X,J),\qquad \left|\log Z-\widehat{\log Z}\right|\lesssim\left(\frac{CR}{n}\right)^{R/2}+e^{-cn}.$$ ## Implementation notes (1) Integration point: insert the block after token or graph embeddings and before the next residual layer. Given X in R^{n x d}, compute q_i=W_q x_i and define J_ij=(beta/n)tanh(q_i^T q_j/sqrt(d)) for i!=j and J_ii=0; symmetrize J as (J+J^T)/2. Enforce lambda_max(J)<=1-kappa with 3-5 power-iteration steps and the rescaling J <- J*min(1,(1-kappa)/(lambda_max(J)+1e-6)). (2) Pseudocode: compute Q=I-J; obtain base=-0.5*logdet(Q) using Cholesky for n<=256 or stochastic Lanczos for larger n. For every subset S with 1<=|S|<=R, evaluate c_S using Gauss-Hermite quadrature over z_S under the Gaussian covariance Q^{-1}; for each scalar residual potential r_i(z_i), subtract its degree-0, degree-1, and degree-2 Taylor terms before multiplying residual factors, ensuring the correction starts at degree three. Sum base and all c_S, then use the result as a global residual bias or interaction feature. (3) Use the paper's tail expression to choose the smallest R satisfying (C*R/n)^(R/2)