# Manifold-Adaptive Positive Quadrature Attention

- ID: 3121
- Canonical URL: https://synthcore.org/idea/3121/manifold-adaptive-positive-quadrature-attention
- API JSON: https://synthcore.org/api/idea/3121.json
- API Markdown: https://synthcore.org/api/idea/3121.md
- Verification status: unverified
- Source: [arXiv:2609.03068](https://arxiv.org/abs/2609.03068)
- Category: architecture
- Solves: speedup, memory, accuracy
- ML areas: attention, graph-nn, inference-speedup, world-model
- Math tags: optimization, convex-analysis, approximation-theory, geometry
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Replace dense aggregation over patches, points, or graph neighbors with a state-dependent positive quadrature rule. A latent encoder produces the current manifold coordinate z, a small controller chooses a subset of entities, and a convex redistribution step assigns nonnegative weights so that the subset preserves the full aggregation on a learned feature basis. The method is especially suitable for neural operators and vision attention where many spatial entities are redundant along the data manifold.

## Mathematical statement

The paper represents the full state through an affine reduced coordinate map, $\bm{d}^{\mathrm{FE}}=\bm{T}\bm{d}+\bm{\hat d}$, where $\bm{d}^{\mathrm{FE}}\in\mathbb{R}^{N_{dof}}$ is the full state, $\bm{d}\in\mathbb{R}^{N}$ is the reduced coordinate, $\bm{T}$ is a lifting matrix, and $\bm{\hat d}$ contains prescribed data. Its full residual is $\mathbf{R}^{\mathrm{FE}}(\bm d;\bm\mu)=\mathbf F^{\mathrm{int}}(\bm d;\bm\mu)-\mathbf F^{\mathrm{ext}}(\bm\mu)$. MAW–ECM starts from a feasible fixed-weight empirical cubature rule and greedily removes entities; after each proposed removal it solves a convex quadratic weight-redistribution problem subject to local matching conditions and positivity. The neural adaptation treats each entity $i$ as carrying a feature vector $v_i\in\mathbb{R}^{p}$, uses a latent state $z\in\mathbb{R}^{r}$, and defines a state-conditioned basis $\phi_i(z)\in\mathbb{R}^{q}$, for example $\phi_i(z)=\mathrm{MLP}_{\phi}([e_i,z])$. The full target moments are $b(z)=\sum_{i=1}^{M}a_i\phi_i(z)$, where $a_i>0$ are baseline area, patch, or degree weights. For a selected subset $S$, adaptive positive quadrature weights solve $\min_{w\ge 0}\|A_S(z)w-b(z)\|_2^2+\lambda\|w-a_S\|_2^2$, where $A_S(z)=[\phi_i(z)]_{i\in S}$ and $a_S$ is the vector of baseline weights. The sparse aggregation is $y_S(z)=\sum_{i\in S}w_i(z)v_i$, while the dense reference is $y(z)=\sum_{i=1}^{M}a_iv_i$. Positivity prevents cancellation and the moment constraint controls approximation error on the learned nonlinear manifold.

## Key formulas

- $$\bm{d}^{\mathrm{FE}}=\bm{T}\bm{d}+\bm{\hat{d}}$$
- $$\mathbf{R}^{\mathrm{FE}}(\bm{d};\bm{\mu})=\mathbf{F}^{\mathrm{int}}(\bm{d};\bm{\mu})-\mathbf{F}^{\mathrm{ext}}(\bm{\mu})$$
- $$b(z)=\sum_{i=1}^{M}a_i\phi_i(z),\qquad A_S(z)=[\phi_i(z)]_{i\in S}$$
- $$w_S(z)=\arg\min_{w\ge 0}\left\|A_S(z)w-b(z)\right\|_2^2+\lambda\left\|w-a_S\right\|_2^2,\qquad y_S(z)=\sum_{i\in S}w_i(z)v_i$$

## Implementation notes

(1) Integration point: apply this to one expensive aggregation in a neural operator, point-cloud network, graph message-passing layer, or vision transformer. Let the input contain M entities with values v_i and positional or graph descriptors e_i. Compute a global latent z from a cheap pooled encoder or the previous layer. Do not use it initially for standard query-key attention; use it to select and quadrature-weight entities before the expensive value projection or message computation. (2) Pseudocode: initialize S with all entities or a fixed-weight coreset; compute phi_i=MLP_phi([e_i,z]) and b=sum_i a_i phi_i; while |S|>K, tentatively remove each candidate i, solve w=argmin_{w>=0} ||A_{S\{i}}w-b||^2+lambda||w-a_{S\{i}}||^2, and permanently remove the candidate with the smallest residual increase; at inference solve the final nonnegative quadratic program and return y=sum_{i in S} w_i v_i. For differentiable training, use a straight-through top-K mask and backpropagate through the unconstrained quadratic-program solution, or use a differentiable nonnegative solver. (3) Compute b and phi from the current latent z; estimate the subset offline on calibration batches, then train the controller and basis by minimizing task loss plus gamma||A_Sw-b||^2. The paper's mathematical contribution is the positive convex redistribution and greedy pruning; the basis, latent encoder, and K are engineering choices. (4) First experiment: use a 2D advection or Darcy neural operator and a small ViT on CIFAR-10. Compare dense aggregation, fixed top-K attention, and this method at K/M in {1/4,1/16,1/64}. Measure validation error, aggregation FLOPs, wall-clock latency, and moment residual. Success is at least 4x lower aggregation cost with less than 1 percentage point accuracy loss, plus stable nonnegative weights and lower error than random or fixed-weight selection at the same K.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.