# Conservative Sparse Mortar Cross-Attention

- ID: 459
- Canonical URL: https://synthcore.org/idea/459/conservative-sparse-mortar-cross-attention
- API JSON: https://synthcore.org/api/idea/459.json
- API Markdown: https://synthcore.org/api/idea/459.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.06045](https://arxiv.org/abs/2607.06045)
- Category: architecture
- Solves: stability, scalability, accuracy
- ML areas: transformer, graph-nn, attention, world-model
- Math tags: convex-analysis, numerical-analysis, geometry, linear-algebra
- Ratings: usefulness 7/10; difficulty 5/10; novelty 8/10

## Idea description

Replace dense coarse-to-fine cross-attention at multiresolution interfaces with a sparse, nonnegative overlap operator whose weighted feature average is exactly conserved between the two resolutions. Use this operator as a low-order path and blend it with an unrestricted neural cross-attention path through a convex limiter that keeps features inside a box or simplex domain. The construction is especially suitable for adaptive token grids, hierarchical graph neural networks, neural operators, and world models with dynamically refined spatial cells.

## Mathematical statement

The paper defines physical interface weights by $$\omega_{i,S}^{-}=\int_S\varphi_i^{-}\,\mathrm{d}s$$ and $$\omega_{j,S}^{+}=\int_S\varphi_j^{+}\,\mathrm{d}s$$, where $$\varphi_i^{-}$$ and $$\varphi_j^{+}$$ are basis functions on the two sides of a nonconforming interface $$S$$. It introduces nonnegative characteristic-function weights $$\psi_i^{-},\psi_j^{+}$$ satisfying the partition-of-unity identities $$\sum_i\psi_i^{-}\equiv1$$ and $$\sum_j\psi_j^{+}\equiv1$$. Their overlap weights $$\widetilde\omega_{ij}^{S}$$ satisfy the exact marginal identities $$\sum_i\widetilde\omega_{ij}^{S}=\omega_{j,S}^{+}$$ and $$\sum_j\widetilde\omega_{ij}^{S}=\omega_{i,S}^{-}$$. For neural features, let $$x_i^{-}\in\mathbb R^c$$ be source features, $$y_j^{+}\in\mathbb R^c$$ target features, and let $$\omega_i^{-},\omega_j^{+}>0$$ be token masses. Define $$p_{ij}=\widetilde\omega_{ij}/\omega_j^{+}$$ and $$q_{ij}=\widetilde\omega_{ij}/\omega_i^{-}$$. Nonnegativity and the marginal identities imply $$\sum_i p_{ij}=1$$, $$\sum_jq_{ij}=1$$, and the weighted conservation law $$\sum_j\omega_j^{+}(\sum_i p_{ij}x_i^{-})=\sum_i\omega_i^{-}x_i^{-}$$. The safe low-order output is $$y_j^{L}=\sum_i p_{ij}x_i^{-}$$. A high-order neural output $$y_j^{H}$$ is blended as $$y_j=y_j^{L}+\alpha_j(y_j^{H}-y_j^{L})$$, with the largest $$\alpha_j\in[0,1]$$ such that $$y_j\in\mathcal G$$, where $$\mathcal G$$ is a chosen convex invariant set such as a coordinate box or probability simplex.

## Key formulas

- $$\omega_{i,S}^{-}=\int_{S}\varphi_{i}^{-}\,\mathrm{d}s,\qquad \omega_{j,S}^{+}=\int_{S}\varphi_{j}^{+}\,\mathrm{d}s$$
- $$\sum_{i\in\mathcal{N}_{S}^{-}}\psi_i^{-}\equiv1,\qquad \sum_{j\in\mathcal{N}_{S}^{+}}\psi_j^{+}\equiv1$$
- $$\sum_i\widetilde\omega_{ij}^{S}=\omega_{j,S}^{+},\qquad \sum_j\widetilde\omega_{ij}^{S}=\omega_{i,S}^{-}$$
- $$y_j^{L}=\sum_{i\in\mathcal N^-}p_{ij}x_i^{-},\quad p_{ij}=\frac{\widetilde\omega_{ij}}{\omega_j^{+}},\quad y_j=y_j^{L}+\alpha_j(y_j^{H}-y_j^{L}),\quad \alpha_j=\max\{\alpha\in[0,1]:y_j^{L}+\alpha(y_j^{H}-y_j^{L})\in\mathcal G\}$$

## Implementation notes

Integrate this at every coarse-to-fine or fine-to-coarse interface in a hierarchical Transformer, adaptive graph neural network, or patch-based neural operator. (1) Exact integration point: before the fine-level residual block, replace dense cross-attention value aggregation with a sparse mortar aggregation; retain ordinary attention as the high-order branch. Assign each source and target token a positive mass, such as patch area, voxel volume, graph-cell volume, or token importance. (2) Pseudocode: construct an overlap list E={(i,j)} using geometric intersection, parent-child ancestry, or local nearest neighbors; compute nonnegative overlap masses w[i,j]; set p[i,j]=w[i,j]/mass_target[j]; compute yL[j]=sum_i p[i,j]*x[i]; compute yH=CrossAttention(Q_target,K_source,V_source); set d=yH-yL; choose the largest alpha in [0,1] for which yL+alpha*d remains in the chosen convex domain; output y=yL+alpha*d. (3) The paper-derived quantities are nonnegative sparse overlaps, partition-of-unity normalization, and exact weighted marginals. Estimate masses from geometry or set them to one for ordinary tokens; estimate alpha analytically for a box using per-coordinate bounds, or use bisection for a simplex or learned convex domain. For a box [l_k,u_k], initialize alpha=1 and for each coordinate solve the interval constraints on yL_k+alpha*d_k, taking the minimum feasible upper bound. (4) First experiment: train a small 2D adaptive neural operator or 4-level vision Transformer on CIFAR-100 or a Burgers/PDE surrogate, comparing dense cross-attention, sparse unnormalized attention, and this conservative module at equal hidden size. Measure validation loss, activation-bound violations, gradient norm spikes, and wall-clock scaling as the fine/coarse token ratio grows. Success is lower instability and equal-or-better accuracy with at least 2x fewer cross-interface edges, while the weighted mean feature error stays near numerical precision.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built and numerically verified a sparse nonnegative overlap transport with analytic box limiting. Weighted conservation error was at floating-point precision (5.55e-16), the limiter transition matched alpha=min(1,gamma*/gamma) exactly, and local overlap edges scaled linearly while dense edges scaled quadratically. The conservative path achieved 32x–256x fewer edges; sparse CPU aggregation was faster at smaller tested sizes but slightly slower at the largest size, so a universal speed win is not established.

### Mechanism check

- Verdict: Built and numerically verified a sparse nonnegative overlap transport with analytic box limiting. Weighted conservation error was at floating-point precision (5.55e-16), the limiter transition matched alpha=min(1,gamma*/gamma) exactly, and local overlap edges scaled linearly while dense edges scaled quadratically. The conservative path achieved 32x–256x fewer edges; sparse CPU aggregation was faster at smaller tested sizes but slightly slower at the largest size, so a universal speed win is not established.
- Confidence: 9/10
- Limitations: Only a regular 1-D geometric toy grid was tested; no learned hierarchical Transformer, PDE surrogate, adaptive mesh, GPU implementation, or training loss comparison was performed. A convex blend preserves the box but generally does not preserve weighted feature conservation when the unrestricted high-order branch receives alpha greater than zero.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.93875
- Paired wins: 2/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/232/bench_report.json)
- [report.md](https://synthcore.org/code/232/report.md)
- [report_bench_2026-08-30T213229.md](https://synthcore.org/code/232/report_bench_2026-08-30T213229.md)
- [results.json](https://synthcore.org/code/232/results.json)
- [run_experiment.py](https://synthcore.org/code/232/run_experiment.py)
- [stage2_bench.py](https://synthcore.org/code/232/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/232)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.