✗ Failed on benchmark 2026

Conservative Sparse Mortar Cross-Attention

Implementation & benchmark of arXiv:2607.06045 — Invariant-domain-preserving limiting with Adaptive Mesh Refinement for Legendre-Gauss-Lobatto Discontinuous Galerkin Spectral Element Methods

Usefulness7/10

Difficulty5/10

Novelty8/10

Architecture Stability Scalability Accuracy Transformer Graph nn Attention World model Convex analysis Numerical analysis Geometry Linear algebra

Source paper: Invariant-domain-preserving limiting with Adaptive Mesh Refinement for Legendre-Gauss-Lobatto Discontinuous Galerkin Spectral Element Methods arXiv:2607.06045 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace dense coarse-to-fine cross-attention at multiresolution interfaces with a sparse, nonnegative overlap operator whose weighted feature average is exactly conserved between the two resolutions. Use this operator as a low-order path and blend it with an unrestricted neural cross-attention path through a convex limiter that keeps features inside a box or simplex domain. The construction is especially suitable for adaptive token grids, hierarchical graph neural networks, neural operators, and world models with dynamically refined spatial cells.

Formulas

$$\omega_{i,S}^{-}=\int_{S}\varphi_{i}^{-}\,\mathrm{d}s,\qquad \omega_{j,S}^{+}=\int_{S}\varphi_{j}^{+}\,\mathrm{d}s$$

$$\sum_{i\in\mathcal{N}_{S}^{-}}\psi_i^{-}\equiv1,\qquad \sum_{j\in\mathcal{N}_{S}^{+}}\psi_j^{+}\equiv1$$

$$\sum_i\widetilde\omega_{ij}^{S}=\omega_{j,S}^{+},\qquad \sum_j\widetilde\omega_{ij}^{S}=\omega_{i,S}^{-}$$

$$y_j^{L}=\sum_{i\in\mathcal N^-}p_{ij}x_i^{-},\quad p_{ij}=\frac{\widetilde\omega_{ij}}{\omega_j^{+}},\quad y_j=y_j^{L}+\alpha_j(y_j^{H}-y_j^{L}),\quad \alpha_j=\max\{\alpha\in[0,1]:y_j^{L}+\alpha(y_j^{H}-y_j^{L})\in\mathcal G\}$$

Mathematical statement

The paper defines physical interface weights by $$\omega_{i,S}^{-}=\int_S\varphi_i^{-}\,\mathrm{d}s$$ and $$\omega_{j,S}^{+}=\int_S\varphi_j^{+}\,\mathrm{d}s$$, where $$\varphi_i^{-}$$ and $$\varphi_j^{+}$$ are basis functions on the two sides of a nonconforming interface $$S$$. It introduces nonnegative characteristic-function weights $$\psi_i^{-},\psi_j^{+}$$ satisfying the partition-of-unity identities $$\sum_i\psi_i^{-}\equiv1$$ and $$\sum_j\psi_j^{+}\equiv1$$. Their overlap weights $$\widetilde\omega_{ij}^{S}$$ satisfy the exact marginal identities $$\sum_i\widetilde\omega_{ij}^{S}=\omega_{j,S}^{+}$$ and $$\sum_j\widetilde\omega_{ij}^{S}=\omega_{i,S}^{-}$$. For neural features, let $$x_i^{-}\in\mathbb R^c$$ be source features, $$y_j^{+}\in\mathbb R^c$$ target features, and let $$\omega_i^{-},\omega_j^{+}>0$$ be token masses. Define $$p_{ij}=\widetilde\omega_{ij}/\omega_j^{+}$$ and $$q_{ij}=\widetilde\omega_{ij}/\omega_i^{-}$$. Nonnegativity and the marginal identities imply $$\sum_i p_{ij}=1$$, $$\sum_jq_{ij}=1$$, and the weighted conservation law $$\sum_j\omega_j^{+}(\sum_i p_{ij}x_i^{-})=\sum_i\omega_i^{-}x_i^{-}$$. The safe low-order output is $$y_j^{L}=\sum_i p_{ij}x_i^{-}$$. A high-order neural output $$y_j^{H}$$ is blended as $$y_j=y_j^{L}+\alpha_j(y_j^{H}-y_j^{L})$$, with the largest $$\alpha_j\in[0,1]$$ such that $$y_j\in\mathcal G$$, where $$\mathcal G$$ is a chosen convex invariant set such as a coordinate box or probability simplex.

Implementation notes

Integrate this at every coarse-to-fine or fine-to-coarse interface in a hierarchical Transformer, adaptive graph neural network, or patch-based neural operator. (1) Exact integration point: before the fine-level residual block, replace dense cross-attention value aggregation with a sparse mortar aggregation; retain ordinary attention as the high-order branch. Assign each source and target token a positive mass, such as patch area, voxel volume, graph-cell volume, or token importance. (2) Pseudocode: construct an overlap list E={(i,j)} using geometric intersection, parent-child ancestry, or local nearest neighbors; compute nonnegative overlap masses w[i,j]; set p[i,j]=w[i,j]/mass_target[j]; compute yL[j]=sum_i p[i,j]*x[i]; compute yH=CrossAttention(Q_target,K_source,V_source); set d=yH-yL; choose the largest alpha in [0,1] for which yL+alpha*d remains in the chosen convex domain; output y=yL+alpha*d. (3) The paper-derived quantities are nonnegative sparse overlaps, partition-of-unity normalization, and exact weighted marginals. Estimate masses from geometry or set them to one for ordinary tokens; estimate alpha analytically for a box using per-coordinate bounds, or use bisection for a simplex or learned convex domain. For a box [l_k,u_k], initialize alpha=1 and for each coordinate solve the interval constraints on yL_k+alpha*d_k, taking the minimum feasible upper bound. (4) First experiment: train a small 2D adaptive neural operator or 4-level vision Transformer on CIFAR-100 or a Burgers/PDE surrogate, comparing dense cross-attention, sparse unnormalized attention, and this conservative module at equal hidden size. Measure validation loss, activation-bound violations, gradient norm spikes, and wall-clock scaling as the fine/coarse token ratio grows. Success is lower instability and equal-or-better accuracy with at least 2x fewer cross-interface edges, while the weighted mean feature error stays near numerical precision.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and numerically verified a sparse nonnegative overlap transport with analytic box limiting. Weighted conservation error was at floating-point precision (5.55e-16), the limiter transition matched alpha=min(1,gamma*/gamma) exactly, and local overlap edges scaled linearly while dense edges scaled quadratically. The conservative path achieved 32x–256x fewer edges; sparse CPU aggregation was faster at smaller tested sizes but slightly slower at the largest size, so a universal speed win is not established.

Agent confidence: 9/10

Baseline: Dense random cross-attention used 4,096–262,144 edges and showed weighted mean drift from 9.18e-3 to 9.58e-4; timing was 0.265–0.335 ms per repeated batch.
Idea: Exact overlap transport used 128–1,024 edges, giving 32x–256x edge reduction and conservation error <=5.55e-16. The limiter had zero prediction error and zero box violations; sparse timing was 0.124–0.349 ms.
Limiter prediction: Observed alpha exactly matched predicted min(1,gamma*/gamma) for gamma/gamma* in {0.25,0.5,1,2,4,8}.
Sparsity prediction: Observed sparse edge counts were exactly proportional to target tokens for the tested 1-D local-overlap grids, versus source-times-target dense edges.

Limitations: Only a regular 1-D geometric toy grid was tested; no learned hierarchical Transformer, PDE surrogate, adaptive mesh, GPU implementation, or training loss comparison was performed. A convex blend preserves the box but generally does not preserve weighted feature conservation when the unrestricted high-order branch receives alpha greater than zero.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.06045) Aug 30, 2026

Ahead of the baseline (−0.0%) but not statistically significant (p=0.9387, wins 2 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 1.4083
Idea mean: 1.4083
Effect (Δ): -0 (−0.0%; negative = idea better)
Wins: 2 / 8 paired seeds
p-value: 0.9387 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

The completed run used a reduced budget of 3 epochs with 160 training and 80 test samples because the initial 12-epoch run exceeded 40 minutes. Only the registered built-in sequence track was tested; no PDE/adaptive-grid, vision, dynamics, longer-training, GPU-scaling, or wall-clock scaling study was run.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 10/10

The conservative sparse mortar variant was evaluated end-to-end against a matched dense-attention transformer on the registered sequence track. It produced essentially the same test MSE as the tuned baseline, with no significant win (p=0.93875), and its trained-model mechanism signature was not confirmed under the selected tolerance.

Baseline: Best lr=0.001; full 8-seed test MSE mean=1.4082962871, std=0.3481744532.
Idea: Best lr=0.001; full 8-seed test MSE mean=1.4082937241, std=0.3481736322; paired delta=-2.563e-06; p=0.93875; 2/8 wins.

Benchmark result