Samplet-compressed kernel interaction layer
Implementation & benchmark of arXiv:2608.24283 — Samplet compression for conditionally positive definite kernels and universal Kriging
Source paper: Samplet compression for conditionally positive definite kernels and universal Kriging arXiv:2608.24283 ⓘ · analyzed Aug 29, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace a dense coordinate-kernel interaction among N points by an orthogonal samplet transform with a sparse detail-detail matrix and a small polynomial branch. Detail basis vectors have vanishing moments, so smooth low-frequency behavior is represented by a few polynomial coefficients while localized residual interactions become sparse in the transformed domain.
Formulas
Mathematical statement
Let P \in \mathbb{R}^{N imes r} contain evaluations of all monomials in \mathcal{P}_q at the data sites, where r=\dim(\mathcal{P}_q), and let Q \in \mathbb{R}^{N imes(N-r)} have orthonormal columns satisfying P^{\mathsf T}Q=0 and Q^{\mathsf T}Q=I. A conditionally positive definite kernel matrix K obeys c^{\mathsf T}Kc>0 for every nonzero c with P^{\mathsf T}c=0, hence Q^{\mathsf T}KQ is symmetric positive definite. The samplet basis constructs Q hierarchically from detail vectors with vanishing moments \sum_i q_i x_i^\alpha=0 for every multi-index \alpha with \|\alpha\|_1\le q. For asymptotically smooth kernels, the transformed matrix \widetilde K=Q^{\mathsf T}KQ is quasi-sparse, so entries below a level- and distance-dependent threshold can be dropped. The neural layer uses the approximation K H\approx Q\operatorname{Sparse}(\widetilde K)(Q^{\mathsf T}H)+P\,B(P^{\mathsf T}H), where H\in\mathbb{R}^{N imes d_h} are point features and B is a learned small r imes r channel-mixing operator.
Implementation notes
Integrate this at the dense pairwise-kernel or coordinate-attention operation in a point-cloud, graph, neural-operator, or implicit-coordinate model. First build a balanced spatial cluster tree over the N input coordinates. At each cluster, construct local detail vectors by orthogonalizing against sampled monomials 1, x_1, ..., x^\alpha for \|\alpha\|_1\le q; concatenate the detail vectors into Q and retain an orthonormal polynomial basis P. In practice, use a numerically stable QR factorization of the local moment matrix, then verify the global residual \|P^\mathsf{T}Q\|_F. Precompute a fixed sparsity mask for the transformed kernel: for every retained pair of samplet indices a,b, evaluate the kernel K_ heta(x_a,x_b), where K_ heta can be an RBF or Matérn kernel with trainable bandwidth, and retain entries whose magnitude exceeds a threshold or whose cluster interaction is near-field. For each forward pass, compute H_d=Q^\mathsf{T}H, apply sparse matrix multiplication H'_d=\widetilde K_{\mathrm{sparse}}H_d, and transform back with QH'_d. Separately compute H_p=P^\mathsf{T}H and use a learned small matrix B or a two-layer MLP on the r polynomial coefficients; add PH'_p to the detail output, followed by the usual channel projection and residual connection. Keep Q and P fixed initially; optionally learn a scalar kernel bandwidth and recompute only values, not the mask. The paper supplies the orthogonality, vanishing-moment, and positive-definiteness structure; estimate the sparsity threshold empirically from validation error and measure \|KH-K_{\mathrm{compressed}}H\| on held-out feature tensors. The first experiment should use N=1024, 4096, and 16384 randomly sampled 2D points, an RBF interaction layer, and a small pointwise regression or Poisson neural-operator task. Compare dense kernel multiplication, Nyström attention, and the compressed layer at matched output error. Success means memory growing approximately O(N log N) rather than O(N^2), at least 2x lower interaction memory for N\ge4096, and no significant increase in validation error or training instability.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built a QR-based orthogonal polynomial/detail kernel-compression MVP and verified the core algebra numerically: polynomial/detail orthogonality was about 1e-14, and the thin-plate reduced kernel had no materially negative eigenvalues. The sparse transformed interaction produced substantial relative errors of 0.31–0.59 and was not faster because Q was explicitly dense, so the claimed neural-layer win was not observed.
- Agent confidence
- 9/10
- Baseline
- Dense RBF K@H; kernel storage grew from 0.5 MB at N=256 to 32 MB at N=2048.
- Idea
- Top-k transformed-detail approximation; relative errors were 0.465, 0.498, 0.594, and 0.479 for N=256, 512, 1024, and 2048; retained detail value/index storage was 0.069, 0.219, 0.613, and 1.928 MB.
Limitations: This was not a hierarchical/localized samplet construction: Q was formed by global dense QR, and sparsity used row-wise top-k rather than a level- and distance-dependent mask. No training task, CUDA benchmark, N=4096/16384 run, Nyström baseline, or implicit fast Q transform was tested.
How to run: python3 samplet_experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- report.md 1.3 KB View
- samplet_experiment.py 4.9 KB View Raw
- samplet_results.json 2.7 KB View Raw