Convolution-Nullspace Feature Layer
Implementation & benchmark of arXiv:2608.24143 — Invariant pointwise closed subspaces of Lipschitz spaces and their preduals
Source paper: Invariant pointwise closed subspaces of Lipschitz spaces and their preduals arXiv:2608.24143 ⓘ · analyzed Aug 29, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Construct a graph or sequence feature layer whose output satisfies a learned or fixed finite-support convolution equation, rather than allowing arbitrary features. For grid data, this is a hard local harmonicity or stencil constraint; for irregular graphs, it is imposed with a sparse incidence operator. The constraint can be exact through nullspace projection or soft through a residual penalty, and should improve robustness when the target signal is locally smooth or obeys known conservation relations.
Formulas
Mathematical statement
The paper describes translation-invariant subspaces on a group G through convolution equations of the form $(f*x_n)(g)=\sum_{h\in G}f(gh^{-1})x_n(h)=0$, where $f:G\to\mathbb{R}$ is a Lipschitz function, $x_n:G\to\mathbb{R}$ is a finite or summable constraint kernel, $g\in G$ is the translated location, and $*$ is group convolution. The associated constrained space is the set of Lipschitz functions annihilated by every translated kernel. The construction in the paper defines $T(x)=\sum_{n=1}^{\infty}x(n)f_n$ and controls its edge gradient using $|\nabla T(x)(e)|\leq\|x\|_{\ell_\infty}\sum_{n=0}^{\infty}2^{-2n}$, where $e$ is a graph edge and $\nabla f(e)$ is the endpoint difference across that edge. We adapt the first result to a finite stencil matrix $C$ and the second principle to a feature-Lipschitz regularizer: constrain $CH=0$ and penalize or normalize the edge-gradient norm. For a batch of node features $H\in\mathbb{R}^{N\times D}$, $C\in\mathbb{R}^{N\times N}$ is a sparse matrix assembled from translated local kernels; on a $d$-dimensional grid, $C$ is a convolution operator. The exact nullspace projection is $P=I-C^{\dagger}C$, where $C^{\dagger}$ is the Moore-Penrose pseudoinverse, so $H_{\mathrm{con}}=PH$ satisfies $CH_{\mathrm{con}}=0$ up to numerical error.
Implementation notes
(1) Integration point: insert the module after a graph-convolution or Transformer token-mixing block and before the next nonlinear MLP. Let $H\in\mathbb{R}^{N\times D}$ be node or token features. Build a sparse local constraint matrix $C$ from a fixed stencil, such as the graph Laplacian $C=L$, a discrete biharmonic stencil, or a user-specified signed neighborhood kernel. For a grid, implement $CH$ as a depthwise convolution; for a graph, use sparse matrix multiplication. (2) Pseudocode: H = mixer(X); R = C @ H; either use Hc = H - lambda * C.T @ solve(C @ C.T + eps*I, R) as an approximate projection, or precompute a low-rank/nullspace basis $U$ and use Hc = U @ (U.T @ H); then Y = mlp(Hc). Add loss += alpha * mean((C @ Hc)**2) if using a soft constraint. Estimate the edge-gradient statistic with G_e = Hc[src]-Hc[dst] and optionally add beta * mean(norm(G_e,2)). (3) Mathematics computed exactly: the local convolution residual $CH$, the projection formula, and the gradient-bound surrogate. Quantities such as the pseudoinverse, rank, and suitable $\alpha,\beta$ are estimated numerically; use conjugate gradients rather than forming $C^{\dagger}$ for large graphs. (4) First experiment: train a 4-layer GraphSAGE or GCN on Cora, Citeseer, and a noisy-grid image-classification task, comparing an unconstrained model, soft residual regularization, and projected features at equal parameter count. Measure validation accuracy under feature noise, the mean constraint residual $\|CH\|_F/\|H\|_F$, and gradient explosion during training. The idea succeeds if projection improves corruption accuracy or calibration without reducing clean accuracy by more than 1%, while keeping the residual near numerical zero and adding less than 20% wall-clock cost.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.