GLM Energy-Preserving Feature Block
Implementation & benchmark of arXiv:2607.22151 — Analysis and structure-preserving discretization of the Heat-GLM system
Source paper: Analysis and structure-preserving discretization of the Heat-GLM system arXiv:2607.22151 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace an unconstrained residual block by a four-field feature dynamics containing a primary feature T, flux-like auxiliary features J, curl-cleaning features psi, and a scalar cleaning feature phi. Couple these fields with learned skew-adjoint spatial operators so that the reversible block preserves the squared feature norm, while a separately controlled relaxation term can remove high-frequency or constraint-violating components. Use an exact Cayley update rather than explicit Euler to prevent norm growth across many layers.
Formulas
Mathematical statement
The reversible heat-GLM system is the four-field PDE $$\partial_t T+c_0\nabla\cdot\mathbf{J}=0,\quad \partial_t\mathbf{J}+c_0\nabla T+c_h\nabla\times\boldsymbol{\psi}=0,\quad \partial_t\boldsymbol{\psi}-c_h\nabla\times\mathbf{J}+c_h\nabla\varphi=0,\quad \partial_t\varphi+c_h\nabla\cdot\boldsymbol{\psi}=0.$$ Here T and phi are scalar fields, J and psi are d-dimensional vector fields, c_0 and c_h are propagation speeds, and divergence and curl are spatial differential operators. Its quadratic energy density is $$\mathcal{E}(T,\mathbf{J},\boldsymbol{\psi},\varphi)=\tfrac12(T^2+\|\mathbf{J}\|^2+\|\boldsymbol{\psi}\|^2+\varphi^2).$$ The paper writes the system as $$\partial_t u+\sum_i B_i\partial_i u=0,$$ with each field-coupling matrix B_i symmetric. For a neural discretization, choose learned operators A_i satisfying A_i^T=-A_i and define $$K=-\sum_i B_i\otimes A_i.$$ Because B_i^T=B_i and A_i^T=-A_i, K^T=-K, so the continuous linear flow preserves ||u||_2^2. The Cayley discretization $$u_{k+1}=\left(I-\frac{h}{2}K\right)^{-1}\left(I+\frac{h}{2}K\right)u_k$$ is norm-preserving for every step size h when the solve is exact. Optional relaxation uses a positive semidefinite R=R^T\succeq0 and applies $$u_{k+1}=\operatorname{Cayley}(hK)(I-h\lambda R)u_k,$$ with lambda >= 0.
Implementation notes
Integrate this as a replacement for a residual CNN block or as one step of a continuous-depth state-space model. First maintain four feature groups at each spatial location: T with C channels, J with dC channels, psi with dC channels, and phi with C channels; concatenate them into u. Parameterize each spatial operator with a convolution W_i and enforce skew-adjointness by A_i=(W_i-W_i^T)/2, where W_i^T is the adjoint convolution under the selected padding and boundary convention. Construct small fixed or learned symmetric field-coupling matrices B_i implementing the c_0 gradient/divergence and c_h curl couplings, then compute K u=-sum_i(B_i tensor A_i)u. Use the Cayley update by solving (I-hK/2)v=(I+hK/2)u with several GMRES iterations, or use an exact dense/block solve when the channel dimension is small. For damping, apply v <- v-h lambda Rv with diagonal R that selectively damps psi and phi. The paper determines the symmetric coupling and energy invariant; learned operators, h, lambda, and R are empirical choices. Measure norm drift ||v||^2/||u||^2 and residual-solve error. The first experiment should compare a 32-layer CIFAR-10 CNN against standard residual, antisymmetric, and orthogonal/Cayley blocks at matched parameters and FLOPs, testing h in {0.25,0.5,1,2}. Record loss descent, gradient norms, accuracy, norm drift, and NaN rate. Success means stable training at h=1 or 2, energy drift below 1e-4 per reversible block, and equal-or-better accuracy than the baselines.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.