Architecture ideas

Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.

△ Mechanism confirmed, baseline not beaten 2026

Positive-Regime Observable ReLU State Space

Constrain recurrent preactivations to remain nonnegative so that ReLU acts as the identity along realized trajectories. The hidden dynamics then admit a classical linear observability matrix, allowing principled hidden-coordinate selection and conditioning control instead of relying on potentially destructive activation masks.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: On the Number of Observation Nodes in Recurrent Neural Networks with Linear Threshold and ReLU Functions arXiv:2608.29650
✓✓ Beats tuned baseline 2026

Fourier-Mode Stability Shaping

Use a cyclic forward-neighbor recurrent or state-space layer and regularize its coupling so selected discrete Fourier modes are contracting while task-critical modes remain weakly damped. The paper's exact mode factors make instability falsifiable: a mode becomes unstable when its scalar factor changes sign, producing a measurable transition rather than a vague smoothness prior.

Useful7/10
Difficulty5/10
Novelty8/10
Paper: Existence and Stability of Dancing Equilibria in Asymmetric Kuramoto Networks arXiv:2608.29630
✗ Failed on benchmark 2026

Topological Fluctuation Graph Layer

Replace a deterministic graph propagation layer by a stable stochastic linearized latent dynamics whose frequency-resolved covariance matrix defines spectral bands. Train or initialize the graph operator so that a selected covariance band has a nonzero Chern number and remains separated by a measurable spectral gap, producing representations that are robust to local perturbations and can support boundary-localized responses.

Useful7/10
Difficulty7/10
Novelty8/10
Paper: Topology of Fluctuation Bands in Chiral Active Matter arXiv:2608.26055
△ Mechanism confirmed, baseline not beaten 2026

Critical Cross-Layer Weight Sharing

Construct deep or recurrent networks whose layer weights are correlated across depth with a prescribed power-law covariance, rather than either fully tying or fully independently sampling layers. The paper predicts two usable design boundaries: \(\gamma=1/2\) for divergence of correlation-induced fourth moments and \(\gamma=1\) for loss of summable-correlation flatness.

Useful7/10
Difficulty6/10
Novelty8/10
Paper: Bulk Phase Transition and Edge Behavior in Temporally Correlated Random Matrices arXiv:2608.23944
✗ Failed on benchmark 2026

Correlation-Exponent-Safe Weight Initialization

Initialize each row of a neural weight matrix as a stationary correlated Gaussian process instead of using independent entries, but constrain its correlation tail to remain on the finite-fourth-moment side of the transition. This creates controllable structured spectra while avoiding the heavy-edge regime predicted for correlations slower than \(t^{-1/2}\).

Useful7/10
Difficulty4/10
Novelty7/10
Paper: Bulk Phase Transition and Edge Behavior in Temporally Correlated Random Matrices arXiv:2608.23944
△ Mechanism confirmed, baseline not beaten 2026

Spectral Cross-Block Averaging Layer

Construct a cheap graph or token-mixing operator by partitioning nodes into k blocks using the bottom nonconstant eigenvectors of P squared, then replacing dense pairwise mixing with conditional averaging inside each block followed by one baseline propagation step. Unlike ordinary spectral clustering, the bottom modes target partitions where block labels are rapidly destroyed by P, producing an aggressively mixing representation layer rather than a community-preserving pooling layer. The…

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Spectral partitioning for $k$-block averaging kernels of finite Markov chains arXiv:2608.21466
△ Mechanism confirmed, baseline not beaten 2026

Quadrature-Whitened Neural Feature Subspace

Freeze a wide neural spatial dictionary, then compress and whiten it using the quadrature mass matrix before solving for output coefficients or latent PDE states. The retained basis removes feature directions that are numerically invisible or nearly dependent under the actual domain discretization, while preserving the represented function space up to the chosen SVD rank.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Evo-GTransNet for Parabolic PDEs: A Fixed-Feature Galerkin Method of Lines with Quadrature-Mass Orthonormalization arXiv:2608.19615
△ Mechanism confirmed, baseline not beaten 2026

Harmonic-Mode Branch for Topological Memory

Do not force Hodge dissipation onto harmonic edge modes, because these modes are precisely the obstruction to global coercivity. Split the latent state into dissipative coexact modes and a finite-dimensional harmonic branch, and use harmonic-decoupled interactions so each harmonic coordinate defines an invariant affine fibre with its own attractor.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Hodge Coercivity and Global Dynamics in Two-Field Edge-Cochain Systems with MHD-Type Cancellation arXiv:2608.19360
△ Mechanism confirmed, baseline not beaten 2026

Koopman Hankel Dual Autoencoder

Replace pointwise sequence reconstruction with reconstruction of overlapping past and future Hankel windows in a shared latent manifold. A first encoder compresses the delay-coordinate trajectory, while a second decoder or predictor reconstructs the future block from the latent state; training therefore penalizes representations that fit observations but do not preserve dynamical evolution.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Fault detection on manifolds of nonlinear dynamical systems with dual autoencoders arXiv:2608.17698
△ Mechanism confirmed, baseline not beaten 2026

Weighted Resolvent-Equivariant Attention

Add a weighted reflection symmetry to an attention or graph-propagation matrix instead of requiring ordinary permutation equivariance. For paired positions or graph nodes related by an involution, penalize the failure of the propagation operator to commute with the weighted reflection; this makes all geometric multi-step propagations symmetry-compatible. The method is suitable for data with mirror, reversal, paired-agent, or left/right structure where the two sides have unequal importance…

Useful7/10
Difficulty4/10
Novelty6/10
Paper: Resolvent intertwining and spectral duality in Markov chains with geometric resetting arXiv:2608.15140
✗ Mechanism failed 2026

Gaussian-mixture kinetic neural solver

Make a neural network predict a positive Gaussian-mixture representation of the distribution function rather than independent values on a momentum grid. Use the mixture parameters inside a differentiable Boltzmann collision operator, so training directly enforces the interaction mechanism and exposes the relaxation spectrum responsible for ballistic-to-hydrodynamic crossover.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Linear response across interaction regimes in two-dimensional ferromagnets arXiv:2608.14477
✗ Failed on benchmark 2026

Gauge-Free Spectral OT Layer

Parameterize an entropic OT cost only in directions that can change the transport plan, removing row-plus-column potential directions that are invisible because of OT gauge invariance. Whiten the remaining feature coordinates using their empirical covariance, producing an OT layer whose identifiable parameters have substantially more uniform sensitivity.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Sinkhorn Linearization and the Spectral Proxy: Unifying the Statistical and Algorithmic Theory of Feature-Parameterized Inverse Optimal Transport via a Single Spectral Sandwich arXiv:2608.13201
✗ Mechanism failed 2026

Pisot-Orbit Deterministic JL Layer

Replace a dense random projection used before retrieval, classification, or expert routing with a publicly reproducible matrix generated by a Pisot beta-transformation orbit. Search over a small public seed and sampling gap to select one matrix that preserves the calibration set's pairwise distances, then freeze it for training and inference. The projection removes random-matrix storage and makes the same embedding transform exactly reproducible across servers or proof systems.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Deterministic Johnson--Lindenstrauss Projections from Pisot $β$-Transformations for Zero-Knowledge Private Routing arXiv:2608.13078
△ Mechanism confirmed, baseline not beaten 2026

Spectral-Gap Synchronizing Neural Graph Dynamics

Build a graph neural dynamical system whose node states are coupled through a graph Laplacian, using the Laplacian spectral gap as a controllable synchronization mechanism. Increasing coupling strength or algebraic connectivity should selectively suppress disagreement modes, producing a measurable faster decay of node-to-node errors without requiring stronger contraction of the common mode.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Contraction Analysis of Holomorphic Dynamical Systems via the Intrinsic Kobayashi Metric arXiv:2608.07551
△ Mechanism confirmed, baseline not beaten 2026

Spectral Message Basis

Replace full agent-to-agent state transmission with coefficients in a learned dominant Koopman mode basis. Agents communicate only the leading spectral coordinates that explain slowly decaying collective behavior, while retaining a certificate based on the spectral gap and subdominant eigenvalue to decide whether the compressed representation is safe.

Useful7/10
Difficulty6/10
Novelty6/10
Paper: Certifying Collective Reasoning in Multi-Agent Systems via Koopman Spectral Analysis arXiv:2608.05956
✓✓ Beats tuned baseline 2026

Divergence-Free Spherical Kernel Layer

Build a kernel aggregation layer whose output is a tangent vector field on the unit sphere and whose surface divergence is identically zero by construction. For each source point, use a matrix kernel obtained by applying a surface-rotated gradient in the query variable to a scalar zonal kernel; this is a differential-form version of the paper's matrix-valued construction. The layer can replace attention or message passing when the target dynamics are incompressible, such as spherical fluid…

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Divergence-free interpolation of tangential vector fields via matrix-valued kernels arXiv:2608.05547
△ Mechanism confirmed, baseline not beaten 2026

Explicit Multi-Timescale Memory Bank

Use a bank of damped rotational state channels with a deliberately spread decay spectrum, allowing one recurrent layer to represent short, medium, and long temporal dependencies without relying on a single learned spectral radius. Concatenate the channels and train a readout or downstream nonlinear head to select the appropriate memory timescale.

Useful7/10
Difficulty4/10
Novelty7/10
Paper: Lindblad-Inspired Multi-Timescale Reservoir Computing with Separable Rotation and Dissipation arXiv:2608.04028
△ Mechanism confirmed, baseline not beaten 2026

Normal Damped-Rotation Recurrent Layer

Replace an unconstrained recurrent matrix with an orthogonally mixed block diagonal matrix whose blocks are independently parameterized damped rotations. The model receives explicit phase mixing from the rotation frequencies and controlled forgetting from the decay rates, while its linear recurrent dynamics have a known contraction factor before the nonlinear activation.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Lindblad-Inspired Multi-Timescale Reservoir Computing with Separable Rotation and Dissipation arXiv:2608.04028
✗ Failed on benchmark 2026

Chemotactic Feature Transport Layer

Replace part of a CNN or continuous-depth feature block with two coupled feature fields. One field is transported up gradients of the other through a conservative cross-gradient flux, creating adaptive spatial organization that ordinary diffusion or symmetric convolution cannot produce. The coupling strength and dominant wavelength are controlled by a directly testable linear-instability boundary.

Useful7/10
Difficulty6/10
Novelty8/10
Paper: Pattern formation: reactivity is not necessary for chemotaxis--driven instabilities arXiv:2608.03685
△ Mechanism confirmed, baseline not beaten 2026

Sparse rational fractional graph layer

Replace a stack of local message-passing layers by a fractional spectral graph filter implemented through a small bank of sparse shifted Laplacian solves. The fractional exponent controls how strongly the layer mixes information across graph distances, while rational approximation avoids dense eigendecomposition and supports efficient differentiation through iterative linear solvers.

Useful7/10
Difficulty6/10
Novelty5/10
Paper: Numerical approximation of fractional diffusion equations on metric graphs arXiv:2608.01932
✓✓ Beats tuned baseline 2026

Holonomy-Twisted Message Passing

Replace an ordinary graph-neural-network edge message by a message transported through a unitary representation of the edge's fundamental-group label. The layer can distinguish globally different holonomy sectors even when the underlying bundles or ordinary graph topology are identical, while inverse edge labels enforce a Hermitian and unitary consistency constraint.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Beyond $K$-Theory: Geometry and Holomorphy in Hyperbolic Band Theory arXiv:2608.01596
✗ Mechanism failed 2026

RG Spectral Feature Gate

Replace fixed PCA-rank selection in a hidden layer with a renormalization-group-inspired gate over covariance eigenvalue bands. The gate retains modes whose effective quartic interaction remains unstable or strongly scale-dependent, while pruning bands that flow toward the Gaussian noise fixed point. Unlike top-eigenvalue truncation, this is designed for extensive-rank signal distributed throughout the bulk spectrum.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Data Field Theory: Theory and Applications of the Functional Renormalization Group for Signal Detection arXiv:2607.27236
△ Mechanism confirmed, baseline not beaten 2026

Spectrally identifiable phaseless recurrent layer

Build a complex-valued recurrent or graph-neural layer whose hidden state evolves under a fixed graph Schrödinger operator and is exposed to the downstream network only through coordinate magnitudes at several times. Choose the diagonal potential so that the spectrum has unique unordered pair sums, the squared-eigenvector matrix is invertible, and every eigenvector pair overlaps in at least one observed coordinate; the resulting temporal intensity code is theoretically injective up to one…

Useful7/10
Difficulty6/10
Novelty8/10
Paper: Dynamical phase retrieval for Schr{ö}dinger evolution on finite graphs arXiv:2607.26705
△ Mechanism confirmed, baseline not beaten 2026

Non-Gaussian Perron–Frobenius Latent Filter

Replace Gaussian covariance propagation in a neural state-space model with a finite Perron–Frobenius operator acting on coefficients of a learned density basis. A neural encoder maps observations to latent states, while an eDMD-derived matrix transports the full coefficient vector and supports multimodal or skewed uncertainty. This creates a cheap deterministic uncertainty layer that can be rolled forward for long horizons without repeatedly sampling particles.

Useful7/10
Difficulty6/10
Novelty6/10
Paper: An extended Perron-Frobenius operator filter for nonlinear state estimation arXiv:2607.26632