Architecture ideas

Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.

✗ Mechanism failed 2026

Fractional-memory recurrent state

Construct an efficient recurrent or state-space layer whose impulse response follows Mittag-Leffler relaxation instead of a single exponential. A bank of stable diagonal state channels approximates the long power-law tail, allowing the layer to retain information over widely separated timescales with only \(K\) states per feature.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Anomalous diffusion memory factorization: Characteristic timescales and application to inverse problem arXiv:2608.21674
✗ Mechanism failed 2026

Sobolev-Calibrated Frozen Sigmoid Features

Replace a trainable shallow MLP hidden layer by a frozen bank of smooth sigmoid ridge functions and train only a linear output head. Choose the feature count and parameter sampling regime using the theorem's explicit dependence on input dimension d, target regularity k, evaluation norm m, and confidence delta. The construction is especially appropriate for smooth regression, scientific surrogate models, and PINNs, where derivatives of the network output are part of the loss.

Useful7/10
Difficulty3/10
Novelty5/10
Paper: Optimal Sobolev Approximation by Deterministic and Random Shallow Sigmoidal Networks arXiv:2608.19797
✗ Failed on benchmark 2026

Quadratic center-manifold bottleneck

Replace a conventional linear decoder in an autoencoder or latent state-space model with an explicit quadratic manifold decoder, allowing a small latent vector to represent curved and transport-like state trajectories. Add a dynamics-aware invariance loss that penalizes the discrepancy between the time derivative of the quadratic manifold and the neural dynamics evaluated on that manifold.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Beyond linear subspaces: Nonlinear moment matching meets quadratic manifolds arXiv:2608.19486
✗ Mechanism failed 2026

Variable-Projection Adaptive Fourier Layer

Replace fixed random Fourier features in a coordinate network or PINN with a small complex Fourier dictionary whose propagation directions are learned from a weighted residual. Given directions, solve the linear feature coefficients exactly or by ridge regression, and optimize only the directions in the outer loop. This should represent low-directional-complexity fields with fewer features and avoid wasting gradient updates on coefficients that can be fitted analytically.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Direction-Adaptive Plane-Wave Discontinuous Galerkin Methods for the Helmholtz Equation arXiv:2608.18380
✗ Mechanism failed 2026

Automaton-Supervised Neural Policy Shield

Attach a deterministic supervisory automaton to a neural policy or sequence model and mask every event disabled by the current supervisor state. Use a short receding-horizon planner over admissible events to resolve conflicts between neural preferences and shared-resource constraints. The network scores useful actions, while the automaton supplies an exact safety layer.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Model Predictive Supervisory Control for Hierarchical and Distributed UAS Traffic Management arXiv:2608.18353
△ Mechanism confirmed, baseline not beaten 2026

Ordered Diffusion Message Passing

Use a learned scalar ordering function to turn a symmetric local Gaussian graph kernel into a directed, row-stochastic message-passing operator. The asymmetric tilt lets neighboring nodes communicate preferentially along an inferred dynamical direction, while the Gaussian factor retains locality and diffusion-like smoothing.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Ordered Diffusion Kernels arXiv:2608.18019
△ Mechanism confirmed, baseline not beaten 2026

Dimension-Free Brenier Transport Layer

Build a neural transport layer by parameterizing a convex potential whose gradient maps a semi-log-concave latent distribution into a compact convex data domain. Use the paper's dimension-free Lipschitz certificate to set the layer's Jacobian scale, initialize the potential, and reject or regularize parameter updates that create excessive curvature. The goal is a bounded-output transport module that is less sensitive to latent dimension than diameter-based spectral heuristics.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Dimension-Free Lipschitz Bounds for Brenier Maps to Compactly Supported Log-Concave Targets arXiv:2608.15906
✗ Mechanism failed 2026

Gaussian-mixture kinetic neural solver

Make a neural network predict a positive Gaussian-mixture representation of the distribution function rather than independent values on a momentum grid. Use the mixture parameters inside a differentiable Boltzmann collision operator, so training directly enforces the interaction mechanism and exposes the relaxation spectrum responsible for ballistic-to-hydrodynamic crossover.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Linear response across interaction regimes in two-dimensional ferromagnets arXiv:2608.14477
△ Mechanism confirmed, baseline not beaten 2026

Persistent Workspace for Online Adaptation

Turn the latent substrate into a persistent computational workspace for sequential inputs: each new observation is written into a designated subspace, processed by the same local rule, decoded, and then selectively retained or reset. This creates a compact recurrent model whose state can accumulate algorithmic information across a stream without expanding the parameter count.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Emergent Models: Intelligence from Tiny Substrates arXiv:2608.14019
✗ Failed on benchmark 2026

Gauge-Free Spectral OT Layer

Parameterize an entropic OT cost only in directions that can change the transport plan, removing row-plus-column potential directions that are invisible because of OT gauge invariance. Whiten the remaining feature coordinates using their empirical covariance, producing an OT layer whose identifiable parameters have substantially more uniform sensitivity.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Sinkhorn Linearization and the Spectral Proxy: Unifying the Statistical and Algorithmic Theory of Feature-Parameterized Inverse Optimal Transport via a Single Spectral Sandwich arXiv:2608.13201
✗ Failed on benchmark 2026

Critical stochastic min-plus tree layer

Replace deterministic binary-tree pooling or hierarchical feature aggregation by a stochastic merge that chooses either elementwise addition or elementwise minimum. The mixing probability p controls whether zero or sparse states proliferate or disappear, with a predicted absorbing-state transition at p = 1/2.

Useful7/10
Difficulty5/10
Novelty8/10
Paper: Finite-depth scaling and an exact Bernoulli-leaf identity for the min-plus process on the binary tree arXiv:2608.12295
△ Mechanism confirmed, baseline not beaten 2026

Intrinsic-Rank Filter Memory for Actor-Critic

Replace an oversized recurrent hidden state or raw history stack with a causal filtered input-output lift followed by an SVD-selected bottleneck. The actor, critic, and Bellman regression operate only on the identifiable memory coordinates, preventing deterministic null directions from being fitted as if they were independent state variables.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Reinforcement Learning-Based Output Feedback LQR for Continuous-Time MIMO Systems arXiv:2608.11750
△ Mechanism confirmed, baseline not beaten 2026

Kernel-Prompted Random Transformer

Freeze a randomly initialized single-layer transformer and use a constructed soft prompt to make its attention weights equal Gaussian-kernel weights over support examples. The resulting model performs Nadaraya-Watson regression in one forward pass, so task adaptation stores prompt tokens rather than modifying network weights. Prompt length becomes the number of kernel centers, while hidden dimension and prompt norm determine whether the required logits can be represented accurately.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Training-Free Universal Approximation by Prompting Random Transformers arXiv:2608.09558
✗ Failed on benchmark 2026

Differentiable Persistence Landscape Layer

Convert each persistence diagram produced from an input, intermediate feature map, or graph filtration into a discretized persistence landscape and feed it to an MLP or concatenate it with ordinary neural features. Unlike a variable-size list of birth-death pairs, the landscape has a fixed tensor shape and is provably nonexpansive with respect to the diagram Wasserstein distance.

Useful7/10
Difficulty4/10
Novelty5/10
Paper: A Hilbert space embedding of persistence diagrams and barcodes arXiv:2608.08858
✗ Failed on benchmark 2026

Signature-conditioned cylindrical law head

Add a conditional-law head that maps a compact representation of an initial distribution and a shared-noise trajectory to a Gaussian mixture, then computes downstream predictions as analytic expectations under that mixture. This can replace expensive particle rollouts or particle pooling in stochastic world models and conditional diffusion systems while retaining multimodality.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: A cylindrical neural approximation theorem for conditional laws of McKean-Vlasov equations with common noise arXiv:2608.08040
△ Mechanism confirmed, baseline not beaten 2026

Hodge-Selective Edge Dynamics

Replace an unconstrained edge-feature residual update in a graph neural network with separate cut-space and harmonic-space updates. The cut branch carries transfer information visible at nodes, while the harmonic branch models cycle circulation and can be given an independently chosen contraction rate, preventing persistent or unstable circulation features from contaminating node predictions.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: From a Scalar Parabolic Oscillator to Topological Thermostats: Selective Feeback Control of Harmonic Flow Modes arXiv:2608.07768
✗ Failed on benchmark 2026

Information-Gated Attention

Use predicted covariance reduction as a differentiable gate for selecting tokens, views, sensors, or retrieved demonstrations. The gate favors inputs with high expected information gain while accounting for acquisition cost, turning attention and data collection into active observability optimization.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Information-Aware Model Predictive Control for Satellite Inspection arXiv:2608.07765
△ Mechanism confirmed, baseline not beaten 2026

Tropical Multisymmetric Pooling

Replace ordinary sum or mean pooling in a permutation-invariant set network by the complete family of basic tropical multisymmetric values. For an input set of n points in R^r, each feature computes the maximum total coordinate score obtainable by assigning disjoint rows to prescribed coordinate channels. The resulting representation is invariant to row permutations, separates all multisets, and inherits a bi-Lipschitz relation to optimal row matching, so nearby sets cannot be arbitrarily…

Useful7/10
Difficulty5/10
Novelty8/10
Paper: The basic tropical polynomials generate the semifield of $r$-symmetric tropical rational functions arXiv:2608.06857
△ Mechanism confirmed, baseline not beaten 2026

Local separating-code positional encodings

Replace globally unique node IDs with a small discrete alphabet labeling whose ordered local neighborhood words uniquely identify nodes. Feed these separating words to a GNN or graph transformer as positional features, preserving node distinguishability with bounded local receptive fields.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Information and Locality in Cayley Graphs arXiv:2608.04608
△ Mechanism confirmed, baseline not beaten 2026

Explicit Multi-Timescale Memory Bank

Use a bank of damped rotational state channels with a deliberately spread decay spectrum, allowing one recurrent layer to represent short, medium, and long temporal dependencies without relying on a single learned spectral radius. Concatenate the channels and train a readout or downstream nonlinear head to select the appropriate memory timescale.

Useful7/10
Difficulty4/10
Novelty7/10
Paper: Lindblad-Inspired Multi-Timescale Reservoir Computing with Separable Rotation and Dissipation arXiv:2608.04028
△ Mechanism confirmed, baseline not beaten 2026

Normal Damped-Rotation Recurrent Layer

Replace an unconstrained recurrent matrix with an orthogonally mixed block diagonal matrix whose blocks are independently parameterized damped rotations. The model receives explicit phase mixing from the rotation frequencies and controlled forgetting from the decay rates, while its linear recurrent dynamics have a known contraction factor before the nonlinear activation.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Lindblad-Inspired Multi-Timescale Reservoir Computing with Separable Rotation and Dissipation arXiv:2608.04028
△ Mechanism confirmed, baseline not beaten 2026

Van der Corput progressive expert scheduler

Replace random or greedy one-expert-at-a-time activation with a deterministic binary van der Corput sequence. At each training or inference step, the schedule chooses an expert whose cumulative usage remains close to its proportional target, while recursively balancing nested expert groups rather than only balancing individual experts.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: House-monotone multi-level apportionment has logarithmic quota discrepancy arXiv:2608.02559
△ Mechanism confirmed, baseline not beaten 2026

Exact-boundary geometric PINN ansatz

Replace penalty enforcement of boundary and asymptotic conditions by a neural-network parametrization that satisfies them identically. For a disc mapped to hyperbolic half-space, construct the positive height coordinate with a vanishing boundary factor and construct tangential coordinates as a fixed boundary extension plus a vanishing correction.

Useful7/10
Difficulty4/10
Novelty6/10
Paper: A user's guide to PINNs in geometric analysis: lessons from the asymptotic Plateau problem arXiv:2607.28733
✗ Failed on benchmark 2026

Exact Event-Chained Neural ODE

Represent a hybrid trajectory with one neural module per known dynamical phase rather than a single network spanning all phases. Feed the predicted terminal state of phase r directly as the initial state of phase r+1, so continuity is satisfied by construction instead of by a soft interface penalty. This should improve learning near abrupt changes and remove an otherwise poorly conditioned loss-weight tradeoff.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Event-Structured Physics-Informed Neural Networks for Differentiable Critical Clearing Boundaries arXiv:2607.27681