Architecture ideas

Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.

✓✓ Beats tuned baseline 2026

Noisy Scrambling-Front Network

Construct a residual sequence or depth network whose nonnegative influence density follows a discretized noisy Fisher-KPP equation: local influence diffuses, grows when small, saturates at a finite carrying capacity, and receives state-dependent noise. Use this density to gate ordinary feature updates rather than relying only on unconstrained residual additions. The mechanism predicts a measurable propagation speed and an instability boundary, allowing the architecture to be falsified…

Useful7/10
Difficulty6/10
Novelty8/10
Paper: Effective Field Theory of Operator Scrambling from Strong-to-Weak Symmetry Breaking arXiv:2607.24925
△ Mechanism confirmed, baseline not beaten 2026

Zero-Augmented Double-Scoring

For each frozen weight tensor, append a second tensor of identically shaped zero weights and assign trainable scores to both the real and dummy edges. Select a fixed number of candidates by top-k score in the doubled space; real edges selected by the competition remain active, while selected dummy edges consume the quota without changing the network. The resulting number of active original edges is learned rather than imposed by a separate layerwise sparsity search.

Useful7/10
Difficulty4/10
Novelty7/10
Paper: Double-Scoring: Reliable Extraction of Strong Lottery Tickets arXiv:2607.20555
✓✓ Beats tuned baseline 2026

Spectral Memory-Lift Ensemble

Replace or augment a recurrent or state-space block with a small ensemble of stable linear memory lifts. Each lift stores a low-dimensional state whose repeated matrix powers generate a structured long-range convolution, and a learned gate mixes the experts using both their current predictions and their slowest block-operator mode. The module can represent multiple memory timescales without explicitly storing a long token history.

Useful7/10
Difficulty6/10
Novelty5/10
Paper: Memory operator ensembles indicate proximity to criticality in simulated AMOC transitions arXiv:2607.24310
△ Mechanism confirmed, baseline not beaten 2026

Warm-Started Exact Rank Pruning

Parameterize a trainable weight update as \(\Delta W=UV^{\top}\) with an excessive initial rank \(r\), and penalize active columns using an exact column \(\ell_{2,0}\) penalty. Increase \(\lambda\) along a warm-started path and hard-delete redundant paired columns, producing an automatically selected rank without training a separate model for every candidate rank. Apply scale balancing after each update so pruning decisions are invariant to reciprocal rescaling of factor pairs.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Automatic Model-Order Selection for Nonnegative Matrix Factorization via Column $\ell_{2,0}$ Regularization arXiv:2607.24193
✗ Mechanism failed 2026

Autocatalytic Hysteresis Memory Cell

Replace or augment a recurrent hidden coordinate with a nonnegative bistable autocatalytic state driven by an external control signal. The cell retains information through metastable low and high states, while a periodic or slowly varying control produces a controlled phase lag and hysteresis useful for temporal regime detection. Explicit noise can be injected to test whether it enhances switching near the predicted intermediate-frequency regime.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Dynamic hysteresis in an autocatalytic reaction network arXiv:2607.24163
△ Mechanism confirmed, baseline not beaten 2026

Feasible Action Mapping Safety Layer

Let a neural policy emit an unconstrained abstract action z, then solve a state-dependent feasibility problem that maps z to an admissible optimal-control parameter p before execution. Unlike coordinate-wise clipping, the mapping accounts for predicted dynamics, coupled state and input constraints, and recursive feasibility, allowing the policy to retain a simple unconstrained output space while the controller enforces plant constraints.

Useful7/10
Difficulty6/10
Novelty5/10
Paper: Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping arXiv:2607.23930
△ Mechanism confirmed, baseline not beaten 2026

Shell-Wise Balanced MoE Routing

Replace a soft auxiliary load-balancing loss with a deterministic router correction that balances expert assignments separately within confidence or score shells. The correction preserves the number of tokens assigned to each shell while making every expert receive equal or nearly equal mass inside each shell, preventing high-confidence tokens from monopolizing a subset of experts.

Useful7/10
Difficulty5/10
Novelty5/10
Paper: Optimal Dynamic Cooling of Multiple Qubits arXiv:2607.23641
△ Mechanism confirmed, baseline not beaten 2026

Exact Neural de Rham Backbone

Replace independently parameterized scalar, vector, and higher-order neural outputs with consecutive spaces of ReLU-power differential forms linked by an exact exterior-derivative layer. The network can then produce curl-free, divergence-free, or more general closed fields by construction, while the complex prevents artificial null-space modes that commonly appear when differential constraints are enforced only through sampled residual losses.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: ReLU$^k$ Neural de Rham Complexes arXiv:2607.22478
△ Mechanism confirmed, baseline not beaten 2026

Maximum-Entropy Relational Block Kernel

Parameterize a multi-relational graph kernel as a finite stochastic block model and fit it by maximum entropy subject to differentiable motif-density constraints. Use the resulting block kernel as a graph-neural-network message-passing operator or structured prior for edge prediction, reducing an O(n^2 r) relation tensor to O(m^2 r+n) parameters for m latent blocks and r relations.

Useful7/10
Difficulty6/10
Novelty6/10
Paper: Constrained Multi-Relational Graphons with Maximum Entropy arXiv:2607.22383
✓✓ Beats tuned baseline 2026

Gauge-Patched Local Experts

Represent hidden features with several local orthogonal or unitary frames, and let group-valued transition maps align neighboring experts. Instead of forcing every expert to learn a globally coherent coordinate system, train local experts independently and impose a patching constraint that produces a globally consistent feature field. This is especially suitable for graph neural networks on data with heterogeneous local geometries or mixture-of-experts models whose experts specialize by region.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: A PDE approach to the 2D Yang-Mills measure arXiv:2607.22236
△ Mechanism confirmed, baseline not beaten 2026

Petri-Net Safety Shield for Neural Policies

Compile Boolean safety requirements over a discrete plant state into a Petri Net and place a transition mask between a neural policy and the environment. The policy may score all candidate actions, but only transitions enabled at the current marking are executable; this prevents invariant-violating actions and exposes deadlocks or unreachable policy proposals explicitly.

Useful7/10
Difficulty5/10
Novelty5/10
Paper: Constraint-Driven Synthesis of Hyper Petri Nets arXiv:2607.22062
✓✓ Beats tuned baseline 2026

Incidence-Matrix Structured Action Head

Build a policy or prediction head that scores atomic graph entities once and converts those scores into logits for legal composite actions through an instance-specific incidence matrix. The neural parameters are independent of the number of actions and their composition, so the same model can process graphs with different action counts and compatibility structures without padding every action space to a global maximum.

Useful7/10
Difficulty4/10
Novelty6/10
Paper: A Graph-Based Control Interface for Traffic Signals on Heterogeneous Road Networks arXiv:2607.21831
△ Mechanism confirmed, baseline not beaten 2026

Subcritical Percolation Jordan Readout

Use several independently percolated versions of an input graph and rank nodes by Jordan centrality inside the largest retained components. Feed only the consistently high-ranked nodes to a graph readout or source-localization head, making predictions depend on structurally stable regions rather than shortcut edges or fragile local neighborhoods.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Subcritical percolation and network archaeology on random recursive tree substrate networks arXiv:2607.21428
✗ Failed on benchmark 2026

Probe-Then-Partitioned Multi-Task Trunk

Train a cheap shared multi-task probe briefly, extract one semantic embedding per task, and use density-based clustering to determine which tasks should share a neural trunk. After clustering, replace the globally shared trunk by one trunk per discovered cluster, with task heads remaining separate; this preserves cooperation among related tasks while isolating destructive task interactions.

Useful7/10
Difficulty5/10
Novelty5/10
Paper: Semantic-Aware Task Clustering for Constructive and Cooperative Multi-Tasking arXiv:2607.21426
✗ Mechanism failed 2026

Dyadic Resolvent Filter Layer

Replace a single polynomial graph filter or dense inverse with a multiscale layer that applies unitary propagations at dyadic times and combines them according to the dyadic resolvent identity. For a symmetric graph operator, this implements a stable rational spectral filter that can selectively retain or suppress frequency bands while exposing logarithmic multiscale structure.

Useful7/10
Difficulty6/10
Novelty6/10
Paper: Dyadic Resolvent Representations of Self-Adjoint Operators: Propagator Expansions, Spectral Measures, and Zeta Functions arXiv:2607.21278
△ Mechanism confirmed, baseline not beaten 2026

Tangent-Branch Neural Evasion Layer

Wrap a neural multi-agent policy with an analytic planner that generates turn-straight trajectories tangent to pursuer surveillance disks, then selects the branch with the smallest predicted completion time. The network supplies high-level preferences or residual corrections, while the geometric layer prevents unnecessarily entering exclusion regions and exposes an explicit branch-switching signal for training.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Semi-Explicit Solutions to the Prying-Pedestrian Surveillance-Evasion Differential Game and Extensions to Two Pursuers arXiv:2607.21087
△ Mechanism confirmed, baseline not beaten 2026

Markov Spectral Equivariant Layer

Replace an orthogonal truncated Fourier or Wigner projection in a compact-Lie-group equivariant layer by a finite-rank Fejér-Markov filter. The filter acts as a normalized positive group convolution, preventing sup-norm amplification and suppressing high-frequency artifacts while retaining exact equivariance.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Structure-Preserving Spectral Dynamic Programming on Compact Lie Groups arXiv:2607.20854
△ Mechanism confirmed, baseline not beaten 2026

Manifold-kernel attention

Replace or augment dot-product attention with a non-increasing radial kernel of pairwise representation distance. The bandwidth is normalized using an estimated local intrinsic dimension and local neighbor scale, creating an explicit locality-controlled attention operator.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Graphon as a Bridge between Graphs and Manifolds arXiv:2607.20213
✗ Failed on benchmark 2026

Confidence-Tested LoRA Pruning

Replace deterministic LoRA importance scores with one-sided tests of whether each rank-one update has population contribution at least a user-selected threshold. Maintain empirical contribution samples during fine-tuning, estimate their uncertainty, and prune the components with the weakest statistical evidence while respecting the target rank budget. The method should avoid deleting components merely because their latest minibatch gradient was small.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Statistical Inference for Rank Allocation in Low-Rank Adaptation arXiv:2607.20205
△ Mechanism confirmed, baseline not beaten 2026

Invariant-domain learned reconstruction

Insert a neural local reconstruction into a finite-volume or graph-based simulator, but hard-cap its contribution so every reconstructed state remains in the physical admissible set. The network learns accuracy-sensitive gradients or stencil weights; a deterministic limiter, rather than a penalty loss, guarantees positive density and pressure for arbitrary network outputs.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Guarantees by Construction for Learned Finite Volume Schemes on Steady Supersonic Flow arXiv:2607.20171
△ Mechanism confirmed, baseline not beaten 2026

Conditional OT barycenter feature augmentation

Construct synthetic latent examples from an optimal-transport barycenter of several source domains, restricting the barycentric mass to the context region relevant to the prediction. The resulting representations preserve cross-source consensus while reducing domain-specific nuisance variation. Train on the original examples plus barycentric latent examples with transported soft labels.

Useful7/10
Difficulty5/10
Novelty4/10
Paper: Harnessing Heterogeneous Data for Conditional Optimization via Optimal Transport arXiv:2607.19761
✗ Failed on benchmark 2026

Doubled-angle orientation order pooling

Add a differentiable orientation-pooling layer after steerable filters or an orientation-bin expansion. It aggregates unoriented line evidence using doubled-angle vectors, so a feature at angle θ is identical to one at θ+π, while symmetric orientations cancel naturally instead of producing an arbitrary mean angle. Feed the network both the Cartesian order parameter and its magnitude-based confidence.

Useful7/10
Difficulty4/10
Novelty6/10
Paper: Perceived vertical and eye level as one orientation order parameter: a closed-form account of the Li-Matin rules for egocentric space arXiv:2607.19681
✗ Failed on benchmark 2026

Volume-Mass Diffusion GNN

Replace ordinary graph propagation by diffusion with a positive node-dependent mass matrix \(\mathbf V\), so high-volume nodes update slowly and low-volume nodes update rapidly. Use node volumes as fixed metadata, a function of degree, or learned positive gates; this makes the architecture sensitive to dynamical localization that degree-normalized GCNs cannot represent.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Localization transitions of diffusion dynamics in physical networks arXiv:2607.19486
△ Mechanism confirmed, baseline not beaten 2026

Cholesky-Structured SPD Classifier

Build an SPD classifier and residual head directly from Cholesky factors, using lower-triangular differences and matrix-power terms instead of generic eigendecomposition-based logarithm operators. This retains covariance geometry while making positive-definiteness automatic and backpropagation more numerically stable for minibatch training.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Riemannian Deep Learning: Modules, Networks, and Geometries arXiv:2607.19305