Architecture ideas

Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.

△ Mechanism confirmed, baseline not beaten 2026

MCIS Safety Shield for Neural Controllers

Compute an inner approximation of the states from which a neural controller can keep the plant inside a prescribed safe domain indefinitely, then use the resulting regulation map as a safety shield around the network. At each state, the network proposes an action, but the shield projects or replaces it with an action certified to remain in the invariant set.

Useful8/10
Difficulty6/10
Novelty6/10
Paper: Computing the Maximal Controlled Invariant Set for Neural Network Control Systems arXiv:2608.07908
✓✓ Beats tuned baseline 2026

Power-Balanced Modular Neural Block

Represent each neural module as a Hamiltonian storage system and connect modules through a state-dependent skew or Dirac interconnection instead of arbitrary residual additions. The coupling may change with the hidden state, but its internal power contribution cancels exactly, so total stored energy is controlled only by external inputs and explicitly added dissipation.

Useful8/10
Difficulty5/10
Novelty5/10
Paper: Port-Hamiltonian modelling of coupled rigid/flexible multibody systems arXiv:2608.05143
✗ Mechanism failed 2026

Orbit-Consistent Equivariant Distillation

Constrain a student policy to transform its action in the same way that the input state is transformed, while constraining its value estimate to remain unchanged. During distillation, augment every teacher-student pair with several symmetry-transformed copies and penalize disagreement after transforming the student action back to the original frame.

Useful7/10
Difficulty5/10
Novelty4/10
Paper: SymVD: Symmetric Vision Language Action Distillation for Robot Manipulation arXiv:2608.29828
✗ Failed on benchmark 2026

Hyperplane-Gated Piecewise Neural Dynamics

Replace a single smooth neural vector field with a finite collection of smooth subnetworks selected by learned affine hyperplanes. The architecture exposes switching geometry directly, allowing it to represent friction-like or threshold dynamics without approximating discontinuities using excessively steep activations.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Learning piecewise-smooth dynamical systems arXiv:2608.19785
△ Mechanism confirmed, baseline not beaten 2026

Measurement-Space Neural Operator with Mesh Transfer

Build a neural operator around explicit input and output measurement spaces rather than forcing the network to consume and emit a fixed grid. The same learned latent surrogate can be reused on alternative sensor layouts or query meshes through reconstruction and re-encoding maps, with a consistency loss enforcing agreement between measurement pipelines.

Useful7/10
Difficulty6/10
Novelty5/10
Paper: Kernel Methods for Learning Operators with Multiple Inputs and Outputs arXiv:2608.11831
△ Mechanism confirmed, baseline not beaten 2026

Finite Hyperplane Representative Verification

Replace dense continuous action search during neural-controller verification with a finite set of representative inputs induced by affine pieces of the interval neural dynamics. This makes safety checking parallel over state cells and candidate actions, enabling much cheaper certification or repeated safe-set updates.

Useful7/10
Difficulty7/10
Novelty8/10
Paper: Computing the Maximal Controlled Invariant Set for Neural Network Control Systems arXiv:2608.07908
△ Mechanism confirmed, baseline not beaten 2026

Toda-Krylov adaptive polynomial layer

Replace a fixed-order polynomial or recurrent state-space block by an Arnoldi basis built from a learned operator and the current input, and use subdiagonal coefficients as geometry-aware gates over Krylov depth. The gates quantify how much genuinely new direction each operator application contributes, allowing the layer to stop early near Krylov breakdown and suppress redundant or unstable directions.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Two-dimensional Toda--Arnoldi correspondence: Holomorphic Krylov geometry and counterdiabatic transport arXiv:2608.04850
✓✓ Beats tuned baseline 2026

Symmetry-Preserving Flow Layer

Construct hidden dynamics from permutation-equivariant vector fields and impose antisymmetry through an explicit antisymmetrizing readout. This prevents optimization from learning multiple equivalent copies of the same configuration and makes forbidden symmetry violations exactly zero, rather than merely penalizing them. The design applies to set models, particle systems, graph networks, and architectures handling unordered tokens.

Useful7/10
Difficulty5/10
Novelty4/10
Paper: Spindrift: Learning quantum degeneracy from thermal purity in restricted path integral Monte Carlo arXiv:2607.29590
✗ Failed on benchmark 2026

Probe-Then-Partitioned Multi-Task Trunk

Train a cheap shared multi-task probe briefly, extract one semantic embedding per task, and use density-based clustering to determine which tasks should share a neural trunk. After clustering, replace the globally shared trunk by one trunk per discovered cluster, with task heads remaining separate; this preserves cooperation among related tasks while isolating destructive task interactions.

Useful7/10
Difficulty5/10
Novelty5/10
Paper: Semantic-Aware Task Clustering for Constructive and Cooperative Multi-Tasking arXiv:2607.21426
△ Mechanism confirmed, baseline not beaten 2026

Cholesky-Structured SPD Classifier

Build an SPD classifier and residual head directly from Cholesky factors, using lower-triangular differences and matrix-power terms instead of generic eigendecomposition-based logarithm operators. This retains covariance geometry while making positive-definiteness automatic and backpropagation more numerically stable for minibatch training.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Riemannian Deep Learning: Modules, Networks, and Geometries arXiv:2607.19305
✗ Mechanism failed 2026

Unconstrained Proper-Velocity Hyperbolic Layers

Replace Lorentz-hyperboloid tensors with proper-velocity tensors whose spatial coordinates can be transformed by standard Euclidean affine layers and activations. Reconstruct the Lorentz time coordinate only at manifold boundaries, preserving the hyperbolic representation while avoiding repeated projection, normalization, or fragile exponential-map calculations.

Useful7/10
Difficulty4/10
Novelty6/10
Paper: Riemannian Deep Learning: Modules, Networks, and Geometries arXiv:2607.19305
△ Mechanism confirmed, baseline not beaten 2026

Harmonic-coordinate neural PDE ansatz

Build a complex-valued coordinate map q(x) whose components are harmonic and whose gradients are mutually null, then feed q(x) into an otherwise unconstrained neural function v. Any learned output of the form u(x)=v(q(x)) is analytically harmonic when the constraints are satisfied, so the network does not need to rediscover the Laplace structure from collocation data. This is especially suitable for two-dimensional elliptic PDEs, where q=x+iy is the canonical example.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Harmonic Variables for Laplace Operators on Homogeneous Spaces arXiv:2607.14132
✗ Failed on benchmark 2026

ZCA In-Context Output Transport

Train a neural surrogate to predict outputs in a source-domain ZCA-whitened space, then adapt to a shifted domain using only the shifted domain's output mean and covariance. At inference, transport the network prediction through the target covariance square root, yielding a weight-free correction that preserves output-coordinate semantics and can be applied to MLP, CNN, graph-NN, or transformer regressors.

Useful7/10
Difficulty4/10
Novelty5/10
Paper: Gradient-Free Topology Adaptation for Power Flow Surrogates via In-Context Whitening arXiv:2607.12241
✓✓ Beats tuned baseline 2026

Newton-Polytope Convex Network

Build a positively homogeneous convex network by representing every intermediate unit as a compact polytope and composing units with Minkowski sums, convex-hull unions, and positive dilations. This gives an explicitly convex and monotone architecture whose geometric complexity can be controlled independently of the number of sampled linear pieces, potentially producing smaller ICNNs for structured convex functions.

Useful7/10
Difficulty7/10
Novelty7/10
Paper: Tropical Circuits with Scalar Multiplication Gates arXiv:2607.11540
△ Mechanism confirmed, baseline not beaten 2026

Moment-Preserving Anisotropic Feature Tree

Replace a dense multiresolution voxel or hash-grid encoder with an omnitree-like anisotropic feature partition. Each cell stores a vector-valued scaling feature and its children are introduced only when local Haar detail energy is large; coarsening replaces children by their mean, so compression does not introduce an arbitrary offset. Splitting can be restricted to the coordinate whose one-dimensional detail coefficient is largest, allowing thin structures to receive resolution only in the…

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Towards Fully Dynamic Omnitrees: Moment-Conserving Anisotropic Compression With Wavelets arXiv:2607.04881
✓✓ Beats tuned baseline 2026

Differentiable R-Function Geometry Gate

Attach an analytic geometry gate to a KAN or MLP so that known feasible regions, exclusions, and unions are represented by differentiable implicit functions instead of being learned only from samples. Use R-conjunctions for intersections and R-disjunctions for unions, then convert the signed support score into a soft gate that modulates the prediction.

Useful7/10
Difficulty4/10
Novelty6/10
Paper: Geometry-Aware R-Structured Kolmogorov-Arnold Networks arXiv:2607.01449
△ Mechanism confirmed, baseline not beaten 2026

Dirichlet Spectral Projection Layers

Replace soft boundary penalties in neural operators with a hard projection onto a finite-dimensional span of homogeneous Dirichlet Laplacian eigenfunctions. Every projected hidden field is identically zero on the boundary, while increasing the number of retained eigenfunctions recovers the expressive capacity needed for operator approximation.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Enforcing Dirichlet Boundary Conditions in Operator Learning arXiv:2608.27256
Unverified 2026

Probability-coordinate robust likelihood head

Replace a Gaussian regression likelihood with a chart-pulled exponential-family head. The network predicts a natural parameter eta, while each target y is mapped to a bounded probability coordinate u=G(y). Because the sufficient statistic T(u) is bounded, extreme targets cannot create arbitrarily large parameter gradients. The chart determines the tail class, while eta controls the distribution in probability space.

Useful6/10
Difficulty4/10
Novelty6/10
Paper: G-Exponential Families through Probability Coordinates arXiv:2609.03208
Unverified 2026

Bernstein Bounded Implicit Representation

Parameterize a scalar or vector implicit neural field on each spatial cell with Bernstein polynomials and constrain its coefficients instead of sampling many points to enforce output bounds. The Bernstein convex-hull property gives a deterministic pointwise bound everywhere in the cell, making the method useful for neural fields representing densities, concentrations, masks, or material parameters.

Useful6/10
Difficulty4/10
Novelty7/10
Paper: Bounds-Constrained Finite Element Approximation of Time-Dependent Partial Differential Equations arXiv:2609.01915
Unverified 2026

Reciprocal Beta-Angle Mixer

Insert a projection-space mixer that combines several fixed or learned directions using reciprocal correlations with the current feature, then normalize the result. The exact construction has a universal beta law for its squared input-output cosine, so it can create controlled angular diversity while remaining deterministic and independent of the chosen direction dictionary.

Useful6/10
Difficulty5/10
Novelty8/10
Paper: Universal Beta Incidence Angles: Cauchy Rigidity and Infinite Arrangements arXiv:2609.00603
Unverified 2026

Dual-gauge cross-stream block

Replace an unconstrained hidden-to-hidden interaction in an MLP or transformer feed-forward block by two gauge-related branches. Split channels with an orthogonal involution Θ, constrain the learned interaction K to anticommute with Θ, and use opposite signs of K in paired branches. This creates a testable inductive bias in which the learned interaction only transfers information between the two channel subspaces.

Useful6/10
Difficulty5/10
Novelty8/10
Paper: Gauge-compatible tensors on statistical manifolds: splitting and submanifold geometry arXiv:2608.31145
Unverified 2026

Stable Hyperplane Feature Layer

Replace a dense directional feature map with a finite stable-zonotope layer: a small set of large random vectors creates a hyperplane arrangement over the input sphere, while a cheaper small-jump residual gives continuous features inside each cell. The layer exposes an explicit compute-versus-resolution knob through the jump cutoff and can be used either as an embedding before an MLP or as a lightweight router for mixture-of-experts blocks.

Useful6/10
Difficulty5/10
Novelty7/10
Paper: Hausdorff Dimension of the Set of Extreme Points of a Random Countable Stable Zonotope arXiv:2608.28004
Unverified 2026

Brauer O(2)-equivariant mixing layer

Construct a neural mixing layer only from Brauer generators for the orthogonal group: identity, pairwise swaps, and pairwise contractions with the Euclidean metric. This gives an exactly O(2)-equivariant alternative to unconstrained tensor mixing, with trainable coefficients but fixed symmetry-preserving basis maps.

Useful6/10
Difficulty5/10
Novelty5/10
Paper: The Brauer category $\mathcal{B}(2)$ has principal graph $D_\infty$ arXiv:2608.18328
Unverified 2026

Newton-envelope monomial layer

Replace a naively evaluated mixture of power-law experts with a Newton-envelope layer that computes all monomial magnitudes in log-space and subtracts their maximum before exponentiation. The layer exposes both a stabilized mixture value and soft dominance weights, allowing a downstream MLP to adapt to whichever scaling regime is active without overflow or hand-designed regime splits.

Useful6/10
Difficulty4/10
Novelty7/10
Paper: Newton Support Functions and Metric Completion of Singular Conformal Metrics at Corners arXiv:2608.17714