Architecture ideas

Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.

Unverified 2026

Constraint-Free Skew Coupling

Compose independently parameterized neural dynamical modules through power-preserving skew coupling instead of equality penalties or projected constraints. This creates a modular graph or world model in which information exchanged between modules is antisymmetric, so internal coupling cannot create or destroy total latent energy.

Useful6/10
Difficulty6/10
Novelty6/10
Paper: Mixed finite element discretization of intrinsic geometrically exact beams for explicit multibody dynamics arXiv:2607.20245
Unverified 2026

Hard-Rod Symmetry Invariant Module

Construct a scalar feature or critic for oscillator-based neural dynamics that is invariant under the transformations imposed by free harmonic motion and elastic collisions. For finite-size rods, the module should represent only quantities compatible with common oscillator-phase rotations and momentum permutations, preventing a learned world model from inventing coordinate-dependent pseudo-conserved quantities that disappear after collisions.

Useful6/10
Difficulty5/10
Novelty6/10
Paper: Absence of hidden analytic conserved quantities in harmonically confined rods arXiv:2607.18872
Unverified 2026

Moment-Resolved Stochastic Reservoir Readout

Replace mean-only readout from a noisy recurrent or Langevin reservoir by concatenating empirical first, second, and fourth raw moments of each hidden coordinate. The second and fourth moments retain input-dependent width and tail information generated by nonlinear confinement, while multiple independently initialized reservoirs can be concatenated before the final linear classifier to preserve complementary features.

Useful6/10
Difficulty5/10
Novelty6/10
Paper: Moment-Resolved Readout and Reservoir Diversity in Nonequilibrium Langevin Computing arXiv:2607.14520
Unverified 2026

Threshold-Projection Recurrent Memory

Replace or augment a recurrent cell with multiple hysteresis memory branches whose states remain unchanged while the input stays within a branch-specific radius, then move toward the current input only when that radius is exceeded. The resulting cell has explicit persistence and bounded state changes, giving it an inductive bias for temporal hysteresis and reducing the need for the network to learn long-term memory behavior from scratch.

Useful6/10
Difficulty4/10
Novelty7/10
Paper: Accounting for Hysteresis and Eddy Currents in Finite Element Simulations of Ferromagnetic Laminated Cores using a Recurrent Neural Network arXiv:2607.14321
Unverified 2026

Signature-memory neural CDE

Replace an unconstrained recurrent memory with a truncated path-signature state that is updated continuously from the input control path. Feed this structured state to a learned vector field, allowing the model to represent path-dependent dynamics through iterated integrals of the entire history rather than only the latest hidden state.

Useful6/10
Difficulty5/10
Novelty5/10
Paper: Dynamic Universal Approximation via Signature Controlled Differential Equations arXiv:2607.13886
Unverified 2026

Finite-Orbit Circulant State Core

Replace the linear state transition in a small recurrent or state-space module by a circulant matrix acting on a vector over a finite field. The hidden state then has only finitely many possible values and follows an exactly periodic orbit after at most \(q^n\) states, eliminating numerical drift on modular-counting and symbolic-memory tasks. A learned real-valued encoder and decoder can surround the discrete core, while the transition itself is fixed, searched, or trained with a…

Useful6/10
Difficulty6/10
Novelty8/10
Paper: Periodicities in the Riordan arrays of polynomials over finite fields arXiv:2607.13442
Unverified 2026

Saturating Trail Memory for Asynchronous Multi-Agent Networks

Equip multiple recurrent agents with a shared spatial or token-level trail field whose influence is a bounded function of accumulated visitation, rather than an unbounded additive memory. Use the paper's simultaneous/sequential invariance as a falsifiable design target: parallel and randomly ordered asynchronous agent updates should produce nearly identical predictions when trail occupancy is saturated, while deliberately nonsaturating controls should show order dependence. This can enable…

Useful6/10
Difficulty5/10
Novelty7/10
Paper: Exact collective first-passage statistics of N trail-interacting walkers arXiv:2607.13213
Unverified 2026

Algebraic-Invariant Residual Layer

Represent a rational-like feature transformation with an auxiliary state y constrained by polynomial equations G(x,y)=0, and update x and y jointly along the tangent space of that constraint manifold. This creates residual blocks in which nonlinear feature identities remain consistent over many layers or time steps, reducing auxiliary-variable drift and potentially stabilizing rational activations and implicit recurrent dynamics.

Useful6/10
Difficulty5/10
Novelty8/10
Paper: Algebraic Invariant Quadratization Schemes for Cahn--Hilliard Equations arXiv:2607.11569
Unverified 2026

Projected Play-State Memory

Turn a recurrent or state-space memory into a constrained hereditary state: the latent state remains in a learned convex domain, and only input motion that reaches the boundary changes the plastic component. This creates a nonexpansive, rate-independent memory that should suppress unstable state growth and make the representation depend on meaningful cumulative changes rather than arbitrary update frequency.

Useful6/10
Difficulty4/10
Novelty5/10
Paper: Optimal history encoding for elastic-plastic hereditary laws: Sharp input and constitutive approximation arXiv:2607.09974
✗ Mechanism failed 2026

Thermodynamic Two-State Expert Gate

Add a slow latent two-state gate to a recurrent, state-space, or world-model network so that separate experts represent two qualitatively different dynamical regimes. Train the gate using the paper's two-state population and fluctuation mechanism rather than allowing an unconstrained softmax to average incompatible regimes. The model should allocate extra capacity near the gate's susceptibility peak, where regime uncertainty and forecast variance are predicted to be largest.

Useful6/10
Difficulty5/10
Novelty6/10
Paper: Structural Origin of Water Heat Capacity Anomaly from Classical and Quantum Simulations arXiv:2607.08957
△ Mechanism confirmed, baseline not beaten 2026

Cross-Ratio Reversible Lattice Layer

Represent a hidden state as complex-valued points on a two-dimensional lattice and replace unconstrained local updates by the exact harmonic-quadrilateral completion rule from discrete conformal geometry. Given three corners of a plaquette, compute the fourth corner by a Mobius-rational formula enforcing cross-ratio minus one, then use a learned readout or forcing term for task-specific predictions. The layer supplies a hard geometric inductive bias and a directly measurable local constraint…

Useful6/10
Difficulty6/10
Novelty8/10
Paper: Integrability of Cauchy problems for discrete conformal maps and circle patterns arXiv:2607.08901
✗ Mechanism failed 2026

Innovation-Compensated Latent Policy

In a partially observed reinforcement-learning or model-based control agent, expose the state-estimator innovation to the action head through a dedicated residual feedback branch. The policy produces a nominal action from the estimated latent state, while a learned innovation-compensation branch corrects actions when observations disagree with predicted latent dynamics. This explicitly separates nominal policy behavior from estimation-induced corrections and should help during fast transients…

Useful6/10
Difficulty5/10
Novelty6/10
Paper: Revisiting Certainty Equivalence: The Structural Coupling Between Estimation and Control in Underactuated Nonlinear Systems arXiv:2607.07276
✗ Failed on benchmark 2026

Age-conditioned semi-Markov router

Augment a neural router with the age of its current expert or latent regime and use an age-dependent hazard to determine when switching is likely. Unlike ordinary token-wise softmax routing, the router can learn non-geometric residence times, suppressing unstable expert oscillations while still allowing rapid transitions when the current regime becomes inappropriate.

Useful6/10
Difficulty5/10
Novelty7/10
Paper: Semi-Markovian switching in a fluctuating harmonic trap: An age-structured formulation arXiv:2607.05173
△ Mechanism confirmed, baseline not beaten 2026

Bilinear Input-Conditioned Koopman Cell

Replace an unconstrained input-conditioned recurrent transition with a bilinear latent update, so controls modulate a fixed linear latent dynamics matrix through low-rank state-input interactions. The resulting cell preserves the computational simplicity of linear propagation while representing multiplicative effects of actions that an additive control term cannot capture efficiently.

Useful6/10
Difficulty5/10
Novelty5/10
Paper: Koopman operator theory: fundamentals, control, and applications arXiv:2607.01819
✗ Mechanism failed 2026

Periodic CMV Unitary Recurrent Layer

Replace a dense recurrent transition matrix with a periodic CMV-style product of alternating local 2x2 unitary cores. The transition is exactly norm-preserving, has O(n) trainable parameters under periodic tying, and can be applied through local factor operations rather than stored as an n-by-n matrix. Use turnover refactorization when changing the ordering or boundary connection of cores, enabling a compact cyclic unitary state-space layer.

Useful6/10
Difficulty5/10
Novelty5/10
Paper: Fast computation of eigenvalues of periodic CMV matrices arXiv:2607.06400
✗ Mechanism failed 2026

Heisenberg latent upsampler

Represent each latent state as a Heisenberg-group element and replace Euclidean interpolation in an upsampling or recurrent transition block by a four-point horizontal refinement plus the exact central signed-area correction. The module preserves the geometry of noncommutative composition, allowing the central latent coordinate to encode path-dependent information that ordinary coordinate-wise interpolation discards.

Useful6/10
Difficulty5/10
Novelty8/10
Paper: A Heisenberg Subdivision Scheme with Central Smoothness Loss arXiv:2607.05446
✗ Mechanism failed 2026

Dual-unitary recurrent state block

Replace a generic recurrent transition with two coupled unitary transitions that share one block column and differ by a sign on the other block column. Each transition preserves hidden-state norm exactly, while the structured difference gives a controlled two-path recurrent architecture for long-context modeling.

Useful6/10
Difficulty5/10
Novelty5/10
Paper: Operator model and a trace formula for pairs of unitary operators arXiv:2607.05334
✗ Failed on benchmark 2026

Orthogonal-Rank Contextual Memory

Replace a discrete or one-hot recurrent state table with a low-dimensional vector memory whose event embeddings are orthogonal whenever the corresponding events are mutually exclusive in an input exclusivity graph. The module uses continuous state vectors and can therefore target dimension \(d=\xi(G)\), whereas a discrete state encoding is lower-bounded by \(N\geq\chi(G)\). This should be tested on graph-defined formal-language recognition tasks, where the graph is known and the claimed…

Useful6/10
Difficulty6/10
Novelty7/10
Paper: Quantum Memory Advantage from Contextuality arXiv:2607.00507
Unverified 2026

Companion Observer Memory for Neural Policies

Replace an unrestricted GRU or attention-based history encoder with a fixed companion-form shift register driven by the current action and observation, followed by a learned nonlinear policy. The register stores a structured finite history, while a learned matrix or MLP readout maps that history to a control-relevant latent state. This should provide a cheaper and more interpretable memory mechanism for partially observed environments, especially when the relevant dynamics are approximately…

Useful5/10
Difficulty4/10
Novelty6/10
Paper: Data-Driven Output Feedback based Analysis and Control for Unknown Discrete-Time Linear System arXiv:2608.18452
Unverified 2026

Velocity-Scaled Symbolic Flow Model

Represent a continuous-time neural dynamical system as a symbolic Markov chain over regions together with a positive learned roof function giving the time spent in each region. Weight local reconstruction and prediction errors by the predicted vector-field speed, following the paper's scaled Hölder coding relation, so that the model does not over-penalize arbitrarily small coordinate errors near equilibria. This produces a hybrid latent model with discrete long-range structure and continuous…

Useful5/10
Difficulty6/10
Novelty7/10
Paper: Symbolic dynamics for non-uniformly hyperbolic flows arXiv:2608.14095
Unverified 2026

Discrepancy-bounded signed fast memory

Replace an unrestricted additive recurrent or fast-weight memory with a sign-selectable update: for each incoming update vector, choose between adding and subtracting it so that a smooth compact potential of the memory state is minimized. This is appropriate when the memory representation has sign symmetry, such as signed random features or a learned linear sketch; it is not a drop-in replacement for ordinary gradient updates where the sign carries semantic information.

Useful5/10
Difficulty5/10
Novelty9/10
Paper: Online balancing of vectors with small coordinates arXiv:2608.12490
Unverified 2026

Spectral Phase-Lifted Features

Augment a hidden representation with positively homogeneous interaction features built from approximate eigenmodes of a linear layer. Fractional products of mode magnitudes and phases provide nonlinear channels whose transformation laws are inherited from the spectrum of the underlying operator, potentially representing oscillatory or multiplicative dynamics more compactly than a generic MLP.

Useful5/10
Difficulty7/10
Novelty8/10
Paper: The spectrum of operator extensions to free Banach Lattices arXiv:2608.11437
Unverified 2026

Seven-Factor Stochastic Transition Layer

Parameterize a learned 3-state transition operator as a product of at most seven elementary row-stochastic matrices rather than learning its nine entries independently. Each factor performs one convex pull-in of row i toward row j, so every intermediate and final matrix remains row-stochastic and the layer has a sparse, bounded-depth interpretation.

Useful5/10
Difficulty4/10
Novelty7/10
Paper: Bang--bang representation of $3\times 3$ embeddable stochastic matrices arXiv:2608.08242
Unverified 2026

Constant-gap brickwork unitary initialization

Initialize a unitary feature-mixing layer with a shallow brickwork circuit of independent random SU(4) gates instead of sampling or factorizing a dense Haar-random unitary. Stack enough layers to obtain a target contraction of non-Haar components, using the paper's constant spectral-gap principle to make the required depth essentially independent of the number of qubits. The resulting layer is local, parameter-efficient, exactly norm-preserving, and should provide Haar-like scrambling at…

Useful5/10
Difficulty5/10
Novelty6/10
Paper: Random unitary circuits with constant spectral gap arXiv:2607.20919