Architecture ideas

Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.

Unverified 2026

Fading-Memory Contractive Recurrent Block

Replace an unrestricted recurrent state with a current input-driven state coupled to a fading average of its own history. The memory coupling provides a distributed dissipative channel, while the recurrent Jacobian is constrained below a measurable contraction boundary. This creates a recurrent or state-space block whose long-horizon sensitivity can be tested against explicit characteristic roots.

Useful6/10
Difficulty5/10
Novelty7/10
Paper: Exponential Consensus and Flocking in Multi-Agent Systems with Infinite Fading Memory arXiv:2609.02454
Unverified 2026

Bicomplex Spatio-Temporal Residual Encoder

Replace independent per-sensor temporal encoders with a shared two-dimensional transform whose temporal and spatial phase channels remain algebraically distinguishable. Train a neural residual detector and localization head on the transformed coefficients, forcing perturbations with the same temporal signature but different positions to differ primarily in their spatial phase.

Useful6/10
Difficulty5/10
Novelty8/10
Paper: Multivariable Geometric Laplace Transform and Fault Detection in Distributed-Converter Lines arXiv:2609.02431
Unverified 2026

Bernstein-Safe Policy Head

Replace a neural controller's pointwise action outputs over a finite horizon with Bernstein control points whose convex hull satisfies actuator and trajectory constraints. The network predicts the control points, while a robust margin accounts for bounded tracking or model-prediction error, making continuous-time actuator feasibility checkable from finitely many inequalities.

Useful6/10
Difficulty5/10
Novelty7/10
Paper: Actuator-Aware Spatiotemporal Tube Synthesis for Temporal Reach-Avoid-Stay Tasks arXiv:2607.23040
Unverified 2026

PCA-Discovered Implicit Latent Dynamics

Replace an explicit recurrent transition with a learned descriptor relation in latent space, allowing some latent coordinates to satisfy algebraic constraints rather than being numerically integrated. Fit the relation using total-least-squares or iterative PCA on the jointly observed trajectory, so noise in every channel is treated symmetrically and the model can discover whether the latent system is index-0 or index-1.

Useful6/10
Difficulty6/10
Novelty7/10
Paper: Subspace Based Identification of Errors-in-Variables Linear Descriptor Systems arXiv:2608.30259
Unverified 2026

Cumulative-Control Surrogate Encoder

Use cumulative control measures as the input to a neural surrogate instead of the full sequence of control values. For a quasi-static free-boundary system satisfying the paper's average-pressure path-independence assumption, two nonnegative control histories with identical integrals up to time t should produce the same state at t, allowing a smaller training input and fewer distinct control trajectories in the dataset.

Useful6/10
Difficulty4/10
Novelty8/10
Paper: Online Gate-Driven Flow Control in Resin Transfer Moulding Using a Neural-Network Surrogate arXiv:2608.29521
Unverified 2026

Relative-Degree-Gated Passive Neural State Space

Construct a neural state-space model with an explicit first-order input-to-output path instead of forcing every output to depend only on deeply propagated hidden states. Penalize or reject learned linearizations whose transfer matrix has relative degree greater than one, then train a storage-function certificate for the remaining passive dynamics. This preserves the paper's relative-degree compatibility condition while allowing high-order internal memory.

Useful6/10
Difficulty6/10
Novelty7/10
Paper: Relative-Degree Wall Restricts Passivity-Based Stability Analysis in Inverter-Dominant Grids arXiv:2608.29474
Unverified 2026

Hard-Saturated Neural Feedback

Build actuator or parameter constraints directly into the neural controller using a differentiable hard-saturation map rather than penalizing violations after the fact. This makes the Lyapunov certificate apply to the actual bounded controller and prevents training from exploiting unrealistically large actions.

Useful6/10
Difficulty3/10
Novelty4/10
Paper: Learning neural controllers for nonlinear systems from data arXiv:2608.29303
Unverified 2026

Cost-Map Finite-Action Head

Replace online enumeration over a finite action set with a classifier or lookup map whose regions directly return the action minimizing a one-step predictive-control cost. For affine dynamics and quadratic tracking loss, exact action regions are separated by pairwise cost boundaries, so the approximation can be audited against exhaustive predictive control rather than treated as an unconstrained policy.

Useful6/10
Difficulty5/10
Novelty6/10
Paper: A Simple and Extremely Efficient Predictive Control for Power Converters arXiv:2608.22416
Unverified 2026

Bures-Shaped Latent State Space

Add a stable linear latent state-space block whose controllability Gramian is trained toward a chosen positive-definite target using squared Bures–Wasserstein distance. Direction-specific semidefinite constraints can suppress disturbance amplification in nuisance coordinates while preserving controllability in coordinates needed for prediction.

Useful6/10
Difficulty6/10
Novelty7/10
Paper: A Controllability Gramain Shaping with LMI Constraints under Bures--Wasserstein Distance arXiv:2608.19754
Unverified 2026

Integrable Elliptic Memory Cell

Replace an unconstrained recurrent transition with block-diagonal planar rotations whose angles are learned or conditioned on a slowly varying context variable. The resulting hidden-state norm and each two-dimensional block energy are exactly invariant in the ideal recurrence, preventing exploding or vanishing recurrent dynamics while retaining phase information over long horizons.

Useful6/10
Difficulty4/10
Novelty4/10
Paper: Outer Contact Billiards arXiv:2608.19393
Unverified 2026

BT-Critical Long-Memory Initialization

Construct a recurrent or state-space layer whose equilibrium Jacobian is placed near a nondegenerate Bogdanov–Takens point, then use a small unfolding parameter to move between damped, oscillatory, and slowly relaxing regimes. Unlike eigenvalue-only initialization near one, this controls both the double-zero center structure and the quadratic nonlinear coefficients that determine the local phase portrait.

Useful6/10
Difficulty6/10
Novelty8/10
Paper: The Bogdanov--Takens normal-form coefficients in $\mathbb{R}^n$ as directional derivatives of the characteristic invariants arXiv:2608.19018
Unverified 2026

Probability-Preserving Zonotopic Neural Uncertainty

Attach a finite mixture of zonotopes to each uncertain neural input or hidden state, and propagate every mixture component through affine layers and conservative nonlinear relaxations. When the number of components grows, merge components only with an enclosing zonotope and sum their probability masses, preserving a formal lower bound on the probability that the true activation lies in the represented set.

Useful6/10
Difficulty6/10
Novelty7/10
Paper: The Zonotopic Mixture Filter arXiv:2608.17897
Unverified 2026

Cactus-Graph Phase Budgeting

Build neural computation graphs with explicitly phase-budgeted serial and parallel branches, treating serial compositions as SRG products and parallel residual branches as SRG sums. Allocate phase centers theta_i so that every loop or branch aggregate stays away from -1, enabling stability-aware architecture search and constructive control of branch gains.

Useful6/10
Difficulty7/10
Novelty8/10
Paper: The $θ$-Symmetric SRG with Applications to Stability of Cactus Dynamic Networks arXiv:2608.12591
Unverified 2026

Single-Node Observable Leaky-RNN

Construct a sparse recurrent network with positive edge weights and Leaky-ReLU updates so that one selected hidden node, observed over a finite time window, contains enough information to reconstruct the full hidden state. Add an auxiliary decoder from the observed trajectory to the initial state or current state, and use graph rewiring or edge-growth until every hidden node has a directed path to the sensor within the observation horizon.

Useful6/10
Difficulty5/10
Novelty8/10
Paper: On the Observability and Controllability of Leaky-ReLU Networks arXiv:2608.09059
Unverified 2026

Invertible Fourier Surrogate for Periodic Sequence Modeling

Represent periodic input-output behavior using a compact real vector of Fourier coefficients and learn an invertible neural map from input coefficients to output coefficients. Inference then obtains the input representation for a desired periodic output by a single inverse pass instead of iterative optimization through a nonlinear forward model, while the Fourier representation reduces sequence dimensionality when high-rate signals are spectrally sparse.

Useful6/10
Difficulty6/10
Novelty4/10
Paper: Certified Feedforward Tracking for Unknown Nonlinear Systems via Invertible Neural Networks arXiv:2608.06419
Unverified 2026

KAM-Stabilized Quasiperiodic Recurrent Memory

Construct a recurrent module with a phase variable and a transverse memory coordinate modeled on a perturbed twist map. Train the transverse state to lie on an invariant graph over the phase, while the phase follows an approximately irrational rigid rotation. A KAM-inspired graph correction and residual penalty should reduce long-horizon drift in recurrent prediction.

Useful6/10
Difficulty6/10
Novelty8/10
Paper: Persistence of invariant graphs for twist maps under analytic perturbations arXiv:2608.05239
Unverified 2026

Noncommutative controllability regularizer

Equip a recurrent or state-space layer with multiple noncommuting transition operators and regularize the span of finite operator words applied to the input injection matrix. This discourages hidden directions that cannot be reached from the input and may improve long-range input influence, gradient propagation, and robustness under operator switching.

Useful6/10
Difficulty5/10
Novelty8/10
Paper: A noncommunicative Kalman condition for null controllability of backward stochastic parabolic systems arXiv:2608.01836
Unverified 2026

Normalized Scheduling-Degree Truncation

Use normalized scheduling variables and explicitly cap the degree of their products in a neural LPV or mixture-of-dynamics model. Instead of allowing every multiplicative interaction between scheduling coordinates and past or future features, retain only monomials below a chosen degree threshold. This produces a controllable approximation knob between a purely linear model and a full lifted predictor, while avoiding unstable extrapolation caused by poorly scaled high-degree features.

Useful6/10
Difficulty5/10
Novelty5/10
Paper: A subspace approach to data-driven predictive control for linear parameter-varying systems arXiv:2607.28490
Unverified 2026

Layered Structural Reachability for Neural States

Treat the hidden-state Jacobian of an RNN, SSM, or graph neural network as a directed matrix-weighted network and decompose repeated block couplings into scalar interaction layers. Use layer-specific structural controllability to select input, skip, reset, or readout channels that can reach all hidden dimensions, and reject architectures with structurally unreachable states before training.

Useful6/10
Difficulty6/10
Novelty7/10
Paper: On the Strong Structural Controllability of Matrix-Weighted Networks arXiv:2607.27852
Unverified 2026

Low-Order Robust Functional Observer

Attach a small dynamical observer to a neural ODE, RNN, or state-space model and make it estimate only a task-relevant functional of the hidden state, such as logits, value features, or control-relevant projections. Use an incremental quadratic constraint and a bounded-real penalty to make the observer robust to hidden-state nonlinearities and input disturbances, instead of reconstructing the full latent state.

Useful6/10
Difficulty6/10
Novelty5/10
Paper: Functional H_infinity Filtering for Descriptor Systems with Incrementally Quadratic Nonlinearities under Disturbances arXiv:2607.25000
Unverified 2026

Connectivity-aware ATC gradient tracking

Use the paper's mode decomposition to choose between ordinary DIGing and augmented ATC-DIGing/AugDGM according to the communication spectrum and curvature. The augmented scheme changes the disagreement feedback gain mode by mode, which can reduce the dominant pole radius on well-connected graphs without changing the neural-network architecture.

Useful6/10
Difficulty5/10
Novelty5/10
Paper: Exact Worst-case Convergence Rates of Distributed Gradient Tracking Methods arXiv:2607.23601
Unverified 2026

Imaginary-Axis Gramian Compression for Neural SSMs

Replace a large stable linear state-space or recurrent layer by a lower-order balanced realization computed from frequency-targeted controllability and observability Gramians. Use generalized low-rank ADI with imaginary-axis shifts concentrated at frequencies that dominate the training data, then retain states associated with the largest approximate Hankel singular values. This should reduce recurrent inference cost while preserving the layer's input-output response in the selected frequency…

Useful6/10
Difficulty6/10
Novelty7/10
Paper: A New Low-Rank Cholesky-Factor ADI Algorithm Allowing Shifts Anywhere in the Complex Plane with Applications to Data-Driven Model Reduction arXiv:2607.21969
Unverified 2026

Linear-solve ensemble controller

Add a shallow neural interpolation controller to a neural ODE or state-space model so one shared vector field matches prescribed derivatives at several anchor trajectories. At every control time, compute controller weights from a small linear system instead of learning all task-specific parameters by backpropagation.

Useful6/10
Difficulty5/10
Novelty5/10
Paper: Exact ensemble controllability for neural differential equations via neural interpolation arXiv:2607.21112
Unverified 2026

Deadline-Aware Fair-to-Greedy Router

Use deadline objectives to train or control a router that explicitly trades off completion probability against completed work by a fixed horizon. Begin with fair allocation for robust exploration, then anneal toward a feedback-greedy rule once per-item difficulty estimates have sufficient evidence.

Useful6/10
Difficulty6/10
Novelty7/10
Paper: Meeting Uncertain Threats with Feedback arXiv:2607.13648