Architecture ideas

Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.

✗ Mechanism failed 2026

Unconstrained Proper-Velocity Hyperbolic Layers

Replace Lorentz-hyperboloid tensors with proper-velocity tensors whose spatial coordinates can be transformed by standard Euclidean affine layers and activations. Reconstruct the Lorentz time coordinate only at manifold boundaries, preserving the hyperbolic representation while avoiding repeated projection, normalization, or fragile exponential-map calculations.

Useful7/10
Difficulty4/10
Novelty6/10
Paper: Riemannian Deep Learning: Modules, Networks, and Geometries arXiv:2607.19305
✓✓ Beats tuned baseline 2026

Time-Shell Long-Horizon Decoder

Replace dense pairwise interactions between all forecast horizons with nested time-shell summaries. For sorted horizons, the readout at shell j receives a cumulative embedding of all coefficients or queries assigned to later horizons, reproducing the paper's dependence on products such as \(\Pi_j=\prod_{l>j}e^{\alpha_l}=e^{\sum_{l>j}\alpha_l}\). This gives an \(O(Kd)\) multi-horizon interaction instead of an \(O(K^2d)\) temporal attention block and should work best for weak-memory…

Useful7/10
Difficulty5/10
Novelty8/10
Paper: Dynamical correlation functions of extensive charges after global quantum quenches arXiv:2607.19208
△ Mechanism confirmed, baseline not beaten 2026

Delay-Aware Frequency-Preserving Recurrent Coupling

For coupled recurrent or state-space modules that represent oscillatory or periodic signals, explicitly account for communication or attention delay in the characteristic equation. Tune the coupling gain or add a phase-lead compensator so that the desired latent frequency remains a closed-loop mode instead of being shifted by small delays.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: How network perturbations distort agreement trajectories in LTI multi-agent systems arXiv:2607.18913
△ Mechanism confirmed, baseline not beaten 2026

OT-Sufficient Bottleneck Flow Matching

Replace a conventional regression bottleneck with an encoder whose representation is trained to preserve the conditional law of the target through conditional optimal transport. The encoder produces a low-dimensional z, while a conditional velocity field transports a fixed reference distribution into the observed target distribution given z; minimizing flow-matching error forces z to retain multimodality, conditional variance, and other distributional information.

Useful7/10
Difficulty6/10
Novelty6/10
Paper: Learning sufficient low-dimensional structures through conditional optimal transport arXiv:2607.18861
✗ Failed on benchmark 2026

Balanced design attention

Construct overlapping attention windows from a block design instead of using one dense sequence-by-sequence attention matrix. Every token appears in exactly $r$ windows and every token pair co-occurs in exactly $\lambda$ windows, giving uniform coverage and avoiding the uneven connectivity of arbitrary sparse masks.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Block designs and systems of pairs arXiv:2607.18499
✗ Mechanism failed 2026

Critical Spectral Mode Compression

Replace a large diagonalizable recurrent or state-space transition operator by a sparse set of retained oscillatory modes selected according to their contribution to the output autocorrelation. Unlike magnitude-based pruning, the objective is to preserve the power-law return signal generated by pairwise spectral differences, enabling long memory with far fewer modes.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Information Compression at Criticality arXiv:2607.18388
✗ Failed on benchmark 2026

Spectral-Band Dual-Timescale Network

Split hidden dynamics into relaxation bands when the Jacobian spectrum has a gap, evolve each band with its own timescale, and retain an explicit cross-band exchange term. This yields a principled dual-timescale RNN or SSM rather than choosing fast and slow branches heuristically.

Useful7/10
Difficulty6/10
Novelty6/10
Paper: Richards' equation as a hydrodynamic limit: Chapman--Enskog reduction of the continuum kinetic equation for unsaturated soil water arXiv:2607.17358
△ Mechanism confirmed, baseline not beaten 2026

Monotone CDT autoencoder bottleneck

Build an autoencoder whose decoder outputs a monotone quantile function rather than an unconstrained spatial field. The latent representation can be compressed with POD or a neural bottleneck in CDT space, while the decoder guarantees valid transport maps and therefore avoids negative densities, mass drift, and spurious oscillations common in unconstrained reduced-order neural decoders.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Reduced Order Modeling of One-Dimensional Conservative PDEs via the Cumulative Distribution Transform arXiv:2607.17066
△ Mechanism confirmed, baseline not beaten 2026

Noise-Triggered Latent Rank Adaptation

Use the recursive errors-in-variables subspace spectrum as a controller for the width of a latent SSM rather than fixing the state dimension in advance. Neurons or state channels are added when corrected covariance eigenvalues rise above the noise floor and pruned when they remain below it, producing a model-order-adaptive recurrent architecture for nonstationary streams.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: A recursive subspace based method for errors-in-variables model identification of time-varying systems arXiv:2607.17065
△ Mechanism confirmed, baseline not beaten 2026

Gain-Weighted Cluster Co-Design

Use small-gain diagnostics to jointly learn module normalization and a communication partition rather than imposing a fixed global spectral constraint. Clusters should be formed around high-gain feedback loops, because grouping weakly related modules cannot improve the certificate and only adds bookkeeping.

Useful7/10
Difficulty7/10
Novelty7/10
Paper: Cluster-Based Distributed Small-Signal Stability Certificates for Grid-Forming Inverter Networks arXiv:2607.16985
✗ Failed on benchmark 2026

Collision-aware physical-support abstention

Modify a learned sparse encoder so that it distinguishes reliable group activation from unreliable child-ray identity. When test separation information is high but dictionary-orientation information is low, the model should output the active coherent group while abstaining on individual child labels, using a permutation- and sign-invariant support representation.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Honest Physical-Support Inference after Latent Dictionary Learning: Collision Singularities and Minimax Resolution arXiv:2607.16813
△ Mechanism confirmed, baseline not beaten 2026

Identity-Paired Progressive Depth

Grow a neural network by appending a trainable block together with an analytically initialized inverse block, so the newly added depth is exactly the identity at insertion time. After insertion, untie and optimize the two blocks independently; this preserves the current function while providing additional trainable degrees of freedom. For architectures with one expensive mixing operation followed by cheap channelwise blocks, the same construction can increase depth without repeatedly paying for…

Useful7/10
Difficulty5/10
Novelty5/10
Paper: Identity-Paired Progressive Depth Training: When Trainability Persists Beyond Expressibility arXiv:2607.16800
✗ Mechanism failed 2026

KS-Adaptive Graph Halting

Use the KS ratio to decide how many message-passing layers to execute per graph or per node, rather than selecting a fixed depth. In the subcritical regime, stop once the predicted remaining effect is below a tolerance; in the supercritical regime, continue until the observed logit change becomes small or a larger budget is reached.

Useful7/10
Difficulty5/10
Novelty5/10
Paper: The Value of Depth in Message Passing on Sparse Graphs: A Kesten-Stigum Dichotomy arXiv:2607.16676
△ Mechanism confirmed, baseline not beaten 2026

Residual-Redundancy Adapter Clustering

Replace one globally shared LoRA adapter with a small set of adapters whose task membership is chosen by residual redundancy. Tasks with strongly correlated validation residuals share an adapter, while tasks with weak or antagonistic residual dependence receive separate adapters. Recompute the partition periodically so the architecture follows the coupling that remains after training rather than correlations in the raw labels or initial gradients.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Capacity and Redundancy Trade-offs in Multi-Task Learning arXiv:2607.16554
✓✓ Beats tuned baseline 2026

Symmetry-Quotiented Local Correlation Encoder

Replace raw molecular orientation vectors with local scalar features invariant under common three-dimensional rotations and the apolar transformation u_i -> -u_i. Feed these channels to a CNN autoencoder, VAE, or contrastive encoder so that configurations on the same physical symmetry orbit have identical inputs or latent codes. This should improve unsupervised phase discovery without supplying order-parameter labels.

Useful7/10
Difficulty4/10
Novelty5/10
Paper: Representation-Dependent Machine Learning of the Isotropic-Nematic Transition in the Lebwohl-Lasher Model arXiv:2607.16481
△ Mechanism confirmed, baseline not beaten 2026

Influence-Adaptive Strategic Quantization

Insert a topology-controlled strategic communication layer into graph neural networks: each node maps a bounded latent scalar to either a clipped amplified signal or an interval-quantized message, with the amplification determined by how much influence the receiver exerts on the sender. Weakly influential communication channels should become aggressively quantized, while highly influential channels retain more resolution. This creates a principled variable-rate message-passing architecture…

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Network-Induced Strategic Communication in Opinion Dynamics arXiv:2607.16036
△ Mechanism confirmed, baseline not beaten 2026

Event-driven shared-neuron graph

Replace a conventional feed-forward block with a sparse temporal graph whose hidden units are shared across many computation paths. Each arriving message updates a shared accumulator, applies a nonlinear response, and schedules delayed messages to downstream neurons; constructive or destructive interaction emerges when multiple paths visit the same unit.

Useful7/10
Difficulty6/10
Novelty8/10
Paper: NeuronSoup: Evolving Asynchronous, Shared-Neuron Temporal Graphs without Backpropagation arXiv:2607.15217
✗ Failed on benchmark 2026

KL-TopK Activation Bottleneck

Construct a neural activation bottleneck by projecting hidden states into a fixed covariance-eigenbasis and retaining only the d largest-magnitude coordinates per sample. For Gaussian, decorrelated activations, the paper proves that adaptive top-d selection in the PCA basis has no greater expected residual energy than adaptive top-d selection after any other orthogonal rotation. This provides a principled alternative to learning an unrestricted rotation before sparsification.

Useful7/10
Difficulty3/10
Novelty5/10
Paper: A Correlation-Gap Bound for Nonlinear Gaussian PCA arXiv:2607.15035
✗ Mechanism failed 2026

Memory-Retaining RG Feature Blocks

Replace scale-blind pooling or downsampling with a coarse-graining block that carries an explicit relevant scale variable \(\eta\) alongside the feature field. The block is constrained to represent features in the memory-retaining form \(h(\xi,\eta)=\eta^{\alpha}F(\xi/\eta^{\beta})\), allowing both feature amplitude and profile shape to depend on the scale inherited from the input or previous RG step.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Memory Retention and the Classification of Renormalization-Group Fixed Points in Self-Similar Dynamics arXiv:2607.14388
✗ Failed on benchmark 2026

Nonlinear Noise-Tightening Drift

Replace a linear restoring drift in score-based sampling, latent dynamics, or stochastic regularization with a state-dependent nonlinear restoring term that is at least as contractive globally and more contractive away from the origin. This should reduce stationary variance without changing the worst-case local contraction certificate.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Wasserstein Stability of Contracting Flows: Effective Rates, Euler Self-Correction, and Noise Tightening arXiv:2607.14291
△ Mechanism confirmed, baseline not beaten 2026

Harmonic-coordinate neural PDE ansatz

Build a complex-valued coordinate map q(x) whose components are harmonic and whose gradients are mutually null, then feed q(x) into an otherwise unconstrained neural function v. Any learned output of the form u(x)=v(q(x)) is analytically harmonic when the constraints are satisfied, so the network does not need to rediscover the Laplace structure from collocation data. This is especially suitable for two-dimensional elliptic PDEs, where q=x+iy is the canonical example.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Harmonic Variables for Laplace Operators on Homogeneous Spaces arXiv:2607.14132
✗ Failed on benchmark 2026

ISS-Constrained Modular Recurrent Network

Replace an unconstrained recurrent block with two coupled modules: a contractive perceptual estimator and an input-to-state-stable cognitive state transition. Spectral normalization and a controlled Euler residual step enforce a quantitative gain condition, preventing hidden-state explosion while retaining long memory when the contraction factor is chosen close to one.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: A modular state-space model of human perception, cognition, and decision dynamics arXiv:2607.14078
✓✓ Beats tuned baseline 2026

Moment-Calibrated Verification Stopping

Replace a fixed-depth all-accept verifier cascade with a depth controller calibrated to the latent distribution of per-instance false-accept rates. The controller should stop when the predicted reliability gain from another gate is smaller than its inference cost, avoiding the severe overconfidence caused by treating correlated verdicts as independent evidence.

Useful7/10
Difficulty4/10
Novelty7/10
Paper: Partially Correlated Verifier Cascades in LLM Harnesses: Concave Log-Odds, Polynomial Reliability, and Blind-Spot Ceilings arXiv:2607.13918
✓✓ Beats tuned baseline 2026

Geometric Feedback Compute Scheduler

Treat unresolved inference items as active threats and allocate a fixed budget of C module evaluations per round. Each evaluation has an item-dependent probability of completing the item, while the scheduler observes only completion or failure after the round. Use fair allocation when completion probabilities are unknown or nearly homogeneous, then switch to a marginal-success greedy policy as feedback estimates become reliable.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Meeting Uncertain Threats with Feedback arXiv:2607.13648