Unverified
2026
Construct a sparse token-interaction graph from attention affinities and recursively split it by removing the closed neighborhoods of a small set of separator tokens. Separator tokens retain global communication, while the resulting anti-adjacent components perform local attention independently, reducing quadratic attention and KV-cache costs. The induced Erdos-Pósa theorem supplies a structural diagnostic: graphs with few anti-adjacent long-cycle or theta packings should admit small…
Useful6/10
Difficulty7/10
Novelty8/10
Unverified
2026
Replace a fixed optimizer learning-rate field by a positive state-dependent scaling rho(theta) and penalize expansion of weighted parameter-space volume. The optimizer is encouraged to contract regions of parameter initializations that have high weighted divergence, potentially reducing sensitivity to initialization and stabilizing training near sharp or anisotropic loss landscapes.
Useful6/10
Difficulty5/10
Novelty7/10
Unverified
2026
Separate a neural network into nonlinear hidden parameters and a linear output layer. Solve the output layer exactly by least squares, then update hidden parameters with a truncated-pseudoinverse Gauss-Newton step that discards numerically singular directions.
Useful6/10
Difficulty6/10
Novelty5/10
Unverified
2026
Replace explicit RK integration in a stiff neural ODE or continuous-depth residual network with the paper's stiffly accurate SDIRK4 discretization. Instead of performing a dense Newton solve for each implicit stage, solve the diagonal stage equation using a Chebyshev-accelerated residual iteration whose polynomial damps the negative, high-magnitude Jacobian modes responsible for stiffness.
Useful6/10
Difficulty7/10
Novelty7/10
Unverified
2026
Replace an unpreconditioned conjugate-gradient solve for a damped Gauss–Newton step with a two-level algebraic preconditioner derived from local Jacobian-row supports. Use overlapping local parameter blocks as Schwarz subdomains and a coarse basis containing low-energy local modes, so the optimizer can correct both localized and globally coupled parameter errors.
Useful6/10
Difficulty6/10
Novelty6/10
Unverified
2026
Replace abrupt optimizer preconditioner changes with a metric trajectory that moves the smallest affine-invariant distance needed to reach a target generalized Hessian condition number. During training, optimize a short horizon of log-diagonal or block-SPD metrics using a terminal curvature penalty and an intrinsic kinetic regularizer, then execute only the first metric in a receding-horizon controller. The method should reduce oscillations caused by rapidly changing second-moment estimates…
Useful6/10
Difficulty6/10
Novelty6/10
Unverified
2026
Use the paper's negative-semidefinite interaction curvature to detect and compensate for destructive coupling among layerwise learning-rate, momentum, or preconditioner mechanisms. Instead of independently tuning mechanism amplitudes, estimate their reduced curvature after hidden optimizer states relax, then apply a low-rank trust-region step or freeze mechanisms whose interaction curvature is too negative.
Useful6/10
Difficulty5/10
Novelty7/10
Unverified
2026
Replace the usual linear predictor in continuation of an implicit neural state with a fractional-power predictor fitted from recent states, then correct the prediction using a pseudo-arclength constraint. This is designed for equilibrium layers, implicit sequence models, or homotopy training schedules where the state Jacobian becomes nearly singular and ordinary Newton correction or fixed-point iteration becomes unstable.
Useful6/10
Difficulty6/10
Novelty8/10
Unverified
2026
Replace a fixed or hand-tuned learning-rate schedule with a slowly exponentially increasing schedule, and restart the schedule whenever the update norm grows at least as fast as the schedule itself. The restart preserves the current parameters but resets the learning-rate multiplier, allowing the optimizer to repeatedly approach the largest locally stable step size without requiring a Hessian spectrum or a reliable initial learning-rate guess.
Useful6/10
Difficulty4/10
Novelty5/10
Unverified
2026
Replace a dense weight matrix by a cross approximation built from selected rows and columns rather than by a conventional truncated SVD. Periodically refresh the selected indices using residual leverage scores, warm-starting from the previous factorization so that the compressed layer can track weight changes during fine-tuning.
Useful6/10
Difficulty5/10
Novelty5/10
Unverified
2026
Enumerate structured pruning masks only up to exact permutations of exchangeable channels, hidden units, or experts. Replace exhaustive mask search with canonical augmentation: retain a subset only when it is lexicographically smallest among all masks obtained by the model's symmetry group, while recursively generating only canonical predecessors.
Useful6/10
Difficulty5/10
Novelty8/10
Unverified
2026
Insert an active-set reduction step into a binary energy layer or Hopfield-style discrete optimizer. Coordinates whose signs are stable and whose local fields have a rigorous margin are frozen, while their interactions are folded into an induced bias and only the unresolved tail is updated. This preserves the exact conditional quadratic objective and can reduce dense interaction cost substantially when the state becomes polarized.
Useful6/10
Difficulty5/10
Novelty7/10
Unverified
2026
Represent every mesh interface degree of freedom by one feature copy per incident cell, and apply local neural blocks directly to these cell tensors. Enforce inter-cell consistency with valence-weighted averaging only after selected layers or hierarchy transitions, avoiding repeated construction of a global sparse graph or assembled feature vector. This is suited to adaptive quadtrees, octrees, and finite-element neural operators.
Useful6/10
Difficulty5/10
Novelty7/10
Unverified
2026
Apply the paper's orthogonal-sketch idea to a trainable low-rank weight update or low-rank optimizer state. After every optimizer step, use a row sketch and Cholesky QR to recondition the factor basis, then compute the update with the standard tangent-space projector rather than an oblique sketched projector.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace repeated full-curvature factorizations when evaluating many Bayesian neural-network hyperparameter settings with one shared low-rank approximation of the data-induced precision update. Reuse the same curvature subspace while changing prior precision, observation-noise precision, or layerwise regularization, evaluating each candidate through a small rank-r matrix and Woodbury solves.
Useful6/10
Difficulty5/10
Novelty5/10
Unverified
2026
Reduce the SDP dimension by removing ReLU neurons that are provably inactive, always active, or incapable of changing the current worst-case margin. The verifier keeps exact affine substitutions for fixed-sign neurons and retains only genuinely unstable neurons whose activation can cross zero under the perturbation set. A stronger optional rule removes an unstable neuron when its certified maximum possible contribution is below a user-controlled tolerance, with an exact mode that prunes only…
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Turn attention weights into a Boolean support scenario and prune edges using local-surjectivity constraints rather than independently thresholding each row. Preserve at least one compatible continuation for every local window, then favor a strongly connected support graph so pruning removes redundant mixtures while maintaining a globally coherent sparse attention pattern.
Useful6/10
Difficulty5/10
Novelty7/10
Unverified
2026
Replace standard heavy-ball momentum with an update derived from a discrete kinetic-minus-loss action and a discrete viscous force. The force discretization produces a rational damping factor that remains controlled over a specified range of step sizes, potentially reducing oscillations and instability without Adam-style second-moment state.
Useful6/10
Difficulty4/10
Novelty6/10
Unverified
2026
Use the paper's non-permutation-invariant overshoot bound as a runtime guard for large learning rates. A proposed step is accepted only if its predicted overshoot contribution is compatible with the observed gradient residual; otherwise the optimizer clips or shrinks the step, preventing isolated very large updates from causing delayed divergence.
Useful6/10
Difficulty4/10
Novelty7/10
Unverified
2026
Use the expander decoder as a structured sparse-coding dictionary and replace dense OMP correlation steps with edge-wise gather-and-reduce operations. This is useful when codes must be inferred iteratively, including interpretable feature extraction, sparse retrieval, or an inference-time latent selector that cannot rely entirely on an amortized encoder.
Useful6/10
Difficulty4/10
Novelty6/10
✗ Mechanism failed
2026
Replace the ordinary gradient of a spatially indexed parameter tensor by a Fourier-domain inverse-metric gradient. FFT the gradient over its spatial dimensions, divide every frequency by a positive spectral symbol, inverse FFT, and then apply the optimizer step. Use a Bessel/Sobolev symbol as a parameter-free baseline and optionally estimate a task-specific symbol from gradient power spectra.
Useful6/10
Difficulty4/10
Novelty6/10
Unverified
2026
Replace a generic neural constitutive law or energy model with an ICNN that consumes the positive singular values of a deformation-like matrix and is convex and coordinatewise nondecreasing in those inputs. Train it as a lower approximation to a nonconvex target energy, so the network acts as a computationally cheap sufficient polyconvex-envelope surrogate rather than merely interpolating unstable samples.
Useful6/10
Difficulty4/10
Novelty4/10
Unverified
2026
Represent the PINN solution in a restricted polynomial or Taylor basis whose exponent set is supplied by tropical support analysis, instead of asking an MLP to discover the local series structure from scratch. The restriction removes coefficients that cannot occur in the formal solution, reducing trainable degrees of freedom and preventing spurious low-order or singular terms.
Useful6/10
Difficulty5/10
Novelty8/10
Unverified
2026
Treat a coupled neural training loop as a delayed feedback system with two hard delays and two first-order implementation filters. Estimate the dominant coupled Jacobian mode and use the characteristic equation to distinguish a recoverable delay-induced oscillation from a filter-induced instability; then reduce stale-gradient delay only in the former case, and slow or retune EMA or relaxation filters in the latter.
Useful6/10
Difficulty5/10
Novelty6/10