ML: Inference speedup

Machine-learning ideas tagged Inference speedup in the ML taxonomy of the Math2NN corpus.

✓✓ Beats tuned baseline 2026

Posterior-Routed Low-Rank Diffusion Experts

Use the noisy diffusion state to estimate posterior probabilities over data clusters, then route the denoising computation to one or a few cluster-specific score experts. Each expert stores a low-rank covariance or low-rank score correction, so after posterior commitment the denoiser operates at the cluster intrinsic dimension rather than ambient dimension. During the early mixing phase, retain a shared full-dimensional score or evaluate a small weighted set of experts.

Useful8/10
Difficulty6/10
Novelty6/10
Paper: Diffusion Models for High-Dimensional Clustered Data: Intrinsic-Dimension Adaptivity via Bayesian Classification arXiv:2608.19067
△ Mechanism confirmed, baseline not beaten 2026

Rate-adaptive KV-cache quantizer

Allocate different entropy budgets to different KV-cache blocks instead of assigning every token and head the same nominal bitwidth. Use the ECASQ Lagrangian so high-variance or attention-sensitive blocks receive more codepoints, while predictable blocks collapse to fewer symbols and become highly compressible. Preserve unbiasedness per scalar or block so reconstructed keys and values have zero mean quantization error conditional on the original tensor.

Useful8/10
Difficulty6/10
Novelty7/10
Paper: Entropy-Constrained Adaptive Stochastic Quantization arXiv:2608.18147
△ Mechanism confirmed, baseline not beaten 2026

Entropy-constrained unbiased block quantization

Replace fixed-grid stochastic quantization of each tensor block with an adaptive ordered codebook selected under both an entropy budget and a maximum number of codepoints. Within every interval between adjacent codepoints, use unbiased stochastic interpolation, so the quantized block remains unbiased while the emitted symbol distribution becomes easier for arithmetic or Huffman coding to compress. The representation should reduce actual compressed bytes at fixed MSE, or reduce MSE at fixed…

Useful8/10
Difficulty6/10
Novelty6/10
Paper: Entropy-Constrained Adaptive Stochastic Quantization arXiv:2608.18147
✓✓ Beats tuned baseline 2026

Path-work correction for exact neural proposals

Use the learned path only as a global proposal, then correct complete trajectories rather than endpoints. Exponentiated negative work gives self-normalized importance weights, while the same path ratio gives an independent Metropolis acceptance probability. This turns an imperfect neural sampler into an asymptotically exact sampler whenever forward and reverse path laws overlap.

Useful8/10
Difficulty4/10
Novelty6/10
Paper: Neural Non-Equilibrium Hamiltonian Monte Carlo for Corrected Boltzmann Sampling arXiv:2607.15682
△ Mechanism confirmed, baseline not beaten 2026

Tail-Aware Verifier Portfolio

Use the paper's tail comparison to decide when another call from the same verifier family is useless and when to switch to a different model, modality, or evidence source. The objective is to reduce the high-alpha survivor population—the incorrect examples that consistently fool one verifier—rather than maximizing average one-shot verifier accuracy.

Useful8/10
Difficulty5/10
Novelty7/10
Paper: Partially Correlated Verifier Cascades in LLM Harnesses: Concave Log-Odds, Polynomial Reliability, and Blind-Spot Ceilings arXiv:2607.13918
✗ Failed on benchmark 2026

Audited Risk-Budgeted Early Exit

Attach a cheap risk score to each neural-network prediction and skip an expensive verifier, ensemble, diffusion refinement, retrieval call, or human review when the score is below a calibrated threshold. Independently audit a random subset of skipped examples using the expensive ground-truth procedure, and select the largest skip threshold whose exact confidence bound keeps the violation rate below a target budget.

Useful8/10
Difficulty4/10
Novelty7/10
Paper: Audited Selective Verification for Risk-Controlled N-1 Thermal Contingency Screening under Deployment Shift arXiv:2607.13221
✗ Mechanism failed 2026

Floor-Aware Adaptive Block Drafting

Estimate the irreducible rejection caused by missing within-block information, then use it to choose the block's conditioning order instead of tuning block length blindly. If the estimated floor is high, expose one or more realized tokens before continuing; if the floor is low but observed rejection is high, spend compute on improving the drafter.

Useful8/10
Difficulty5/10
Novelty7/10
Paper: Beyond Parallel Blindness: Information Floors and Model Gaps in Block Drafting arXiv:2608.27339
✗ Mechanism failed 2026

Retry-aware ignition-threshold router

Route requests between model-quality tiers using retry-adjusted satisfied-answer throughput instead of nominal completion throughput. Add hysteresis so degradation begins only above an upper backlog threshold and ends only after the backlog is safely below a lower threshold with negative retry-adjusted drift.

Useful8/10
Difficulty4/10
Novelty7/10
Paper: The Shadow Price of Intelligence: Quality Degradation in LLM Inference as a Supply Chain Problem arXiv:2608.23986
✓ Mechanism works 2026

Leave-One-Out Corrective Parallel Sampler

Replace standard tau-leaping in discrete diffusion generation with a first-order sampler whose per-coordinate transition is conditioned on all other current coordinates and excludes the coordinate being updated. After a parallel proposal, use the same leave-one-out conditionals to correct coordinates whose newly sampled values are inconsistent with the rest of the state, allowing large timesteps without permanently propagating simultaneous denoising errors.

Useful8/10
Difficulty5/10
Novelty7/10
Paper: Provably adaptive sampling with uniform and remasking discrete diffusion models arXiv:2608.23554
△ Mechanism confirmed, baseline not beaten 2026

Utility-Weighted Left-Edge Quantization

Replace MSE-calibrated scalar quantization with a conservative left-edge quantizer whose thresholds are denser where activation probability and task utility slope are both high. For a monotone utility function, this should preserve high-impact activation regions better than uniform or MSE-optimal bins at the same number of codes, while retaining an explicit rate-versus-quality design rule.

Useful7/10
Difficulty4/10
Novelty7/10
Paper: A Quantization Problem Posed by Adaptive Streaming arXiv:2609.03745
✗ Mechanism failed 2026

Ranking-Aware Projection Dimension Rule

Replace the usual JL dimension choice with a task-specific dimension budget based on the paper's asymptotic ranking law. For retrieval, nearest-neighbor search, or compressed attention keys, choose m from a target Kendall correlation and automatically expand the projection when measured ranking quality falls below the target.

Useful7/10
Difficulty3/10
Novelty7/10
Paper: Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian Models arXiv:2609.02155
✗ Failed on benchmark 2026

Centered-Geometry Projection Loss

Train a low-dimensional projection of embeddings against centered pairwise geometry instead of only using raw-distance preservation or a JL-style guarantee. The loss removes the population or minibatch distance baseline before comparing distances, forcing the bottleneck to retain the fluctuations that carry ranking and task information.

Useful7/10
Difficulty4/10
Novelty6/10
Paper: Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian Models arXiv:2609.02155
✗ Mechanism failed 2026

Pointwise complexity-gated inference

Use a local chaining complexity computed from an empirical input metric to predict stochastic output error for each individual input. Easy, locally concentrated inputs can use fewer dropout, ensemble, or diffusion samples, while high-complexity inputs receive additional computation; unlike a global confidence threshold, the allocation varies with the input.

Useful7/10
Difficulty5/10
Novelty6/10
Paper: Pointwise Majorization for sub-Weibull and Mixed Tail Processes with Applications in Quadratic Chaos and Ergodic Diffusions arXiv:2609.01576
✗ Failed on benchmark 2026

Wavelet Conditional Sampler for Neural EBMs

Represent an image or spatial latent state in an orthogonal multiresolution wavelet basis and learn the conditional distribution of detail coefficients at each scale given all coarser coefficients. At inference time, sample coarse coefficients first and recursively sample finer coefficients, using a small conditional network or a few local Langevin steps at each level instead of running a long pixel-space Markov chain. The mechanism should remove critical slowing down caused by long-range…

Useful7/10
Difficulty6/10
Novelty6/10
Paper: Overcoming critical slowing down in frustrated spin systems by learned multiscale sampling arXiv:2608.31114
✗ Mechanism failed 2026

Two-Solve Robust Expert Router

Replace point-estimate expert routing with a nominal allocation and a least-favourable allocation under uncertainty in expert quality. If both allocations agree, use that route confidently; if they disagree, profile or evaluate only the expert-input pairs responsible for the disagreement. The same mechanism can be used offline to assign workloads to LLMs or online to choose among heterogeneous experts under a latency or FLOP budget.

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation arXiv:2608.29560
△ Mechanism confirmed, baseline not beaten 2026

Signed spectral attention

Replace a quadratic pairwise attention or graph aggregation kernel with a compact, translation-invariant indefinite kernel approximated by signed random Fourier features. The feature map preserves the kernel's negative spectral mass through a diagonal sign matrix, so the resulting linear-time aggregation can represent similarities that ordinary positive-definite random features cannot.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: Signed random Fourier features for fast density estimation with indefinite kernels arXiv:2608.29265
✗ Failed on benchmark 2026

AoI Water-Filling for Neural Data Refresh

Use the renewal Age of Information model to schedule refreshes from heterogeneous federated clients, sensors, retrieval indexes, or world-model observation streams. Sources with high downstream importance and reliable, cheap updates receive shorter refresh periods, while unreliable or expensive sources are refreshed less often. Pack the resulting requests into a non-overlapping communication schedule.

Useful7/10
Difficulty4/10
Novelty7/10
Paper: Age-Optimal Target Wake Time: Provably Good Wake Schedules for Energy-Constrained Wi-Fi Status Updating arXiv:2608.21596
△ Mechanism confirmed, baseline not beaten 2026

Spectral Cross-Block Averaging Layer

Construct a cheap graph or token-mixing operator by partitioning nodes into k blocks using the bottom nonconstant eigenvectors of P squared, then replacing dense pairwise mixing with conditional averaging inside each block followed by one baseline propagation step. Unlike ordinary spectral clustering, the bottom modes target partitions where block labels are rapidly destroyed by P, producing an aggressively mixing representation layer rather than a community-preserving pooling layer. The…

Useful7/10
Difficulty6/10
Novelty7/10
Paper: Spectral partitioning for $k$-block averaging kernels of finite Markov chains arXiv:2608.21466
△ Mechanism confirmed, baseline not beaten 2026

Residual-Pivoted Kernel Attention

Replace full PSD self-attention with a pivoted Cholesky/Nyström approximation whose landmarks are sampled from the unexplained diagonal mass. Tokens with large residual self-similarity are more likely to become landmarks, so the rank budget is spent on difficult regions rather than uniformly selected tokens.

Useful7/10
Difficulty5/10
Novelty5/10
Paper: A new analysis of the randomly pivoted Cholesky algorithm arXiv:2608.20633
✗ Failed on benchmark 2026

Target-Law Neural Stopping

Learn a path-dependent stopping policy for a stochastic neural trajectory so that the state at stopping time matches a prescribed target distribution, instead of optimizing only a scalar terminal reward. This can turn a fixed-length diffusion sampler or iterative latent refinement process into an adaptive sampler that stops early when its sample distribution is already sufficiently close to the target.

Useful7/10
Difficulty5/10
Novelty7/10
Paper: When to Sell an Asset? - A Distribution Builder Approach arXiv:2608.18783
✗ Mechanism failed 2026

Permutation-Mixed Orthogonal Quantization

Precondition activation or cache blocks with a fixed product U = A Sigma B of orthogonal transforms and a random signed permutation before quantization or coordinate sampling. The random permutation makes the product incoherent, so energy is less concentrated in a few coordinates and lossy compression should introduce less worst-case distortion.

Useful7/10
Difficulty5/10
Novelty5/10
Paper: Randomly Permuted Orthogonal Products and Fast Dimension Reduction arXiv:2608.18557
△ Mechanism confirmed, baseline not beaten 2026

Tau-leaped parallel discrete Hamiltonian sampler

Approximate the exact event-by-event lifted sampler by drawing independent Poisson jump counts over a short interval and applying compatible discrete moves in parallel. This converts sequential neighbor events into batched GPU-friendly updates while retaining the Hamiltonian rate structure; the step size controls the error-versus-throughput tradeoff.

Useful7/10
Difficulty5/10
Novelty8/10
Paper: Hamiltonian dynamics for sampling on discrete spaces arXiv:2608.17961
✗ Failed on benchmark 2026

Kac-rotated fast projection

Replace a dense Haar or Gaussian random projection with a streamed product of random two-coordinate rotations followed by coordinate subsampling. The transform is exactly orthogonal before subsampling, requires only a list of rotation triples, and the paper's pseudo-mixing result predicts that degree-two statistics relevant to norm preservation and Johnson–Lindenstrauss embeddings become Haar-like after only O(n polylog(n)) rotations.

Useful7/10
Difficulty4/10
Novelty5/10
Paper: On the Pseudo-Mixing of Kac's Walk arXiv:2608.17374
△ Mechanism confirmed, baseline not beaten 2026

Square-Root Error-Density Timestep Grid

Construct a nonuniform diffusion timestep grid from an empirical local discretization-error density instead of using uniform time spacing or a fixed hand-designed schedule. The optimal allocation places shorter intervals where the score or posterior mean varies rapidly and longer intervals in regions where the reverse vector field is smooth.

Useful7/10
Difficulty4/10
Novelty6/10
Paper: Forward-Evolution Error Analysis and Adaptive Design for Matrix-Valued Diffusion Models arXiv:2608.15103