ML: Batching

Machine-learning ideas tagged Batching in the ML taxonomy of the Math2NN corpus.

2 ideas found

ML: Batching Math: Dynamical systems Clear all

✗ Failed on benchmark 2026

Saturation-Adaptive Prefill Chunking

Replace fixed chunked-prefill settings in an LLM serving engine with a feedback controller that decreases the number of prompt tokens processed per scheduling quantum as GPU saturation or long-context load increases. The controller targets a high-quantile bound on the absolute GPU-power ramp while preserving the existing peak-power ceiling and measuring the resulting latency-throughput tradeoff.

Useful7/10

Difficulty5/10

Novelty6/10

Dynamics Stability Scalability Memory

Paper: Smoothing the Ramp, Not the Peak: Scheduling-Induced Power Dynamics of LLM Inference and Their Grid-Scale Consequences arXiv:2608.01250

✗ Mechanism failed 2026

UCB Drift Router for Cheap-or-Expert Inference

Replace a fixed confidence threshold in cascaded inference or mixture-of-experts routing with a queue-aware UCB-DPP controller. The controller sends an input to a cheap model when its optimistic estimated success is sufficiently high and the expert backlog is large, while escalating uncertain or high-value inputs when the penalty for an error dominates congestion. This should reduce expensive-model utilization without allowing latency or escalation queues to diverge.

Useful7/10

Difficulty4/10

Novelty6/10

Dynamics Speedup Stability Accuracy

Paper: Learning When to Automate: Queue Control in Human-AI Service Systems arXiv:2607.06017