arXiv:2608.06796 2026 Architecture 2 ideas extracted · analyzed Aug 31, 2026
The paper provides a constructive online batching framework in which maximum batch delay is controlled by a one-parameter family of nested partitions rather than by request-by-request urgency. The transferable asset is the combination of submodular service cost, a quadrangle inequality, and a timer whose partitions become monotonically coarser as the parameter increases. This can become a neural-inference or distributed-training scheduler that trades batching efficiency against staleness while preserving an explicit worst-case-style envelope. The randomized global parameter is especially useful for avoiding adversarial synchronization patterns that defeat any fixed batching threshold.
Replace a fixed maximum-wait timeout in dynamic neural inference batching with a service-aware timer. The timer estimates the marginal reduction in computation or communication cost from adding another request and delays service only while that saving justifies the additional oldest-request delay.
Sample one batching aggressiveness parameter once per serving epoch and use it for all busy periods, instead of independently randomizing every batch. This globally coupled randomness preserves nested batching behavior while preventing a fixed traffic pattern from repeatedly aligning with a deterministic timeout.