✗ Failed on benchmark 2026

Saturation-Adaptive Prefill Chunking

Implementation & benchmark of arXiv:2608.01250 — Smoothing the Ramp, Not the Peak: Scheduling-Induced Power Dynamics of LLM Inference and Their Grid-Scale Consequences

Usefulness7/10

Difficulty5/10

Novelty6/10

Dynamics Stability Scalability Memory Transformer Inference speedup Batching Scheduler Stochastic processes Statistics Optimization Dynamical systems

Source paper: Smoothing the Ramp, Not the Peak: Scheduling-Induced Power Dynamics of LLM Inference and Their Grid-Scale Consequences arXiv:2608.01250 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace fixed chunked-prefill settings in an LLM serving engine with a feedback controller that decreases the number of prompt tokens processed per scheduling quantum as GPU saturation or long-context load increases. The controller targets a high-quantile bound on the absolute GPU-power ramp while preserving the existing peak-power ceiling and measuring the resulting latency-throughput tradeoff.

Formulas

$$R^{*}(\epsilon)=\min_{R}\left\{R:\Pr\left(|D'(t)|>R\right)\leq\epsilon\right\}=Q_{1-\epsilon}\left(|D'(t)|\right),$$

$$c_t=\operatorname{clip}\left(c_{\max}-k_s s_t-k_w w_t,\ c_{\min},\ c_{\max}\right),\qquad s_t=\frac{\text{active sequences at }t}{\text{maximum active sequences}},$$

$$\widehat R^{*}(\epsilon;c)=\operatorname{Quantile}_{1-\epsilon}\left(\left\{\frac{|P_{i+1}-P_i|}{\Delta t}\right\}_{i=1}^{N_c}\right),\qquad \widehat c_t=\max\{c:\widehat R^{*}(\epsilon;c)\leq R_{\mathrm{target}}\},$$

$$\text{Reserve reduction}(\epsilon)=1-\frac{\widehat R^{*}_{\mathrm{adaptive}}(\epsilon)}{\widehat R^{*}_{\mathrm{fixed}}(\epsilon)}.$$

Mathematical statement

Let $D(t)$ be aggregate GPU demand or measured power at time $t$, and let $R^{*}(\epsilon)$ be the smallest reserve or ramp capacity exceeded with probability at most $\epsilon$. The paper defines $R^{*}(\epsilon)=\min_R\{R:\Pr(|D'(t)|>R)\leq\epsilon\}$, equivalently the $(1-\epsilon)$ quantile of $|D'(t)|$. For a serving scheduler, let $c$ be the number of prefill tokens executed per quantum, $s_t$ be a saturation statistic such as active sequences divided by capacity, and $w_t$ be the fraction of queued whale requests whose prompt length exceeds a threshold. Use a monotone controller $c_t=\operatorname{clip}(c_{\max}-k_s s_t-k_w w_t,c_{\min},c_{\max})$, with nonnegative gains $k_s,k_w$. Estimate $R^{*}(\epsilon;c)$ from power traces under chunk size $c$ and select the largest $c$ satisfying $\widehat R^{*}(\epsilon;c)\leq R_{\mathrm{target}}$. The mechanism predicts that smaller $c$ reduces ramp-rate quantiles mainly at high saturation or whale-request load, while peak-power quantiles change little.

Implementation notes

1. Exact integration point: modify the continuous-batching scheduler in vLLM, TensorRT-LLM, or a minimal PyTorch transformer server. At every scheduling interval, the scheduler chooses how many tokens from each queued prompt enter the prefill forward pass; decoding requests retain priority and the model weights are unchanged. Measure GPU power through NVIDIA NVML at 10--100 Hz, active-sequence count, queue length, prompt lengths, token throughput, and time-to-first-token. 2. Pseudocode: initialize candidate chunk sizes $c$; at each interval compute $s_t$ and $w_t$; set $c_t=\operatorname{clip}(c_{\max}-k_s s_t-k_w w_t,c_{\min},c_{\max})$; run at most $c_t$ prefill tokens; record $P_t$; periodically bootstrap windows of consecutive power samples and compute the $(1-\epsilon)$ quantile of $|P_{t+1}-P_t|/\Delta t$; if it exceeds $R_{\mathrm{target}}$, reduce $c_{\max}$ or increase the feedback gains. 3. Taken from the paper: the chance-constraint objective, quantile criterion, and prediction that chunking affects ramps more than peaks. Estimated empirically: $k_s,k_w$, power-trace quantiles, and latency or throughput costs. 4. First cheap experiment: serve Llama-2-7B or a comparable decoder-only model on one GPU using ShareGPT prompts, mixing short requests with 0%, 25%, and 50% long-context prompts at low, medium, and high concurrency. Compare unchunked, fixed chunking, and adaptive chunking. The predicted signature is nearly unchanged peak power, a monotone decrease in mean and high-quantile ramp with saturation, and approximately 7--35% mean-ramp reduction from light to heavy concurrency; whale-heavy workloads should approach the reported 42.6% reduction. Bootstrap estimates should show roughly a 20--23% reduction in the 95--99.9% reserve quantile at the representative operating point, subject to hardware and serving-stack differences.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a readable synthetic serving/power-trace experiment implementing the saturation/whale feedback controller, ramp quantile estimation, and largest-feasible-chunk selection. All four quantitative mechanism predictions passed: controller slope -28.0 versus -28 predicted, ramp reserve increased by 0.237 units/token, high-load reduction was 56.4% versus 8.6% at low load, and peak-power q99 changed only -3.0%. Against fixed chunking, adaptive chunking reduced q95 ramp by 45.8%, but also reduced the throughput proxy by 34.2%, so the benefit is a real ramp/throughput tradeoff rather than an unconditional serving win.

Agent confidence: 8/10

Baseline: Fixed chunk=64: q95 ramp 28.73, peak q99 560.38, throughput proxy 64 tokens/quantum
Idea: Adaptive mean chunk=42.14: q95 ramp 15.58 (-45.8%), peak q99 543.68 (-3.0%), throughput proxy 42.14 (-34.2%); largest tested chunk under ramp target 20 was 16

Limitations: This is a controlled synthetic mechanism test, not an actual vLLM, TensorRT-LLM, or PyTorch LLM server benchmark. It does not measure real NVML power, GPU utilization, TTFT, end-to-end throughput, queueing effects, FLOPs, or model quality, and the peak-power invariance is enforced by the toy demand model rather than established on hardware.

How to run: python3 experiment.py > results.json

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.01250) Sep 2, 2026

Ahead of the baseline (−28.6%) but not statistically significant (p=0.364, wins 6 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0043
Idea mean: 0.003
Effect (Δ): -0.0012 (−28.6%; negative = idea better)
Wins: 6 / 8 paired seeds
p-value: 0.364 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±63.9%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

This is a dynamics-track transfer rather than a real LLM serving benchmark. It measures task MSE and a hardware-independent recurrent-state ramp proxy, not NVML GPU power, TTFT, queueing, throughput, or real prefill scheduling. The report's baseline sweep used learning rates 0.001, 0.003, and 0.01; the idea used the same union and eight paired seeds.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented Saturation-Adaptive Prefill Chunking as a dynamics-track chunked-GRU scheduling intervention with the identical rnn_small base architecture. The idea reduced mean MSE from 0.0042593 to 0.0030401, but the paired permutation test was not significant (delta_mean=-0.0012192, p=0.36405). The trained-model activation-ramp signature contradicted the prediction (0.04104 adaptive versus 0.02959 baseline), so there is no confirmed transfer win.

Baseline: dynamics/rnn_small, best lr=0.01: mean test MSE 0.004259285735315643 over 8 seeds
Idea: adaptive chunked rnn_small, best lr=0.01: mean test MSE 0.0030400985560845584; paired delta -0.0012191871792310849; 6/8 wins; p=0.36405; activation ramp 0.041038555558770895 versus 0.029594257241114974

Benchmark result