# Saturation-Adaptive Prefill Chunking

- ID: 1663
- Canonical URL: https://synthcore.org/idea/1663/saturation-adaptive-prefill-chunking
- API JSON: https://synthcore.org/api/idea/1663.json
- API Markdown: https://synthcore.org/api/idea/1663.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.01250](https://arxiv.org/abs/2608.01250)
- Category: dynamics
- Solves: stability, scalability, memory
- ML areas: transformer, inference-speedup, batching, scheduler
- Math tags: stochastic-processes, statistics, optimization, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace fixed chunked-prefill settings in an LLM serving engine with a feedback controller that decreases the number of prompt tokens processed per scheduling quantum as GPU saturation or long-context load increases. The controller targets a high-quantile bound on the absolute GPU-power ramp while preserving the existing peak-power ceiling and measuring the resulting latency-throughput tradeoff.

## Mathematical statement

Let $D(t)$ be aggregate GPU demand or measured power at time $t$, and let $R^{*}(\epsilon)$ be the smallest reserve or ramp capacity exceeded with probability at most $\epsilon$. The paper defines $R^{*}(\epsilon)=\min_R\{R:\Pr(|D'(t)|>R)\leq\epsilon\}$, equivalently the $(1-\epsilon)$ quantile of $|D'(t)|$. For a serving scheduler, let $c$ be the number of prefill tokens executed per quantum, $s_t$ be a saturation statistic such as active sequences divided by capacity, and $w_t$ be the fraction of queued whale requests whose prompt length exceeds a threshold. Use a monotone controller $c_t=\operatorname{clip}(c_{\max}-k_s s_t-k_w w_t,c_{\min},c_{\max})$, with nonnegative gains $k_s,k_w$. Estimate $R^{*}(\epsilon;c)$ from power traces under chunk size $c$ and select the largest $c$ satisfying $\widehat R^{*}(\epsilon;c)\leq R_{\mathrm{target}}$. The mechanism predicts that smaller $c$ reduces ramp-rate quantiles mainly at high saturation or whale-request load, while peak-power quantiles change little.

## Key formulas

- $$R^{*}(\epsilon)=\min_{R}\left\{R:\Pr\left(|D'(t)|>R\right)\leq\epsilon\right\}=Q_{1-\epsilon}\left(|D'(t)|\right),$$
- $$c_t=\operatorname{clip}\left(c_{\max}-k_s s_t-k_w w_t,\ c_{\min},\ c_{\max}\right),\qquad s_t=\frac{\text{active sequences at }t}{\text{maximum active sequences}},$$
- $$\widehat R^{*}(\epsilon;c)=\operatorname{Quantile}_{1-\epsilon}\left(\left\{\frac{|P_{i+1}-P_i|}{\Delta t}\right\}_{i=1}^{N_c}\right),\qquad \widehat c_t=\max\{c:\widehat R^{*}(\epsilon;c)\leq R_{\mathrm{target}}\},$$
- $$\text{Reserve reduction}(\epsilon)=1-\frac{\widehat R^{*}_{\mathrm{adaptive}}(\epsilon)}{\widehat R^{*}_{\mathrm{fixed}}(\epsilon)}.$$

## Implementation notes

1. Exact integration point: modify the continuous-batching scheduler in vLLM, TensorRT-LLM, or a minimal PyTorch transformer server. At every scheduling interval, the scheduler chooses how many tokens from each queued prompt enter the prefill forward pass; decoding requests retain priority and the model weights are unchanged. Measure GPU power through NVIDIA NVML at 10--100 Hz, active-sequence count, queue length, prompt lengths, token throughput, and time-to-first-token. 2. Pseudocode: initialize candidate chunk sizes $c$; at each interval compute $s_t$ and $w_t$; set $c_t=\operatorname{clip}(c_{\max}-k_s s_t-k_w w_t,c_{\min},c_{\max})$; run at most $c_t$ prefill tokens; record $P_t$; periodically bootstrap windows of consecutive power samples and compute the $(1-\epsilon)$ quantile of $|P_{t+1}-P_t|/\Delta t$; if it exceeds $R_{\mathrm{target}}$, reduce $c_{\max}$ or increase the feedback gains. 3. Taken from the paper: the chance-constraint objective, quantile criterion, and prediction that chunking affects ramps more than peaks. Estimated empirically: $k_s,k_w$, power-trace quantiles, and latency or throughput costs. 4. First cheap experiment: serve Llama-2-7B or a comparable decoder-only model on one GPU using ShareGPT prompts, mixing short requests with 0%, 25%, and 50% long-context prompts at low, medium, and high concurrency. Compare unchunked, fixed chunking, and adaptive chunking. The predicted signature is nearly unchanged peak power, a monotone decrease in mean and high-quantile ramp with saturation, and approximately 7--35% mean-ramp reduction from light to heavy concurrency; whale-heavy workloads should approach the reported 42.6% reduction. Bootstrap estimates should show roughly a 20--23% reduction in the 95--99.9% reserve quantile at the representative operating point, subject to hardware and serving-stack differences.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a readable synthetic serving/power-trace experiment implementing the saturation/whale feedback controller, ramp quantile estimation, and largest-feasible-chunk selection. All four quantitative mechanism predictions passed: controller slope -28.0 versus -28 predicted, ramp reserve increased by 0.237 units/token, high-load reduction was 56.4% versus 8.6% at low load, and peak-power q99 changed only -3.0%. Against fixed chunking, adaptive chunking reduced q95 ramp by 45.8%, but also reduced the throughput proxy by 34.2%, so the benefit is a real ramp/throughput tradeoff rather than an unconditional serving win.

### Mechanism check

- Verdict: Built a readable synthetic serving/power-trace experiment implementing the saturation/whale feedback controller, ramp quantile estimation, and largest-feasible-chunk selection. All four quantitative mechanism predictions passed: controller slope -28.0 versus -28 predicted, ramp reserve increased by 0.237 units/token, high-load reduction was 56.4% versus 8.6% at low load, and peak-power q99 changed only -3.0%. Against fixed chunking, adaptive chunking reduced q95 ramp by 45.8%, but also reduced the throughput proxy by 34.2%, so the benefit is a real ramp/throughput tradeoff rather than an unconditional serving win.
- Confidence: 8/10
- Limitations: This is a controlled synthetic mechanism test, not an actual vLLM, TensorRT-LLM, or PyTorch LLM server benchmark. It does not measure real NVML power, GPU utilization, TTFT, end-to-end throughput, queueing effects, FLOPs, or model quality, and the peak-power invariance is enforced by the toy demand model rather than established on hardware.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.36405
- Paired wins: 6/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/616/bench_report.json)
- [bench_run.py](https://synthcore.org/code/616/bench_run.py)
- [experiment.py](https://synthcore.org/code/616/experiment.py)
- [report.md](https://synthcore.org/code/616/report.md)
- [report_bench_2026-08-31T195100.md](https://synthcore.org/code/616/report_bench_2026-08-31T195100.md)
- [results.json](https://synthcore.org/code/616/results.json)
- [Download all files as ZIP](https://synthcore.org/download/616)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.