TL;DR — Instead of a fixed prefill chunk size, we propose SLOWeave, an online scheduler that binary-searches for the “largest prefill chunk that finishes before the earliest decode deadline” at every iteration. Against the strongest fixed-chunk baseline it improves mixed-workload goodput by 39% and long-workload goodput by 38% at a 25 ms TPOT target, and pushes those gains to 3.3× and 2.4× respectively under the strict 10 ms target (source: Abstract).

Serving-Systems on Jaehun's Blog

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

Deadline-Filling Prefill Chunks: SLOWeave — An Adaptive Chunking Scheduler for LLM Serving