1. The Processes One Request Passes Through
Open an LLM serving framework for the first time and you will probably go looking for the scheduler. But no amount of reading scheduler code …
8 min
Mini-SGLang: reading the scheduler of an LLM serving framework
Eight parts reading sgl-project/mini-sglang
,
pinned to commit 9a91cfa. Every quoted snippet links to a permalink at that commit.
It starts from the process boundaries one HTTP request crosses, then works through the Req/Batch state ledger, the chunked-prefill budget, page allocation, the radix cache and eviction that make prefix reuse possible, hiding CPU scheduling behind two streams, and how several TP ranks reach the same decision.
Each part assumes the previous one, so reading from part 1 works best.
8 pages
2. Req and Batch: the ledger for one request
In chapter 1, what reached the scheduler was a 1-D int32 tensor on the CPU. While that token sequence grows into an answer, the system has …
8 min
3. The budget behind a prefill batch, and chunked prefill
In chapter 2 a request became a ledger expressed by three lengths. Now there is one question left: with several requests waiting, what goes …
7 min
4. Paged allocation, and a page table that holds raw locations
In chapter 3 a request “secured its space”. What that space actually is, is this chapter’s subject. The KV cache is …
8 min
5. The radix cache that reuses prefixes, and eviction
Chapter 4 showed how one request secures its space. But if a hundred requests arrive with the same system prompt and each repeats the same …
9 min
6. Hiding CPU scheduling behind two streams
Across five chapters we have watched what the scheduler does. It receives messages, computes budgets, walks a tree, allocates pages, fills a …
7 min
7. Where the ledger becomes kernel arguments, and CUDA graphs
At the end of chapter 6 we listed the seven lines of _prepare_batch and left two of them alone: pad_batch and prepare_metadata. This chapter …
8 min
8. How several ranks share one ledger
Chapter 1 showed that one scheduler starts per TP rank, and the six chapters since read what happens inside one of them. Now all of it gets …
8 min