1. The Processes One Request Passes Through
Open an LLM serving framework for the first time and you will probably go looking for the scheduler. But no amount of reading scheduler code answers the question that…
Mini-SGLang: reading the scheduler of an LLM serving framework
Eight parts reading sgl-project/mini-sglang
,
pinned to commit 9a91cfa. Every quoted snippet links to a permalink at that commit.
It starts from the process boundaries one HTTP request crosses, then works through the Req/Batch state ledger, the chunked-prefill budget, page allocation, the radix cache and eviction that make prefix reuse possible, hiding CPU scheduling behind two streams, and how several TP ranks reach the same decision.
Each part assumes the previous one, so reading from part 1 works best.
8 pages
2. Req and Batch: the ledger for one request
In chapter 1, what reached the scheduler was a 1-D int32 tensor on the CPU. While that token sequence grows into an answer, the system has to write down somewhere how far…
3. The budget behind a prefill batch, and chunked prefill
In chapter 2 a request became a ledger expressed by three lengths. Now there is one question left: with several requests waiting, what goes into this step, and how much…
4. Paged allocation, and a page table that holds raw locations
In chapter 3 a request "secured its space". What that space actually is, is this chapter's subject. The KV cache is managed in pages. Yet open the pagetable that each…
5. The radix cache that reuses prefixes, and eviction
Chapter 4 showed how one request secures its space. But if a hundred requests arrive with the same system prompt and each repeats the same computation, that space runs…
6. Hiding CPU scheduling behind two streams
Across five chapters we have watched what the scheduler does. It receives messages, computes budgets, walks a tree, allocates pages, fills a table. All of it is CPU work.…
7. Where the ledger becomes kernel arguments, and CUDA graphs
At the end of chapter 6 we listed the seven lines of preparebatch and left two of them alone: padbatch and preparemetadata. This chapter opens both. Everything built so…
8. How several ranks share one ledger
Chapter 1 showed that one scheduler starts per TP rank, and the six chapters since read what happens inside one of them. Now all of it gets replicated once per rank. That…