1. The Processes One Request Passes Through
Open an LLM serving framework for the first time and you will probably go looking for the scheduler. But no amount of reading scheduler code …
8 min
Posts
All posts on technology, daily life, and thoughts.
74 pages
1 /
13
2. Req and Batch: the ledger for one request
In chapter 1, what reached the scheduler was a 1-D int32 tensor on the CPU. While that token sequence grows into an answer, the system has …
8 min
3. The budget behind a prefill batch, and chunked prefill
In chapter 2 a request became a ledger expressed by three lengths. Now there is one question left: with several requests waiting, what goes …
7 min
4. Paged allocation, and a page table that holds raw locations
In chapter 3 a request “secured its space”. What that space actually is, is this chapter’s subject. The KV cache is …
8 min
5. The radix cache that reuses prefixes, and eviction
Chapter 4 showed how one request secures its space. But if a hundred requests arrive with the same system prompt and each repeats the same …
9 min
6. Hiding CPU scheduling behind two streams
Across five chapters we have watched what the scheduler does. It receives messages, computes budgets, walks a tree, allocates pages, fills a …
7 min