Mini-SGLang: reading the scheduler of an LLM serving framework on Jaehun's Blog

1. The Processes One Request Passes Through

Fri, 18 Sep 2026 00:00:00 +0900

Open an LLM serving framework for the first time and you will probably go looking for the scheduler. But no amount of reading scheduler code answers the question that actually blocks you: who calls this function, and who puts things in that queue? Mini-SGLang is not a single-process program. One HTTP request passes through at least four kinds of process before it becomes an answer, and every gap between them is crossed by a message and nothing else.

2. Req and Batch: the ledger for one request

Fri, 18 Sep 2026 00:00:00 +0900

In chapter 1, what reached the scheduler was a 1-D int32 tensor on the CPU. While that token sequence grows into an answer, the system has to write down somewhere how far this request has got. That ledger is Req.

Calling it a ledger is deliberate, and not because it has many fields. It is because progress is expressed by three lengths whose ordering is pinned down as an invariant. This chapter is about what those three numbers mean, and who moves them when.

3. The budget behind a prefill batch, and chunked prefill

Fri, 18 Sep 2026 00:00:00 +0900

In chapter 2 a request became a ledger expressed by three lengths. Now there is one question left: with several requests waiting, what goes into this step, and how much of it.

The answer is two budgets and five checks. And in how a prompt larger than the budget is handled, this repository makes one more tidy choice.

There are two budgets

Whatever builds the batch constructs a fresh PrefillAdder every time. Its first two arguments are the budgets.

4. Paged allocation, and a page table that holds raw locations

Fri, 18 Sep 2026 00:00:00 +0900

In chapter 3 a request “secured its space”. What that space actually is, is this chapter’s subject.

The KV cache is managed in pages. Yet open the page_table that each request owns a row of, and you find not page numbers but the location of every single token. Why write token locations into the table when allocation happens in pages? The repository answers that in a one-line comment.

The table holds raw locations

PYTHON

Collapse Copy

 # ======================= Page table initialization ========================
 # NOTE: 1. aligned to 128 bytes; 2. store raw locations instead of pages
 self.max_seq_len = min(config.max_seq_len, num_tokens)
 aligned_max_seq_len = _align_up_32(self.max_seq_len)
 self.ctx.page_table = self.page_table = torch.zeros( # + 1 for dummy request
 (config.max_running_req + 1, aligned_max_seq_len),
 dtype=torch.int32,
 device=self.device,
 )
 # ======================= Page table initialization ========================
 # NOTE: 1. aligned to 128 bytes; 2. store raw locations instead of pages
 self.max_seq_len = min(config.max_seq_len, num_tokens)
 aligned_max_seq_len = _align_up_32(self.max_seq_len)
 self.ctx.page_table = self.page_table = torch.zeros( # + 1 for dummy request
 (config.max_running_req + 1, aligned_max_seq_len),
 dtype=torch.int32,
 device=self.device,
 )

 
 
 

 
 

— python/minisgl/engine/engine.py:65-73

5. The radix cache that reuses prefixes, and eviction

Fri, 18 Sep 2026 00:00:00 +0900

Chapter 4 showed how one request secures its space. But if a hundred requests arrive with the same system prompt and each repeats the same computation, that space runs out fast.

So already-computed prefixes get reused. Once you start reusing, two new questions appear: how do you quickly find how far two sequences agree, and how do you know what may be thrown away when space runs short?

Walk the tree, and split where it diverges

Prefix lookup starts at the root and follows children down.

6. Hiding CPU scheduling behind two streams

Fri, 18 Sep 2026 00:00:00 +0900

Across five chapters we have watched what the scheduler does. It receives messages, computes budgets, walks a tree, allocates pages, fills a table. All of it is CPU work.

What is the GPU doing meanwhile? If the answer is nothing, then however fast the model runs, the gaps between runs are empty. This chapter reads the loop that fills those gaps.

Put the two loops side by side

There are two loops in the same file. The simple one first.

7. Where the ledger becomes kernel arguments, and CUDA graphs

Fri, 18 Sep 2026 00:00:00 +0900

At the end of chapter 6 we listed the seven lines of _prepare_batch and left two of them alone: pad_batch and prepare_metadata. This chapter opens both.

Everything built so far has been Python objects: three lengths on a Req, a slot number, a row of the page table. Kernels know none of that. What a kernel receives is a handful of tensors. This chapter is about where and how that conversion happens, and how a CUDA graph pins those tensors down.

8. How several ranks share one ledger

Fri, 18 Sep 2026 00:00:00 +0900

Chapter 1 showed that one scheduler starts per TP rank, and the six chapters since read what happens inside one of them. Now all of it gets replicated once per rank.

That creates a problem. The scheduler makes decisions every step: which requests enter the batch, which pages to take, what to discard and when. If ranks decide independently and arrive at different answers, GPUs holding slices of the same model compute different things. How do they reach the same answer?