Mini-SGLang: reading the scheduler of an LLM serving framework

Mon, 01 Jan 0001 00:00:00 +0000

Eight parts reading sgl-project/mini-sglang , pinned to commit 9a91cfa. Every quoted snippet links to a permalink at that commit.

It starts from the process boundaries one HTTP request crosses, then works through the Req/Batch state ledger, the chunked-prefill budget, page allocation, the radix cache and eviction that make prefix reuse possible, hiding CPU scheduling behind two streams, and how several TP ranks reach the same decision.

tiny-vLLM: reading an LLM inference engine written in CUDA

Mon, 01 Jan 0001 00:00:00 +0000

jmaczan/tiny-vllm is a 1,500-line inference engine written directly in CUDA. Seven parts read it end to end, pinned to a single commit.

It begins with loading weights and laying out GPU buffers, then the prefill path that streams a whole prompt at once, why matrices are handed to cuBLAS transposed, the decode path that produces one token and the kernel written for it, splitting the KV cache into 16-token blocks read through a block table, and continuous batching over slots and queues.

Series on Jaehun's Blog

Mini-SGLang: reading the scheduler of an LLM serving framework

tiny-vLLM: reading an LLM inference engine written in CUDA