1. The Processes One Request Passes Through
Open an LLM serving framework for the first time and you will probably go looking for the scheduler. But no amount of reading scheduler code answers the question that…
code-series
Pick a small inference-engine repository, pin it to a single commit, and read it from front to back. Every quoted snippet links to a permalink at that commit, so the writing and the code never drift apart.
Repositories covered so far:
- jmaczan/tiny-vllm (7 parts) — a 1,500-line CUDA inference engine: weight loading and GPU buffers, the prefill path, the cuBLAS transpose trick, the decode kernel, a 16-token block KV cache with a block table, and continuous batching over slots and queues.
- sgl-project/mini-sglang (8 parts) — starting from the process boundaries one request crosses: the Req/Batch state ledger, the chunked-prefill budget, page allocation, the radix cache and eviction, and overlapping work across two streams.
Each part assumes the previous one, so reading from part 1 works best.
15 pages
2. Req and Batch: the ledger for one request
In chapter 1, what reached the scheduler was a 1-D int32 tensor on the CPU. While that token sequence grows into an answer, the system has to write down somewhere how far…
3. The budget behind a prefill batch, and chunked prefill
In chapter 2 a request became a ledger expressed by three lengths. Now there is one question left: with several requests waiting, what goes into this step, and how much…
4. Paged allocation, and a page table that holds raw locations
In chapter 3 a request "secured its space". What that space actually is, is this chapter's subject. The KV cache is managed in pages. Yet open the pagetable that each…
5. The radix cache that reuses prefixes, and eviction
Chapter 4 showed how one request secures its space. But if a hundred requests arrive with the same system prompt and each repeats the same computation, that space runs…
6. Hiding CPU scheduling behind two streams
Across five chapters we have watched what the scheduler does. It receives messages, computes budgets, walks a tree, allocates pages, fills a table. All of it is CPU work.…
7. Where the ledger becomes kernel arguments, and CUDA graphs
At the end of chapter 6 we listed the seven lines of preparebatch and left two of them alone: padbatch and preparemetadata. This chapter opens both. Everything built so…
8. How several ranks share one ledger
Chapter 1 showed that one scheduler starts per TP rank, and the six chapters since read what happens inside one of them. Now all of it gets replicated once per rank. That…
1. An Inference Engine in 1,500 Lines of CUDA: The Map
jmaczan/tiny-vllm is an inference engine for Llama 3.2 1B Instruct, written from scratch in C++ and CUDA. No PyTorch, no Hugging Face, not even a tokenizer. It opens the…
2. Weight Loading and GPU Buffer Design
In this engine, right after creating the cuBLAS handle, main() immediately does two things (src/main.cpp:557-569): it takes the weights out of model.safetensors and puts…
3. The prefill path: flowing the entire prompt at once
In Part 1 I said prefill() is a function that processes the entire prompt in one pass. All 17 tokens pass through everything from embedding to next-token selection within…
4. The cuBLAS Transposition Trick — Why the Matrices Are Passed Reversed
In part 3, when looking at the Q/K/V projections, I wrote this sentence and moved on: "Activations and weights are row-major, but cuBLAS assumes column-major, so the call…
5. The decode path that produces one token — why separate kernels were made
Once prefill has streamed the whole prompt through in one pass, everything after that has to produce tokens one at a time based on the tokens produced so far. That step…
6. Splitting the KV cache into 16-token blocks — reading non-contiguous blocks via the block table
Every time decode produces one token, attention re-reads all the K/V that the sequence has accumulated so far. If that K/V is held in one large contiguous buffer per…
7. Overlapping multiple requests with slots and a queue — implementing continuous batching
If the GPU processes only one sequence at a time, it sits idle while that sequence finishes prefill and moves on to decode. Continuous batching is the approach of feeding…