7. Where the ledger becomes kernel arguments, and CUDA graphs
At the end of chapter 6 we listed the seven lines of _prepare_batch and left two of them alone: pad_batch and prepare_metadata. This chapter …
8 min
Posts
All posts on technology, daily life, and thoughts.
74 pages
2 /
13
8. How several ranks share one ledger
Chapter 1 showed that one scheduler starts per TP rank, and the six chapters since read what happens inside one of them. Now all of it gets …
8 min
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Paper Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0One-line summary (TL;DR)Even at 4-bit, a 35B-class …
18 min
Mixture of Experts
Quantization
Efficient Inference
LLM Inference
Hardware Architecture
Open-Source LLM
1. An Inference Engine in 1,500 Lines of CUDA: The Map
jmaczan/tiny-vllm is an inference engine for Llama 3.2 1B Instruct, written from scratch in C++ and CUDA. No PyTorch, no Hugging Face, not …
10 min
2. Weight Loading and GPU Buffer Design
In this engine, right after creating the cuBLAS handle, main() immediately does two things (src/main.cpp:557-569 ): it takes the weights out …
14 min
3. The prefill path: flowing the entire prompt at once
In Part 1 I said prefill() is a function that processes the entire prompt in one pass. All 17 tokens pass through everything from embedding …
24 min