1. An Inference Engine in 1,500 Lines of CUDA: The Map
jmaczan/tiny-vllm is an inference engine for Llama 3.2 1B Instruct, written from scratch in C++ and CUDA. No PyTorch, no Hugging Face, not even a tokenizer. It opens the…
tiny-vLLM: reading an LLM inference engine written in CUDA
jmaczan/tiny-vllm is a 1,500-line inference engine written directly in CUDA. Seven parts read it end to end, pinned to a single commit.
It begins with loading weights and laying out GPU buffers, then the prefill path that streams a whole prompt at once, why matrices are handed to cuBLAS transposed, the decode path that produces one token and the kernel written for it, splitting the KV cache into 16-token blocks read through a block table, and continuous batching over slots and queues.
7 pages
2. Weight Loading and GPU Buffer Design
In this engine, right after creating the cuBLAS handle, main() immediately does two things (src/main.cpp:557-569): it takes the weights out of model.safetensors and puts…
3. The prefill path: flowing the entire prompt at once
In Part 1 I said prefill() is a function that processes the entire prompt in one pass. All 17 tokens pass through everything from embedding to next-token selection within…
4. The cuBLAS Transposition Trick — Why the Matrices Are Passed Reversed
In part 3, when looking at the Q/K/V projections, I wrote this sentence and moved on: "Activations and weights are row-major, but cuBLAS assumes column-major, so the call…
5. The decode path that produces one token — why separate kernels were made
Once prefill has streamed the whole prompt through in one pass, everything after that has to produce tokens one at a time based on the tokens produced so far. That step…
6. Splitting the KV cache into 16-token blocks — reading non-contiguous blocks via the block table
Every time decode produces one token, attention re-reads all the K/V that the sequence has accumulated so far. If that K/V is held in one large contiguous buffer per…
7. Overlapping multiple requests with slots and a queue — implementing continuous batching
If the GPU processes only one sequence at a time, it sits idle while that sequence finishes prefill and moves on to decode. Continuous batching is the approach of feeding…