1. An Inference Engine in 1,500 Lines of CUDA: The Map
jmaczan/tiny-vllm is an inference engine for Llama 3.2 1B Instruct, written from scratch in C++ and CUDA. No PyTorch, no Hugging Face, not even a tokenizer. It opens the…
Posts
All posts on technology, daily life, and thoughts.
119 pages
10 /
20
2. Weight Loading and GPU Buffer Design
In this engine, right after creating the cuBLAS handle, main() immediately does two things (src/main.cpp:557-569): it takes the weights out of model.safetensors and puts…
3. The prefill path: flowing the entire prompt at once
In Part 1 I said prefill() is a function that processes the entire prompt in one pass. All 17 tokens pass through everything from embedding to next-token selection within…
4. The cuBLAS Transposition Trick — Why the Matrices Are Passed Reversed
In part 3, when looking at the Q/K/V projections, I wrote this sentence and moved on: "Activations and weights are row-major, but cuBLAS assumes column-major, so the call…
5. The decode path that produces one token — why separate kernels were made
Once prefill has streamed the whole prompt through in one pass, everything after that has to produce tokens one at a time based on the tokens produced so far. That step…
6. Splitting the KV cache into 16-token blocks — reading non-contiguous blocks via the block table
Every time decode produces one token, attention re-reads all the K/V that the sequence has accumulated so far. If that K/V is held in one large contiguous buffer per…