5. The radix cache that reuses prefixes, and eviction
Chapter 4 showed how one request secures its space. But if a hundred requests arrive with the same system prompt and each repeats the same computation, that space runs…
Posts
All posts on technology, daily life, and thoughts.
119 pages
9 /
20
6. Hiding CPU scheduling behind two streams
Across five chapters we have watched what the scheduler does. It receives messages, computes budgets, walks a tree, allocates pages, fills a table. All of it is CPU work.…
7. Where the ledger becomes kernel arguments, and CUDA graphs
At the end of chapter 6 we listed the seven lines of preparebatch and left two of them alone: padbatch and preparemetadata. This chapter opens both. Everything built so…
8. How several ranks share one ledger
Chapter 1 showed that one scheduler starts per TP rank, and the six chapters since read what happens inside one of them. Now all of it gets replicated once per rank. That…
Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches
Fathom: Accelerating 1M-Token Decoding by 1.67x with Query-Adaptive Depth Scanning TL;DR When agent sessions stretch to 1M tokens and dozens of sessions reside…
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Waking the MoE on SSD: Serving a 35B MoE at 20 tok/s on a 24 GB Desktop with Edge0 One-line summary (TL;DR) Even at 4-bit, a 35B-class MoE occupies 19.5 GB and cannot…