4. The cuBLAS Transposition Trick — Why the Matrices Are Passed Reversed
In part 3, when looking at the Q/K/V projections, I wrote this sentence and moved on: “Activations and weights are row-major, but …
14 min
Posts
All posts on technology, daily life, and thoughts.
74 pages
3 /
13
5. The decode path that produces one token — why separate kernels were made
Once prefill has streamed the whole prompt through in one pass, everything after that has to produce tokens one at a time based on the …
16 min
6. Splitting the KV cache into 16-token blocks — reading non-contiguous blocks via the block table
Every time decode produces one token, attention re-reads all the K/V that the sequence has accumulated so far. If that K/V is held in one …
18 min
7. Overlapping multiple requests with slots and a queue — implementing continuous batching
If the GPU processes only one sequence at a time, it sits idle while that sequence finishes prefill and moves on to decode. Continuous …
15 min
StepAudio 3 Gen Technical Report
Paper StepAudio 3 Gen: Unifying Speech, Song, Music, and Sound Effects with a Single Discrete Autoregressive (RVQ) ModelTL;DRStepAudio 3 …
5 min
Multimodal Learning
Large Language Models
Transformer
Audio Generation
Text-to-Speech
Reinforcement Learning
How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus
Paper Is “Lossless” Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing OrthrusTL;DR — …
13 min
Speculative Decoding
Numerical Precision
Quantization
LLM Inference
Efficient Inference
Benchmark Evaluation