SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering
SteerDistill: Reviving the Judgment of Quantized Agents through Teacher Steering Agents call tools multiple times to finish a task, so inference cost is deployment cost.…
Notes on LLM inference, research papers, and code from a deep learning compiler engineer.
SteerDistill: Reviving the Judgment of Quantized Agents through Teacher Steering Agents call tools multiple times to finish a task, so inference cost is deployment cost.…
Introduction: every new view reads the entire cache Among models that generate video from multiple photographs at a new angle, the highest-quality design stores scene…
Introduction This paper proposes MACaQ, an MXFP4 W4A4 quantization method that reduces both weights and activations to 4 bits. The key is that each linear layer is…
Korean renderings of Abseil's Tip of the Week — string_view, unique_ptr, temporaries and lifetimes, and the other C++ questions that keep coming back.
Reading an open-source inference engine end to end, pinned to one commit. Each part follows the execution path through a specific region of the code and the reasoning behind its design.
Paper reviews on making LLM inference faster and cheaper: speculative decoding, KV cache compression, prompt compression, sparsity, and heterogeneous pipelines.
Paper reviews on shrinking, evicting and reusing the KV cache: compression, quantization, eviction signals, offloading and prefix reuse.
Paper reviews on coping with long context: sparse attention, context compression and recompute, state-space models, and million-token serving.
Paper reviews on LLM serving and training across GPUs and nodes: prefill/decode disaggregation, pipeline parallelism, overlapping communication with compute, scheduling and energy.