Speculative Sampling in Interactive Video Generation
Speculative sampling reuses future video predictions to reduce response latency. How it works, recomputation costs, and the limits of the serving experiments.
Distributed Computing
What happens once you go past a single GPU. The recurring axes:
- Splitting the phases — disaggregating prefill and decode to cut interference
- Pipelines — graph pipeline parallelism, max-flow placement over heterogeneous GPUs
- Hiding communication — overlapping transfer and compute through kernel fusion
- Scheduling and cost — throughput-latency tradeoffs, preemptible instances, energy efficiency
6 pages
FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
An analysis of FlashMask-3’s Hopper kernels, mask localization, load balancing, communication overlap, and long-context training results and limitations.
Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills
Who pays for the KV cache? — Kubernetes, gateway, and provider bills in one ledger TL;DR — The cost of a single AI feature is scattered across three disconnected ledgers:…
Analytical Power-Aware Provisioning for Prefill-Decode Disaggregated AI Inference
Analytical PD-inference provisioning uses instance counts and KV-cache reservations to find the lowest-power deployment for a target capacity.
Miles v0.1: Production-Level Post-Training
Miles v0.1: A "verified, clean, and scalable" full-stack system for frontier RL post-training TL;DR: Miles is a full-stack RL post-training framework that resolves the…
Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training
Two Designs That Make Speculative Decoding Practical for Large-Scale, Long-Context RL Post-Training TL;DR : Rollout generation dominates the wall-clock time of RL…