Speculative Decoding on Jaehun's Blog

Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training

Thu, 10 Sep 2026 00:00:00 +0900

Paper

Two Designs That Make Speculative Decoding Practical for Large-Scale, Long-Context RL Post-Training

TL;DR : Rollout generation dominates the wall-clock time of RL post-training. This work accelerates it with speculative decoding, and adds online draft co-training so the draft does not lag behind as the policy evolves. However, existing systems fail to handle the branch attention used by advanced drafts (unsupported by context parallelism) and target features scattered across stages (a pipeline parallelism problem). This paper proposes (1) a CP technique that decomposes branch attention into a causal main-sequence component and a rank-local branch and merges them, and (2) TapChannel, which delivers target features outside the pipeline schedule, achieving 1.16–1.88× end-to-end speedup from 8B to 122B (source: §3.3, Tab. 1).

Unlocking Lossless Speedups in LLMs via Discrete Diffusion

Tue, 08 Sep 2026 00:00:00 +0900

Paper

Uno: AR and Diffusion in One Model — Lossless Parallel Generation for LLM Acceleration

TL;DR — LLMs are slow because next-token prediction (NTP) can emit only one token at a time by its very sequential structure. Uno plants AR weights that own quality and lightweight LoRA diffusion weights that own speed into a single architecture; the diffusion path drafts token blocks in parallel, and the AR path verifies them via rejection sampling. With neither a separate draft model nor a lossy AR→diffusion conversion, it achieves lossless acceleration that exactly preserves the AR distribution. In practice it delivers up to 3× speedup over the base AR model and still up to 2× at the largest batch sizes, while beating open d-LLMs (26B DiffusionGemma) and proprietary models (Mercury 2) on agentic, coding, and long-form reasoning benchmarks (source: Abstract).