How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

Wed, 16 Sep 2026 00:00:00 +0900

Paper

Is “Lossless” Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing Orthrus

TL;DR — Orthrus attaches a diffusion view to a frozen AR backbone to generate multiple tokens in parallel, and claims “lossless” speculative decoding. The authors independently reproduced it and directly compared token trajectories against the original AR model across 1,190 prompts (12 domains). Under BF16 inference, the exact match rate was only 0.45 (authors’ checkpoint) / 0.43 (their own trained model), but switching to FP32 raised it to 1.00 (100%). In other words, “lossless” depends on numerical precision, and matching benchmark scores does not guarantee matching token trajectories.

Numerical Precision on Jaehun's Blog

How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

Is “Lossless” Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing Orthrus