How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

Paper

Is “Lossless” Speculative Decoding Really Lossless? — The Numerical-Precision Trap Exposed by Reproducing Orthrus

TL;DR — Orthrus attaches a diffusion view to a frozen AR backbone to generate multiple tokens in parallel, and claims “lossless” speculative decoding. The authors independently reproduced it and directly compared token trajectories against the original AR model across 1,190 prompts (12 domains). Under BF16 inference, the exact match rate was only 0.45 (authors’ checkpoint) / 0.43 (their own trained model), but switching to FP32 raised it to 1.00 (100%). In other words, “lossless” depends on numerical precision, and matching benchmark scores does not guarantee matching token trajectories.

Key Idea

The central claim of this paper can be summarized in one sentence.

The notion of “lossless” acceleration requires a more precise operational definition in neural network inference systems implemented on finite-precision arithmetic and a stateful KV cache.

That is, the authors go beyond simply reproducing Orthrus’s proposition that it “produces the same output as the original AR model,” and instead verify it by introducing numerical precision as an experimental variable (source: §1). As a result, they find that trajectories diverge under BF16 and match exactly under FP32 (source: §3, §5).

The paper presents three contributions (source: §1):

1. Independently implement Orthrus’s training and inference, and evaluate it side by side with the public checkpoint
2. A model trained on teacher-generated (on-policy) distillation data achieves equal or better TPF in most domains
3. Show that exact sequence-level equivalence is extremely sensitive to numerical precision — substantial trajectory divergence under BF16, exact match under FP32

Background: The Problem They Solved

Decoding in AR language models is inherently sequential. Given a prefix, the next-token distribution must be computed before the next token can be appended, so generating N tokens generally requires N decoding steps with repeated access to a growing KV cache (source: §1). This sequential dependency lowers hardware utilization and steeply increases inference cost as the model and context grow.

Diffusion language models and parallel decoding methods bypass this bottleneck by predicting multiple future tokens at once. However, relaxing the strict AR dependency can introduce a discrepancy with the original model’s distribution. Here Orthrus offers an attractive alternative. Instead of replacing or heavily modifying a pretrained AR model, it adds a lightweight diffusion view to a frozen AR backbone. At inference time, the AR component builds the context representation, the diffusion component predicts multiple future tokens in parallel, and an intra-model consensus mechanism verifies the tokens proposed by the AR view (source: §1). Orthrus claims this acceleration is strictly lossless — that is, it exactly preserves the original AR model’s prediction behavior (source: §1).

This lossless claim is precisely what the paper probes.

New Approach: A ‘Lossless’ Audit with a Precision Axis

The “method” of this paper is not a new model proposal but an experimental design that independently reproduces and audits an existing claim. It breaks down into three main axes.

1) Independent reproduction + on-policy distillation

The authors independently implement the Orthrus architecture and training procedure, and build a framework in which the training objective, data distribution, and hyperparameters can be systematically controlled (source: §1). The key insight is that because the diffusion view is trained with teacher forcing, training on ordinary human-written continuous text fails to reproduce the states the frozen AR model encounters during its own generation (source: §1).

So they built a teacher-generated distillation corpus by prompting the frozen AR model (Qwen/Qwen3-1.7B) and recording continuous text via greedy decoding (source: §2.1). The resulting data follows the model’s own prediction trajectory. Keeping only prompts of length 50–1,000 characters, they produced a total of 4,113,358 prompt-response pairs (source: §2.1).

2) Training configuration

Training was done on 8 NVIDIA H100 GPUs with CUDA 13.3.73, PyTorch 2.13.0, and Transformers 5.8.0. One epoch, initial learning rate $2\times10^{-4}$, batch size 10, cross-entropy loss, block size 8, number of blocks 32, and maximum sequence length 3,072 tokens (source: §2.2). This configuration differs considerably from the original Orthrus experiments (source: §2.2).

3) Trajectory evaluation

They used 12 text domains × 100 prompts per domain (except gec-en, with 90), for a total of 1,190 prompts (source: §2.3, §3). Inference ran on an RTX 3090 (24 GB) GPU with BF16 + eager attention, max_new_tokens=128, do_sample=False, temperature=0.0 — that is, greedy decoding only (source: §3). There are two key metrics.

• TPF (Tokens Per Forward) — measures parallel generation efficiency (source: §2.3)
• Sequence Match Rate — the exact match rate between the Orthrus trajectory and the AR reference trajectory (source: §3)

How It Works: The Moment Trajectories “Slip”

The foundation of the lossless claim is that “under an idealized arithmetic model, the consensus mechanism guarantees exact trajectory equivalence.” However, when implemented with finite-precision arithmetic, it may fail to reproduce the reference computation bit-for-bit (source: §2). Since generation is discrete, even a tiny numerical difference can eventually change which token is selected and split the trajectory (source: §2).

Consider a small example. Suppose the AR model assigns logits $[10.0,\ 9.999,\ 3.0]$ to three candidate tokens $t_1, t_2, t_3$. With greedy decoding, it always picks $t_1$, the maximum logit. But if Orthrus’s parallel path computes these logits internally with a different operation order or precision and produces a slightly flipped value like $[9.9998,\ 10.0001,\ 3.0]$, the consensus step approves $t_2$. From that moment on, the two systems output different tokens, and the subsequent context differs as well.

This “flip” is directly tied to precision. BF16 has an 8-bit mantissa, giving roughly $\log_{10}(2^8)\approx2.4$ decimal digits of precision, while FP32 has a 23-bit mantissa (plus the implicit 1 bit), giving roughly 7 digits. Thus, near magnitude 10, a difference of $0.001$ (about 5 digits) is distinguishable in FP32 but not in BF16. When the logit difference falls below the resolution of the precision, the consensus mechanism approaches a coin flip.

The paper backs up with statistics the fact that this divergence does not spread uniformly at random. It defines the reference model’s response-conditional perplexity as follows (source: §3),

and shows that the larger this value is (the less “confidently” the reference model generates that response), the higher the probability that the trajectory diverges (source: §3, Tab. 5, Tab. 6). In other words, one can interpret this as: the more ambiguous the token the reference model picks, the more easily a flip due to precision differences occurs.

Performance Validation: Main Results

1) Under BF16, “lossless” does not hold

Comparing 1,190 trajectories, the exact match rate under BF16 inference is 0.45 ± 0.03 for the authors’ checkpoint and 0.43 ± 0.03 for their own trained model. That is, more than half of the trajectories (0.55 and 0.57 respectively) diverged at least once (source: §3, Tab. 3).

Domain-level variation is large (source: §3, Tab. 4). Domains with high match rates are gec-en 0.88 (identical for both), qa-ru 0.73/0.70, and wmt ru-en 0.64/0.63. On the other hand, creative-en 0.12/0.11, poetry-en 0.11/0.08, and poetry-ru 0.14/0.11 — creative and poetry-writing domains — drop into the 10% range. This reads as: creative generation is inherently a high-dimensional distribution in which “the model is less certain,” so precision-driven flips occur far more easily.

2) Divergence is strongly associated with response-conditional perplexity

The mean PPL of matching trajectories is 1.11 ± 0.01, and that of diverging trajectories is 1.28 ± 0.01, with both models showing the same pattern (source: §3, Tab. 5). The association holds in a logistic regression controlling for response length and domain (source: §3, Tab. 6):

Here $Y=1$ denotes exact trajectory match and $\gamma_D$ is the domain effect. $\beta_1$ is $-8.10$ (95% CI $[-10.50,-5.71]$, $p=3\times10^{-11}$) for the authors’ checkpoint and $-10.92$ (95% CI $[-13.60,-8.24]$, $p=1.3\times10^{-15}$) for their own model — large in the negative direction, with confidence intervals far from zero. That is, responses with higher perplexity are significantly less likely to be reproduced exactly (source: §3).

3) Divergence does not necessarily mean performance degradation

The results of evaluating GSM8K, HumanEval, and IFEval with lm-eval-harness are as follows (source: §4, Tab. 7).

Their own Orthrus model recorded higher point estimates than the AR reference (Qwen3-1.7B) on all three benchmarks. However, the authors explicitly state that this difference should not be interpreted as a statistically significant improvement (source: §4). Rather, the important lesson is that benchmark scores can be the same or higher while token sequences differ. The two systems can obtain statistically indistinguishable task scores while producing different tokens, and sometimes a small numerical perturbation pushes generation toward an answer the benchmark prefers (source: §4).

There is also a notable counterexample. The authors’ checkpoint scores 0.3659 on HumanEval pass@1, lower than the AR reference (0.4024) (source: Tab. 7). In other words, trajectory divergence is not always neutral or favorable.

4) Under FP32, 100% match

Re-evaluating the same 1,190 trajectories in FP32, both Orthrus variants recorded a sequence match rate of 1.00 ± 0.00 and a divergence rate of 0.00 ± 0.00 (source: §5, Tab. 8). The trajectory divergence observed under BF16 disappeared completely. This strongly suggests that the observed divergence stems from finite-precision numerical effects, not from a defect in a specific layer or operation (source: §5).

5) On-policy distillation raises TPF

Their own trained model recorded higher TPF than the authors’ checkpoint in 10 of the 12 domains (source: §2.3, Tab. 2). For example, code 3.24 → 3.44, code-ru 3.59 → 3.93, and wmt ru-en 2.04 → 2.46. However, on math the authors’ checkpoint (8.08 ± 0.65) substantially leads their own model (4.99 ± 0.16), and on gec-en the authors also lead, 5.00 versus 4.23 (source: Tab. 2). That is, this also reveals that TPF varies greatly by domain depending on the distribution of the teacher-generated data.

Our Perspective: Strengths, Limitations, and Why This Research Matters

Strengths

• A rare audit paper that operationally verifies “lossless.” While most acceleration research focuses only on throughput and benchmarks, this paper separately measures exact equivalence at the trajectory level (source: §1, §3). It demonstrates experimentally that the word “lossless” actually depends on precision.
• Solid controls. By evaluating two variants — the authors’ checkpoint and an independently trained model — it shows the observed effect is not a byproduct of a particular training setup (source: §2).
• Statistical rigor. A logistic regression controlling for domain and response length robustly supports the perplexity-divergence association (source: §3, Tab. 6).

Limitations

• Small scale. The results are for a single Qwen3-1.7B backbone. Whether the same conclusion holds for larger models, MoE, and sampling-based decoding remains open (source: §2, §3).
• Greedy decoding only. With do_sample=False and temperature=0.0 fixed, the divergence behavior may differ under the stochastic decoding common in real serving (source: §3).
• The cause of divergence was not localized. The authors themselves state that they “did not identify a specific layer or operation step” (source: §5). Exactly where the logits flip in BF16 remains unresolved.
• Domain dependence of on-policy distillation. In some domains such as math (8.08 vs 4.99), their own model’s TPF is instead substantially lower (source: §2.3, Tab. 2).

Why It Matters

The significance of this paper is not that “Orthrus was wrong.” The authors do not claim that the consensus mechanism is ineffective or that Orthrus’s acceleration value is absent (source: §1). Rather, it lies in redefining the boundaries of the term “lossless.” As long as neural network inference runs on finite-precision arithmetic plus a stateful KV cache, “an algorithm that in principle preserves the intended AR computation” and “an implementation that reproduces even the discrete outputs identically” are strictly different propositions (source: §1). And matching benchmark scores does not mean matching trajectories (source: §4). This distinction offers an evaluation-standard that every future paper claiming “lossless acceleration” should follow.

What’s Next?: The Road Ahead

The future direction the authors specify is one, but its implications are large (source: §5):

1. Localizing the cause of divergence. Identifying the specific operation steps where logits flip under BF16 (attention scaling, layer norm, softmax accumulation, consensus threshold, etc.) remains the next task.

To this we can add two branches from our perspective.

2. Scaling to sampling and large models. Beyond greedy 1.7B, there is a need to verify how the precision-divergence relationship changes in real serving settings (stochastic decoding, tens-of-B-scale models, MoE). In particular, since substantial divergence already appeared under BF16, FP8/INT8 quantized serving is expected to show even more dramatic trajectory deviation.
3. Establishing a “lossless” standard. As the authors propose, lossless acceleration evaluation must explicitly state (a) the operational criterion for equivalence and (b) the measured numerical precision (source: §6). This is not a mere recommendation but a practical norm that subsequent papers should adopt as a default.

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. Datasets and task domains used for trajectory evaluation.

Table 2. Tokens Per Forward across different evaluation domains. Higher values indicate more effective parallel token generation. Values are means with 95% confidence intervals.

Table 3. Orthrus–Qwen3 trajectory matching statistics under BF16 inference. Values are proportions with 95% confidence intervals.

Table 4. Orthrus–Qwen3 trajectory matching rates by domain under BF16 inference. Values are proportions of exactly matching trajectories with 95% confidence intervals.

Table 5. Mean response-conditional PPL for matching and diverging Orthrus trajectories. Values are means with 95% confidence intervals.

Table 6. Logistic regression results for the association between response-conditional perplexity and exact trajectory matching.

Table 7. Comparison of Qwen3-1.7B, the authors’ Orthrus-Qwen3-1.7B, and our Orthrus-1.7B-final on lm-eval-harness benchmarks.

Table 8. Orthrus–Qwen3 trajectory matching statistics under FP32 inference.