DLite: History-free Speculative Decoding for Agentic AI

As a coding agent reads files and calls tools, its conversation and execution history grows. Adding a small model to draft answers faster can create another burden if that model must separately process and retain the same long history.

DLite lets the target model handle the history, while the small drafter uses the target’s internal representations to propose upcoming tokens. It removes the drafter’s separate historical KV cache. It does not remove the target’s cache or discard historical information.

Paper · Code

Why can adding a small model help?

A language model usually generates one token at a time. A token is a unit of text processing, such as a word or part of a word. Instead of running the large model once for every token, a small model can propose several candidates for the large model to check together. This is the basic idea of speculative decoding (§2.1).

The two roles are distinct. The drafter proposes candidates quickly; the target model verifies them and determines the actual output. The approach benefits from accepting many candidates while keeping drafting costs small. A slow or frequently rejected drafter reduces that benefit.

Agents accumulate tool results and prior conversation. Conventional drafters process this history themselves and store internal data in a KV cache for later generation. A KV cache retains information about processed tokens so the model need not recompute everything from scratch. When the target and drafter maintain separate caches, newly confirmed tokens also require updates on the draft side (§3.1).

The left of Figure 1 illustrates agent execution with repeated long inputs and short outputs. The right shows fewer tokens being produced per verification round on such workloads. The question is not merely whether a small model is available, but whether it can keep proposing useful candidates while handling a growing history.

Overall pipeline: what stays and what disappears?

After processing the history, the target supplies hidden states to DLite. These are intermediate neural-network values, not a human-readable summary. The drafter uses them together with the latest confirmed token to propose new candidates (§4.1).

The following is an illustrative example of candidate verification. It uses visible English words for clarity, rather than an actual tokenizer’s segmentation or measured results from the paper.

In the first panel, the full history and target cache remain. The drafter receives target representations and information about the current position. Thus, “history-free” means that the drafter keeps no separate historical KV cache, not that it forgets the history.

In the second panel, the drafter proposes jumps over a dog. The third assumes a verification outcome that accepts jumps and over, then rejects a. In that case, dog is not confirmed either, and generation continues with the from the verification procedure. This does not describe a simple comparison against a reference string; it illustrates which candidates survive an acceptance or rejection decision.

One round performs the following operations.

The drafter does not store the long history again, but target computation and verification remain. These must be distinguished from the work DLite removes (§4.1, Figure 3).

Prepare positions together, then use earlier selections

Computing several upcoming positions together is efficient. But choosing their words independently can make the sequence less coherent. When selecting the word after jumps, for example, the model needs to account for having actually selected jumps.

DLite separates these operations. Its parallel backbone computes representations for several positions together. A small sequential head then uses earlier draft selections when choosing subsequent candidates. The larger computation runs in parallel; only the small computation that accounts for earlier candidates runs sequentially (§4.1.1–4.1.2).

The sequential head handles information inside the current candidate block. It does not reread the entire conversation or file contents. It also incorporates sequential information in the hidden representations before projecting them into vocabulary scores, rather than projecting first and refining afterward. This avoids repeating an expensive intermediate vocabulary projection.

Figure 3 connects the explanation to the model’s components. Follow the path from target representations to the drafter, and the placement of the sequential head after parallel computation. The illustrative figure shows how candidates change; this source figure shows how implementation components connect.

The architecture also needs training preparation. The authors first use an intermediate teacher with a similar structure that can access historical cache, then train DLite to align directly with the target. Removing draft-side cache during inference comes with an additional training procedure (§4.2).

Results: acceptance length and speed are different

The experiments use Qwen3-4B and Qwen3-8B as targets on NVIDIA H800 serving hardware, with a speculative block size of 7. Two metrics need separating. Acceptance length measures output per verification round; speedup measures improvement over ordinary token-by-token generation (§5.1).

The following results are from SWE-Bench. DSpark is another drafting approach used for comparison (Table 3).

For 4B, the change from 1.89 to 3.15 is about a 66.7% increase in acceptance length. The 2.68× speedup, however, is relative to ordinary generation, not DSpark. A direct speed comparison divides 2.68 by 1.64, giving approximately 1.63×. Accepting more candidates and speeding up a service are related, but they are different measurements.

On shorter general-purpose tasks, differences are smaller. For example, 4B GSM8K acceptance length is 6.27 for DSpark and 6.38 for DLite (Table 1). Rather than expecting large gains everywhere, examine how useful this design is when histories grow long.

Speedup also changes with concurrency. For 4B Alpaca, DLite’s throughput relative to ordinary generation is 2.58× with 1 concurrent request and 1.32× with 64. Evaluation should match the service’s actual concurrency and input lengths (Table 4).

Limitations to check before adoption

The serving engine must expose target hidden states and pass them to the drafter. Assessing compatibility therefore requires examining that implementation path, beyond the model’s name. The target KV cache remains necessary, so the memory cost of the whole long input does not disappear.

Training costs remain too. The intermediate teacher and two-stage training are part of the authors’ procedure. The results do not imply that attaching an arbitrary pretrained small model will reproduce these gains (§4.2).

From a reviewer’s perspective, applicability remains a question. The targets evaluated are 4B and 8B Qwen3 models on H800 serving hardware. Larger targets, different architectures, or placing the drafter and target on separate devices require further validation. Acceptance length and speed measurements also do not directly measure task success rates.

DLite is worth examining for agent services that repeatedly reuse long histories. The central question is who processes the history and who reuses the result, rather than whether all cache usage reaches zero. The design retains target-side history processing while reducing redundant historical storage and updates in the drafter.