Large Flow Language Models

Paper

The sequential bottleneck of sampling one token at a time

Large language models can compute an entire sentence in parallel during training, but when generating text they must know the preceding token before they can choose the next one. No matter how fast the GPU is, this order cannot be skipped in one pass, so generation latency grows. This post covers Lafa (Large flow language model), which attacks this bottleneck with block-level flow.

Lafa revives an already-trained autoregressive model. A single computation produces candidate tokens for an entire block, and autoregressive prediction verifies and commits them from the front. As a result, it achieves up to 2.54× the throughput of the existing autoregressive baseline at the same quality level. The exact conditions and numbers are revisited below.

Why flow, and why split into blocks

The existing parallel generation method, discrete diffusion, erases tokens with [MASK] and then restores them. Sampling multiple positions independently in one step severs the relationships between positions, which can degrade quality.

Flow differs: it does not erase tokens. It mixes Gaussian noise into token embedding vectors, then integrates the continuous path from noise back to clean embeddings to produce tokens. Because it moves in continuous space, the acceleration and control techniques developed for image and video generation carry over directly. However, existing flow language models were demonstrated only on small-scale, simple tasks; one scaling attempt reported training inefficiency; and how to scale to billions of parameters is not yet established (§1, §2).

Lafa splits flow into blocks. Instead of generating the whole sentence at once, it conditions on the previously committed text (the prefix) and generates token blocks of size $B$ one at a time, chaining blocks causally. The sentence probability is factorized as a product over blocks.

Splitting into blocks yields two benefits. First, the prefix key-value (KV) cache—the attention intermediates from previous token computations—can be reused for each block. Second, parallel generation can be confined to within a block (§2.1). Handling tokens on embeddings reduces the dimensionality of a single token from the vocabulary size $|V|$ to the embedding dimension $d$, which also makes training and scaling easier.

Asynchronous flow within a block

If every position in a block escaped noise at the same rate, the left-to-right directionality of text being read would disappear. Lafa gives each position $i$ its own progress clock $\tau^i(t)$. The clocks of earlier positions run faster, so earlier tokens are revealed first. The path from noise $x_0$ to the clean embedding $x_1$ is as follows.

$\tau$ is not fixed. It is learned from data so that training passes slowly through regions with high loss. This concentrates training samples and inference steps around difficult noise levels. The learned schedule generally advances earlier positions first (in Fig. learned_time, the horizontal axis is reference time $t\in[0,1]$, the vertical axis is the per-position $\tau^i(t)$, and the curves correspond to block positions 1, 4, 7, …, 16); this left→right bias is not manually imposed but emerges from training (§2.1).

Illustrative example. Suppose the three embeddings of the prefix “the weather today is” are already committed, and with block size $B=3$ we must produce “sunny/cloudy/.”

The KV of the committed tokens is appended and we move to the next block. This example is for intuition only and is unrelated to the measurements below.

Designing flow to be learned from an autoregressive model

The key is not to train a flow model from scratch but to retrofit an already-trained autoregressive model into a block denoiser. The denoiser is a function that, given a noisy block, predicts the clean-token probability at each position. The problem is that this retrofit must not damage autoregressive prediction or the prefix cache. The paper solves this with three designs (§2.2).

First, tokenwise time conditioning. The prefix receives time 1 (no noise), and each position in the block receives its own time $\tau^i(t)$. This value is injected via adaptive layer normalization (adaLN)—a mechanism that re-normalizes a layer’s output based on its input. Initializing adaLN as the identity function means time conditioning initially leaves the existing computation untouched and is learned gradually.

Second, block bidirectional attention. Tokens within a block can see each other, so less-refined later tokens can help earlier ones. The prefix is still viewed only causally, as before. As a result, the prefix’s keys and values do not depend on block state and can be reused across denoising steps.

Third, fixed embedding norm. If embeddings are updated freely, their L2 norm grows relative to the noise scale, creating a shortcut that makes denoising easier. The pretrained norm is fixed and only the direction is learned.

The training loss sums three terms: autoregressive cross-entropy on the prefix, denoising cross-entropy on block tokens, and squared error between predicted and actual embeddings (Equation 6). Keeping the autoregressive loss also improves flow accuracy. Thanks to this joint training, autoregressive ability is preserved—or even improved—while learning flow. This is consistent with the existing observation that multi-token prediction improves next-token prediction (§2.2, §3.2).

The figure above (original Figure training) shows the state transitions of parallel training. The upper clean sequence produces autoregressive predictions with causal attention, while the lower copy is split into blocks of size 3, where each block sees only the preceding clean tokens and produces denoising predictions bidirectionally within the block. A single forward pass yields both the autoregressive loss and the denoising losses of all blocks, without recomputing overlapping prefixes.

Design effects confirmed by ablation

At 0.5B, block 4, and 32 Euler steps, each design was reverted one at a time and average accuracy was compared (Table design-ablations).

The original design wins on all four axes: attention, time, embedding, and loss. In particular, adding the autoregressive loss actually raises flow accuracy, so AR preservation and flow generation do not hurt each other.

Reducing the number of steps with flow map distillation

Flow decoding is expensive because it must call the denoiser many times per block. To reduce this, Lafa uses flow map distillation. A student model learns a function $\hat X_{s,t}$ that moves the state directly from time $s$ to $t$, replacing several Euler steps with a single jump. Predictions on the diagonal ($s=t$) are matched to the teacher, while off-diagonal predictions ($s\lt t$) are matched to the composition of two short jumps in a semigroup fashion. The student maintains the prefix cache and can generate in as few as 1 step ($\hat X_{0,1}$) (§2.2).

Draft and verify at once: self-speculative decoding

Reaching AR-level quality with flow decoding requires many steps. Lafa discards the expensive multi-step process and by default draws a block draft with 1-step flow, then verifies it with the autoregressive predictor (§2.3).

The draft comes from a single denoiser evaluation starting from noise. Each position of the draft is compared against the autoregressive prediction from the front, and only the longest matching prefix is committed; the rest is discarded. If even the first token differs, it is replaced with the autoregressive token. The final output is exactly the same as the model’s greedy autoregressive output. Flow is used only to increase speed, and the quality bar equals greedy autoregression.

A naive implementation needs a separate verification forward and a next-draft forward, which costs extra. The paper eliminates this with fused decoding: verification of the previous draft and drafting of the next block are mixed into a single forward pass.

The figure above (original Figure fused_decoding) shows how one stage proceeds at block size 3. The horizontal direction is sequence position, and the vertical blocks are the input fragments fed into each forward.

The clean draft is appended like a prefix to produce verification logits via causal attention, and at the same time candidate noise blocks corresponding to “the case where $j$ preceding draft items are accepted” are appended and processed together. From the verification result, the candidate block with the actually accepted length is chosen as the next draft, eliminating a separate re-drafting forward.

Because this method performs both drafting and verification with a single trained model, it is called self-speculative. No separate small draft model or additional parameters are needed. Alternatively, confidence decoding—accepting only prefixes whose token probability exceeds a threshold instead of using autoregressive verification—is possible, but lowering the threshold trades accuracy for speed, so the paper keeps self-speculative as the default (§3.3, Appendix confidence_results).

Results: maintaining quality while increasing speed

The models use Qwen2.5 0.5B·7B Instruct and Qwen3 8B Base as initialization and are trained for only 5–15 epochs (9–27B tokens) on Dolci-Instruct-SFT data. This is at least 18× fewer than the up to 500B tokens reported by comparable-scale diffusion models (Dream, LLaDA, Efficient-DLM) (§3, Table training-hyperparameters). The benchmarks are six covering instruction following, reasoning, and coding: IFEval, BBH, MMLU-Pro, GPQA, GSM8K, and HumanEval.

Learning flow does not hurt autoregression

Comparing average single-token decoding accuracy across the six benchmarks (Table ntp_main), Lafa beats the AR SFT built with the same initialization, data, and training budget at every scale. At 8B it approaches Efficient-DLM’s 63.5% while using at least 18× fewer training tokens. The process of learning flow actually improved autoregressive prediction (Table 1, §3.2).

Parallelism: how far does accuracy hold as TPF rises

TPF (tokens per forward) is the number of tokens committed per forward pass. For AR it is 1 by definition. Since self-speculative output equals greedy autoregression, TPF comparisons are parallelism comparisons at the same AR quality.

In the figure above (original Figure selfspec_frontier), the horizontal axis is TPF and the vertical axis is average accuracy over the six benchmarks. At 7B, block 4 reaches TPF 2.6 and 60.9% accuracy, beating DeepSeek-MTP on both accuracy and TPF, while block 16 reaches 58.0% at TPF 4.9, close to AR SFT’s 58.4% while nearly doubling parallelism. Across all scales, AR-level accuracy is maintained in the roughly 2.5–5 TPF range (Table selfspec_lafa_sixbench).

Wall-clock time: throughput measured in TPS

TPF shows only parallelism, not actual time. Because the cost of a forward pass differs by method, tokens per second (TPS) was measured on a single H100 GPU at batch 1.

In the figure above (original Figure throughput), the horizontal axis is TPS and the vertical axis is average accuracy over IFEval, GSM8K, and HumanEval. Lafa 7B with block 16 reaches 100.2 TPS, 2.54× the throughput of AR SFT’s 39.5 TPS at similar average accuracy (74.5% vs 73.9%). With block 4 it is about 2× faster at 79.9 TPS while achieving higher average accuracy (76.5%). DeepSeek-MTP (53.6 TPS) and MARS (48–49 TPS) reach at most 1.4× even when their accuracy equals or exceeds AR’s. Diffusion-based methods (Fast-dLLM v2, Dream, Efficient-DLM) can achieve higher TPS depending on configuration, but their average accuracy is below AR SFT, and the faster the configuration, the larger the accuracy loss (Table throughput_full, §3.3).

What “maintaining quality” means

Because self-speculative output equals greedy autoregression, the 2.54× is a speedup obtained while preserving the same AR quality. Flow decoding itself requires multiple steps. At 7B and block 4, the 32-step average is 75.2%, already above AR SFT (73.9%), and at 64 steps it reaches 75.9% (Table flow_full, §3.1). Turning multi-step flow into a 1-step draft plus verification is the source of the practical speedup. Flow map distillation improves accuracy in the 1–2 step regime, and at 7B the 2-step student approaches the 4-step teacher.

Limitations and when to apply it

The limitations the paper itself notes are mainly related to flow map distillation. The distilled 1- and 2-step models improve accuracy while reducing the number of steps, but a quality gap remains relative to flow decoding with more steps. Fully incorporating self-conditioning and prefix guidance into distillation is left as future work (§3.1, §4). Confidence decoding loses accuracy when the threshold is lowered, so it could not be used as the default (Appendix confidence_results).

There are also points a reviewer should note. First, the 2.54× was measured at 7B, block 16, a single H100, and batch 1; there is no guarantee the same factor holds for larger models, larger batches, or long contexts. Second, evaluation is limited to instruction, math, and coding benchmarks, so generalization to other domains is not established in the paper. Third, 8B starts from Base instead of Instruct (the paper reports that additional training of the Instruct counterpart degraded performance). This is why IFEval numbers at the 8B scale are generally low (AR SFT is also 38.1%).

Given these conditions, Lafa is suited to serving environments that want higher throughput while preserving the autoregressive model’s greedy output exactly. If you already have a trained model and can train on a small number of additional tokens, you can obtain the benefits of speculative decoding without a separate draft model. Conversely, if you want to change the output distribution itself, or if you have enough latency headroom to run multi-step flow decoding in real time, the method’s strengths are less prominent.

Figures in this post are taken from the original arXiv:2610.28559 (CC BY 4.0). Only size and format were changed.