Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs

Paper

The problem: re-reading the image at every step

Diffusion language models that answer questions from images do not complete a response in one pass. While filling in the blanks over many steps, they re-read an image that barely changes from the beginning at every step. LOWN asks “does this step really need the image?” every time, and re-reads only when necessary, cutting decoding cost substantially.

Instead of drawing one token at a time from the left like autoregressive models, diffusion language models start with the entire response masked. At each step they examine all remaining blanks and simultaneously resolve the $k$ masks with the highest confidence. This is called “unmask,” and $k$ is the response length divided by the number of steps (source: §3.1). In image-input diffusion language models (DVLM, e.g., LLaDA-V), image tokens are fixed in the input sequence rather than the response and enter along with it. An “image token” is a numeric fragment that makes the picture processable by the model.

The share of image tokens is overwhelming. Across the 10 benchmarks evaluated, image tokens account for 89–97% of the total input, and the cost of a single forward pass without the image is only about 3–10% of the original (benchmark averages 2.5–10.1%) (source: §1, Figure 1b, Table 8). Because DVLM uses bidirectional attention, it cannot reuse the key-value cache of autoregressive models as is, and passes the entire input through again at every step. As a result, most of the decoding cost goes into “re-reading an image that does not change.”

When the image is actually needed

Before designing LOWN, we numerically checked whether the image is truly needed at every step. Following the standard decoding path as is, we ran two forward passes side by side at each step. The “FULL forward” computes probabilities with an input that includes image tokens, while the “TEXT-ONLY forward” computes probabilities from the same partial response with only the image tokens removed and the position IDs kept unchanged (source: §3.2, Appendix A). Because the progress state is determined solely by the FULL forward, the decoding path itself is identical to the baseline. The TEXT-ONLY output is only recorded and not reflected in the state.

The comparison results come down to three points:

• For 83.4% of the positions to be unmasked, the two forwards predicted the same token as top-1. For 85.0% of the final tokens, the value had already been predicted as top-1 at the same position before unmasking (source: §3.2, Table 6). Cases where the image flips the decision are rare.
• The top-1 disagreement between the two forwards was largest in the early denoising steps for 8 of the 10 benchmarks, and shrank as the response took shape. There are exceptions such as DeCapBench (source: §3.2, Figure 2a).
• The fraction of steps where the image changed at least one unmask decision—the image-dependent step ratio—varies widely by benchmark.

The average across the 10 is 22.8% (source: §3.2, Figure 2b, Table 6). Fixing a schedule of “read the image every n steps” cannot track this variation well. The point at which the image is needed depends on the input and the step, so it has to be judged on the fly at each step. Fortunately, the material for that judgment is already there. The TEXT-ONLY forward provides predictions at the same positions at about 3–10% of the cost of the FULL forward (source: §3.2, Table 8).

LOWN: looking only when the image is needed

LOWN starts from “let us reuse the prediction made the last time we looked at the image.” Whenever it runs a FULL forward, it separately stores the top-1 predictions and their confidence for the positions still blank. These stored values are called “anchors.” An anchor is just a token value and a few confidence numbers, so LOWN itself does not keep large state like attention keys and values.

Each step first computes predictions for the remaining positions with a TEXT-ONLY forward. If these predictions match the anchors, LOWN certifies the anchor predictions as “predictions likely to remain valid without the image and after the state has changed.” Because the two predictions differ in both whether the image was seen and which point in the state they were computed from, a match is used as an approximate signal that both conditions were satisfied together. No absolute confidence threshold is needed (source: §4.1).

Concretely, it lists the blank positions in descending order of anchor confidence and counts from the front the positions where “TEXT-ONLY prediction == anchor” to form a certified prefix. The scan stops at the first mismatch. If the prefix length is at least $k$, it skips the image and unmasks the entire prefix with the anchor values. If it is below $k$, it runs the FULL forward to refresh the anchors, then unmasks $k$ positions—the positions where the two predictions differ first, and the rest in order of FULL confidence (source: §4.2).

  flowchart TD
    A["Current response state (partially masked)<br/>run TEXT-ONLY forward without the image"] --> B["Compare each masked position's top-1 prediction and<br/>confidence against the anchor (previous FULL values)"]
    B --> C{"Number of consecutive matches<br/>in descending confidence ≥ k ?"}
    C -->|"Yes"| D["Skip the image<br/>fix all certified positions to the anchor values"]
    C -->|"No"| E["FULL forward with the image<br/>refresh the anchors with new values"]
    E --> F["Fix k positions: mismatched positions first,<br/>the rest in order of confidence"]
    D --> G{"Any masked positions left?"}
    F --> G
    G -->|"Yes"| A
    G -->|"No"| H["Response complete"]

Below is an illustrative example. The token split and numbers are constructed independently of the paper’s example, and $k=2$ is assumed.

Suppose that at some step the response is in the state “The cat ? ?”. Positions 1 and 2 are already fixed, and positions 3 and 4 are still masked. The anchors saved by the previous FULL forward (predictions and confidence for the masked positions) and the predictions of this step’s TEXT-ONLY forward are as follows.

Scanning in descending confidence (position 3 → 4), position 3 matches and position 4 diverges, so the certified prefix is {position 3} with length 1. Because this is less than $k=2$, the image cannot be skipped, so the FULL forward runs. The FULL forward predicts position 3→“sat” and position 4→“on”, refreshes the anchors, unmasks position 4 (where the two predictions differed) first, and then unmasks position 3. Output: “The cat sat on”.

Conversely, if TEXT-ONLY had matched the anchors at both positions, the prefix length 2 ≥ $k$ would hold, the FULL forward would be skipped, and the two values would be fixed as is. If the certified prefix is longer than $k$, more than $k$ positions are fixed in the same step, reducing the total number of steps as well. Both branches fix at least $k$ positions, so LOWN never uses more steps than the $T$ steps of baseline decoding (source: §4.2).

There is one caveat. Two adjacent positions predicted independently in the same step can receive the same token, and fixing them as is produces a repeated word in the response. To prevent this, the prefix is cut if the token ID equals that of an already fixed neighbor or a neighbor accepted earlier in the same scan (source: §4.1, Appendix B.1).

Cost decomposes as $R = f + \sigma\rho$. $f$ is the fraction of the baseline $T$ steps that run a FULL forward, $\sigma$ is the ratio of the number of steps LOWN runs to the baseline $T$, and $\rho$ is the relative cost of one TEXT-ONLY forward (source: §4.3). Certification lowers $f$, and accepting the whole prefix lowers $\sigma$. $\rho$ is at the 2.5–10.1% level per benchmark depending on the share taken by the image (source: Table 8). Unlike token pruning, which reduces the cost of a single FULL forward, LOWN reduces the number of times the FULL forward runs. The two methods operate on different axes (source: §4.3).

Performance: how much is saved and how much is kept

Comparison at the same compute

For competing methods, we chose the cheapest setting in each method’s own hyperparameter grid that uses FLOPs at or above LOWN’s, so if LOWN scores lower, it does so at equal or higher cost (source: Table 1, §C.3). The table below compares methods at matched compute on LLaDA-V.

The average column is the mean of normalized scores across all 10 benchmarks. Only CapArena is normalized as $100+\text{score}-\text{baseline score}$ (source: §C.1). LOWN retains 99.6% on average at 0.353× decoding FLOPs and ranks first among acceleration methods on 8 of the 10 benchmarks (source: §5.2). On 4 of the 10 benchmarks, LOWN scores higher than the original. In the table above, ScienceQA (78.83 vs 76.50) and ChartQA (78.36 vs 78.16) are such cases. Clipping the cap at 100% still gives 98.6% (source: §5.2). RedVTP drops more sharply, to 93.0%. Pruning erases visual information permanently, whereas LOWN can look at the full image again at the steps where it is needed (source: §5.3, Figure 6).

Combining with pruning and caching

Whereas pruning reduces the number of visual tokens processed per forward, LOWN decides at each step whether to process the image. Because the two axes overlap, using them together compounds the savings. Combined with RedVTP (50% retention), it retains 98.3% at 0.223× FLOPs and ranks first on all 10 benchmarks (source: §5.2, Table 1). Combined with the key-value cache (Fast-dLLM), it cuts generation time by an additional 38.1–50.1% relative to Fast-dLLM, saves an additional 44.1–58.9% of FLOPs, and decodes 5.6–7.0× faster than LLaDA-V alone (source: §5.2, Table 2). However, the scores of this combination are lower than Fast-dLLM alone on all three tasks (by 5.0 points on LLaVA-Bench) and within 2.0 points of the original (source: §5.2).

Wall-clock time and other backbones

The performance table above is based on analytical compute (FLOPs). Actual wall-clock time, measured on an H200 with batch size 1, shows LOWN alone cutting generation time by 57.4% on average and up to 73.2% (source: §5.2, §D.1). The gap from the FLOPs reduction (64.7%) is hard to compare directly because of differences in what is measured. FLOPs is a matrix-multiplication-centric analytical value excluding the vision encoder and non-matrix operations, while time includes multimodal preparation and vision encoding within the execution flow (source: §B.2, §D.1).

The same rule transfers as is to LaViDa, and extends to VidLaDA under a controlled video protocol. On LaViDa it retains 99.7% of the score at 0.174× FLOPs. On the video model VidLaDA-8B, under a controlled 128-token decoding protocol, EgoSchema scores 60.2 at 0.255× FLOPs (original 60.6) and MLVU scores 43.3 at 0.272× (original 41.8) (source: §5.2, Table 3).

Why adaptive judgment beats a fixed schedule

Compared with “read the image every n steps” or random schedules, adaptive judgment is indeed better. It achieves 4.9–13.0% higher task-normalized quality than random schedules matched on the number of FULL passes, and a period-4 schedule at similar compute is 4.0 points lower on MathVista (source: Table 4). Breaking down the cost contribution, image-access routing alone reduces FLOPs by 27–68%, and adaptive acceptance—consisting of disagreement-first ordering and accepting the whole prefix—reduces it by another 21–43% on top. Adaptive acceptance raises scores on 7 benchmarks, and on the rest the normalized