HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models

Paper

arXiv:2610.05739 · Monash University · The University of Adelaide · Zhejiang University

One-Line Summary (TL;DR)

Recurrent linear attention (Gated DeltaNet) in video world models compresses memory into a fixed-size state, which is efficient, but when the camera leaves and returns, distant scenes are progressively forgotten. HLA-WM caches chunk-level affine summaries in a training-free way and uses camera geometry (frustum overlap) to retrieve and recompose the relevant scenes, achieving PSNR +0.74 dB, rotation error −28.5%, and a 12× reduction versus Full KV in history memory over a 60-second rollout (about 6× the memory of GDN) (source: §Abstract, Fig. 1).

Key Idea

Maintaining revisit consistency requires a middle ground between a “token-level KV cache over the full history” and a “single recurrent state.” The core claim of HLA-WM is:

By exploiting the affine structure of GDN’s recurrent state transition, each time chunk is compressed into a single transition summary $(A_i, B_i)$, and at query time only the chunks overlapping the current camera geometry are selected and recomposed in temporal order, bypassing the decay introduced by irrelevant intermediate scenes.

In other words, this is a training-free technique that changes not how memory is stored but how memory is addressed and accessed (source: §1, §4.1).

Key Numbers

Background: the Problem They Solved

A world model predicts the dynamics of an environment so that an agent can simulate future observations and the consequences of its actions. Recent video world models implement this via autoregressive generation conditioned on camera trajectories or actions (source: §1). To go beyond short-term temporal continuity and achieve long-horizon simulation, past scenes outside the recent context must remain recoverable later. When the camera returns, the model must restore the earlier scene structure rather than produce a semantically similar but spatially inconsistent view (source: §1).

Here two extremes diverge:

• Softmax attention + Full KV cache: Direct access to the entire history, but the storage cost keeps growing in proportion to the rollout length (source: §1, Fig. 4).
• Recurrent linear attention: Compresses the history into a fixed-size state, so memory is cheap, but distant information is vulnerable to decay and interference from subsequent updates (source: §2).

The authors quantitatively capture the long-range forgetting problem in Gated DeltaNet (GDN). GDN’s chunk-level recurrent transition is:

where $A_i$ is the state transition applied to past memory and $B_i$ is the information newly written by the current chunk. Unrolling this across several chunks gives (source: §3, Eq. 1–2)

that is, the information $B_i$ written by an early chunk $i$ must pass through every subsequent transition matrix in turn before it can influence a distant $j$. The authors measure this decay with an average norm:

The result is dramatic. In a representative trajectory, the camera leaves the Chunk 3 region, makes a large loop, and returns to a nearby viewpoint at Chunk 35; the retained influence of Chunk 3, $W_{3\to 35}$, drops to 0.0416 (source: Fig. 2). Chunk 3 and Chunk 35 see a region with large overlap, but the irrelevant intermediate viewpoints in between successively multiply the transitions and repeatedly distort the original memory.

Key insight: GDN memory is updated according to “the passage of time,” but in a world model the relevance of information is determined not by time but by spatial/geometric proximity. This mismatch is the root cause of forgetting (source: §3).

New Approach: HLA-WM

HLA-WM replaces the single accumulated GDN memory with a collection of independently addressable per-chunk memories (source: §4.1). It consists of three stages.

1. Per-chunk GDN memory — Cache each time chunk as a single affine transition summary $(A_i, B_i)$.
2. Geometry-based retrieval — Compute a frustum overlap score from the camera pose, intrinsics, and scene median depth to select relevant chunks.
3. Selective state recomposition — Compose the selected summaries in temporal order to produce a query-specific recurrent state.

How It Works: A Concrete Walkthrough

1) GDN’s affine structure. The token-level GDN update is (source: §4.2, Eq. 4):

where $k_t, v_t$ are the key and value projected from the current token, $\gamma_t$ is the memory decay, and $\beta_t$ is the delta update strength. Rearranging this gives the affine form $S_t = S_{t-1}A_t + B_t$, with

Unrolling the $L$ tokens within a chunk $i$, the entire chunk can be represented by a single affine transition $(A_i, B_i)$. These summaries satisfy the associative law (source: Eq. 9):

This is the crux. Thanks to the associative law, any subset of history chunks can be recomposed efficiently without re-running the original visual tokens (source: §4.2).

2) Geometry retrieval. The overlap between each query chunk $q$ and history chunk $i$ is estimated from camera geometry alone, not from RGB or learned features. At a sampled pixel location $(u,v)$, the camera-space ray is formed as

and 3D points are scattered at three proxy depths around the scene median depth, $D = \{0.5d_{\text{med}}, d_{\text{med}}, 1.5d_{\text{med}}\}$, then projected into the camera view of the relative chunk to obtain the visible ratio $V$. The symmetric frustum overlap score is (source: §4.3, Eq. 12)

3) Selection and recomposition. The first chunk is always retained as a sink, the recent chunks preserve local continuity, and the top $K$ of the remaining history by $R(i,q)$ are selected (source: Eq. 13)

and the selected chunks are sorted in temporal order and composed to produce the query-specific history state (source: Eq. 17):

Concrete example. Return to the loop trajectory of Fig. 2. The camera goes Chunk 3 (first visit) → Chunk 4–34 (irrelevant distant departure) → Chunk 35 (revisit). When building $S_{35}$, the existing GDN passes the information written by Chunk 3 through the roughly 32 transitions of Chunk 4–34 in turn, eroding it down to $W_{3\to 35} = 0.0416$. HLA-WM, by contrast, detects when generating Chunk 35 that the frustum overlap with Chunk 3, $R(3, 35)$, is high, and selects Chunk 3’s summary $(A_3, B_3)$ along with the sink and recent context, composing them while skipping the transitions of Chunk 4–34. This vastly reduces the number of irrelevant transitions a distant but relevant memory is exposed to before reuse (source: §4.4).

Performance Validation: Main Results

Experiments were run on SANA-WM-Bench (80 trajectories each for simple/hard, 4 scene categories) and MBench-A (547 samples, 4 subsets), with the official 1.6B SANA-WM as the base model. HLA-WM modifies only the Stage-1 memory mechanism and does no fine-tuning (source: §5.1).

SANA-WM-Bench. In Stage-1 (the stage that directly modifies recurrent memory), HLA-WM improves both revisit consistency and camera control metrics across both splits (source: §5.2, Tab. 1):

| Pipeline | Metric | SANA-WM | HLA-WM | Δ | |—|—:|—:|—:| | Hard · Stage-1 | PSNR ↑ | 9.37 dB | 10.11 dB | +0.74 dB | | Hard · Stage-1 | SSIM ↑ | 0.1774 | 0.1997 | +0.0223 | | Hard · Stage-1 | LPIPS ↓ | 0.6044 | 0.5848 | −0.0196 | | Hard · Stage-1 | RotErr ↓ | 20.38° | 15.09° | −26.0% | | Simple · Stage-1 | PSNR ↑ | 9.18 dB | 9.91 dB | +0.73 dB | | Simple · Stage-1 | RotErr ↓ | 19.70° | 13.59° | −31.0% |

That is, the abstract’s headline numbers “PSNR +0.74 dB, rotation error −28.5%” combine the Stage-1 results for Hard (+0.74 dB, −26.0%) and Simple (−31.0%) (source: §Abstract, §5.2). Interestingly, the camera control metrics also improve. Preserved scene geometry reduces structural drift, providing a more stable spatial reference for camera-conditioned generation (source: §5.2).

Retained after downstream refinement. In bidirectional refinement all six metrics improve (PSNR +0.52/+0.46 dB, RotErr −22.0%/−20.1%), and revisit consistency improves consistently under causal AR refinement as well. However, in the refinement stage there is a trade-off in which some metrics worsen slightly depending on the mode (TransErr and CamMC for Simple) (source: §5.2, Tab. 1).

Generalization: MBench-A. Across all 547 samples, PSNR, SSIM, and LPIPS improve consistently in all three inference modes. In Stage-1 the largest gains appear: PSNR +0.84 dB, SSIM +0.0303, LPIPS −0.0428 (source: §5.2, Tab. 2). The gains are not concentrated in a specific subset; all three metrics improve simultaneously across all 12 mode–subset combinations (source: Appx. A.8.2).

Efficiency. HLA-WM offers a practical compromise between Full KV attention and recurrent linear attention. Because the history is cached in chunks rather than tokens, memory scales with the number of chunks rather than the number of tokens. On a 60-second context it uses only 2.15 GiB (a 12× reduction over Full KV) while incurring a throughput loss of at most 1.6% (source: §5.2, Fig. 4).

Key ablations. The authors answer two questions (source: Appx. A.3–A.5).

1. Which components matter? Removing the sink, disabling transition selection (“w/o Select A”), disabling write selection (“w/o Select B”), and removing turn-aware recent context all degrade revisit consistency and camera control. In particular, leaving in irrelevant intermediate transitions drops PSNR and SSIM substantially, supporting the claim that “irrelevant transitions are the cause of forgetting” (source: Appx. A.3, Tab. 3).
2. Which matters more, GDN recomposition or KV alignment? GDN-only yields a far larger gain than KV-only, and combining the two gives the best performance. In the Hard trajectories of bidirectional refinement, GDN-only raises SSIM but worsens RotErr from 13.17° to 33.04°, showing a “memory-path mismatch”; applying the same selection to the KV cache stabilizes it again (source: Appx. A.4, Tab. 4).

Retrieval budget and selection strategy. Surprisingly, $K{=}1$ (the single most relevant history chunk) is best on PSNR, SSIM, and LPIPS. Retrieving more gives no consistent improvement (source: Appx. A.5, Tab. 6). Also, geometry-based selection gives better revisit consistency than heuristics such as Random, Recent, and Uniform, showing that beyond “broad temporal coverage,” explicitly choosing view-relevant history provides additional benefit (source: Appx. A.6, Tab. 7).

Effect of revisit interval. The gain grows with the revisit interval (PSNR improvement rises from +0.375 dB at short intervals <10s to +0.992 dB at long intervals ≥40s). This result aligns exactly with the method’s intent: the benefit of selective long-term memory is most pronounced for the “heavily forgotten” long-range revisits (source: Appx. A.7, Fig. 6).

Our Perspective: Strengths, Limitations, and Why This Work Matters

Strengths. The most striking aspect is the practicality of being training-free. It leaves the pretrained 1.6B model untouched and only changes how memory is accessed at inference time to raise long-range consistency. Exploiting GDN’s affine structure analytically to scale memory as $O(\text{chunks})$ rather than $O(\text{tokens})$ while simultaneously achieving a 12× reduction and a throughput loss of around 1.6% is a strong demonstration of the viability of the idea of “selectively addressable recurrent memory” (source: §4.4, Fig. 4). The GDN-only vs KV-only ablation, which isolates that “recurrent state recomposition is the main driver,” is also analytically clean (source: Appx. A.4).

Limitations. As the authors explicitly acknowledge, the training-free property itself constrains the ceiling. Retrieval relies only on camera geometry and FOV overlap and does not model occlusion, visibility, or semantic relevance (source: §6). We would point to three additional limitations. First, retrieval assumes a prescribed camera trajectory, whereas in open-loop generation by a real agent the pose must be estimated separately, and that error propagates into retrieval quality. Second, the “fixed 24-frame chunk” and “sink 1 + recent 1 + Top-1” design are values tuned on the specific SANA-WM-Bench and MBench-A pipelines, so robustness to different chunking and fps remains open. Third, the result that $K{=}1$ is optimal also means, conversely, that “richer retrieval does not help,” suggesting a representational limit in the retrieval signal itself (source: Appx. A.5).

Why this work matters. This paper connects the LLM KV-cache compression and long-context literature with the video world model literature. On the spectrum between a “full history cache” and a “single compressed state,” it offers a third point — selective recomposition using geometric information as an addressing signal — and shows that it works without training. As video world models grow longer, “how to address memory” will become the bottleneck, and this paper provides one concrete, measurable answer to that question.

What’s Next?: The Road Ahead

The authors propose integrating selective state retrieval into training and richer or learned retrieval signals that include occlusion, visibility, and semantic relevance as future directions (source: §6). Beyond these, the directions we would look forward to are as follows.

• Combining with a learned router. For now, camera geometry is a “free” addressing signal, but it has limits for semantic revisits (the reappearance of the same object rather than the same place). A hybrid in which geometric signals serve as a prior and a lightweight router corrects them is natural.
• Occlusion-aware retrieval. Frustum overlap cannot see “behind walls.” Adding depth-buffer or occupancy signals could reduce occlusion errors in indoor scenes.
• Automating chunking and the retrieval budget. Instead of a fixed 24 frames and Top-1, dynamically adjusting the chunk size and $K$ based on scene change rate or motion magnitude could achieve a better efficiency–quality trade-off.
• Deeper optimization of the recomposition kernel. The forward scan the authors implemented on top of a Triton kernel is a starting point. Batching and memory-layout optimizations could reduce that 1.6% throughput loss further (source: §4.4, Appx. A.2).

In summary, HLA-WM starts from a clear diagnosis of “the forgetting of recurrent linear attention,” offers an elegant and practical solution — geometry-based selective recomposition — and robustly validates its effectiveness on two benchmarks. By driving this much long-range consistency improvement without any training, this work will be an important stepping stone toward efficient long-horizon world models.

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.