ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

Paper

ShallowStream: Index Shallowly, Answer Deeply — Unlocking the Computational Bottleneck of Streaming Video Understanding

TL;DR — In streaming video understanding, the most expensive cost is prefilling every frame through the MLLM’s full depth (28–32 layers). ShallowStream builds frame encoding and a retrieval index with only 4–5 shallow layers, and runs full-depth computation on the retrieved evidence only at the moment a question arrives. As a result, it keeps SOTA-level performance (OVO-Bench 69.5 / StreamingBench 78.2) while cutting per-frame prefill by up to 52.1× and 10-second end-to-end latency by up to 11.9× (source: Abstract).

Core Idea

Streaming video understanding is asymmetric in its workload. Video frames keep arriving without pause, whereas user questions occur only intermittently (source: §1). Yet existing methods push every incoming frame through the full Transformer stack to accumulate KV cache, “not knowing what question will come later.” The deep KV cache built this way is ultimately computation that may never be attended to (source: §1).

ShallowStream’s starting point is a single empirical observation. Qwen3-VL-8B already shows strong retrieval ability at the 4th of 28 layers, and LLaVA-OneVision-7B at the 3rd of 32 layers (source: §3, Fig. 2). In other words, determining “which past frame is relevant to this question” does not require full depth.

Building on this observation, ShallowStream splits the streaming pipeline into two phases (source: §4):

1. Query-agnostic stream processing — encodes incoming frames with only the shallow layers and maintains a lightweight full-history index from their KV cache.
2. Query-time answering — a lightweight gate decides whether history retrieval is needed, and only when it is does it concentrate full-depth computation on the retrieved evidence.

In short, the strategy is “skim shallowly in the ordinary flow, and re-examine in depth only the relevant scenes when a question arrives.” Expensive deep-layer computation is deferred until the evidence a question actually needs is pinpointed (source: §1).

Background: The Problem They Tackled

MLLM-based streaming video understanding is a core capability for real-time applications such as embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants (source: Abstract). Unlike offline long-video understanding, streaming settings require causally processing an open stream when neither future frames nor future questions are known (source: §2).

The authors explicitly point to three challenges shared by prior work (source: §1):

• Expensive Stream Processing — KV-centric methods such as ReKV, StreamMem, and InfiniPot-V prefill every frame through the full Transformer stack without knowing future questions. Even if the cache is later compressed, the computation already paid cannot be undone, and it produces deep KV cache that may never be attended to (source: §1).
• Lossy Evidence Reduction — query-agnostic compression, merging, and eviction to save memory risk discarding evidence that future retrospective queries will need. Conversely, preserving the full state makes memory balloon and forces reliance on bandwidth-limited offloading (source: §1).
• Inefficient History Use — always appending history to the context increases latency and interferes with current-scene awareness. On the other hand, approaches that trigger retrieval only when needed, such as OASIS and WeaveTime, spend one expensive answer-level reasoning pass on routing itself, and then a second reasoning pass once retrieval is activated (source: §2).

To summarize these three problems in one sentence: existing methods overlook the “model depth” dimension — this is the paper’s core diagnosis (source: Abstract).

The New Approach: ShallowStream

ShallowStream keeps a pretrained MLLM as is but redesigns where (shallow/deep layers), when (stream/query time), and what (only the selected evidence) is computed. There is no additional training at all (training-free) (source: §5).

The overall pipeline is as follows (source: Fig. 3):

1) Query-Agnostic Shallow Index Construction (§4.1)

• Streaming shallow prefill — each video unit $$x_t$$ becomes $$\bm{H}t^0$$ through the vision encoder $$E{\mathrm{v}}$$, and passes through only the shallow layers $$[0, P)$$ to produce retrieval KV. Because the deep layers $$F^P, \dots, F^{L-1}$$ are skipped, per-unit computation drops from $$L$$ layers to $$P$$ layers (source: §4.1). Here the pruning boundary $$P$$ is 5 for Qwen3-VL-8B and 4 for LLaVA-OneVision-7B (source: Tab. Appx).
• Full-history shallow cache — the shallow KV of every observed unit is preserved: $$ \mathcal{C}T^{\mathrm{shallow}} = \left{ (\bm{K}{1:T}^{i}, \bm{V}{1:T}^{i}) \right}{i=0}^{P-1} $$ Cache size shrinks to roughly $$P/L$$ compared with preserving full depth (source: §4.1).
• Per-unit diversity descriptor — keys from the last shallow layer are averaged and $$\ell_2$$-normalized into a fixed-length descriptor $$\bm{k}_t$$ (source: Eq. 3).
• Selective long-cluster compression — when history exceeds budget $$B$$, older units are grouped into temporally adjacent clusters that keep fixed-size representative KV. Host-to-device transfer per cluster is fixed at one representative regardless of member count (source: §4.1).

2) Query-Time Answering (§4.2)

• Query-logit gate — the question is placed into a 10-shot routing prompt, and the logit difference of a single-token choice (A: past memory needed / B: recent scene suffices), $$\ell_{\mathrm{ret}} - \ell_{\mathrm{recent}}$$, is compared against a threshold $$\tau_g$$. Routing finishes in a single text forward, so there is no separate router training and no expensive reasoning pass (source: §4.2). The threshold is set once on a benchmark-independent calibration set and then fixed (source: §4.2, Appx).
• Token voting — when the gate activates retrieval, the last prompt token computes RoPE-aware attention scores attending to all history visual tokens at each shallow layer. The top $$M=64$$ tokens are drawn from each layer, and each history unit receives votes equal to the number of selected tokens belonging to it (source: §4.2).
• Max-min diversity selection — after keeping the top $$4K$$ candidates by vote ranking, the method starts from the farthest pair and iteratively adds the candidate whose minimum cosine distance to the selected set is largest, so the budget is not wasted on duplicate scenes. Finally $$K=8$$ units are chosen, and each unit gets its temporal neighbor (the immediately preceding unit) attached (source: §4.2).
• Selective re-prefill — only the input states $$\bm{H}_t^0$$ of the selected units are gathered and passed together with the question through all $$L$$ layers from scratch to generate the answer. Shallow KV is used solely to decide “what to look at in depth”; partially computed states are not carried over into generation (source: §4.2).

How It Works: A Concrete Walkthrough

Let us trace the pipeline with a very small toy example. Assume Qwen3-VL-8B ($$L=28$$ layers total, $$P=5$$, video unit = 2 frames).

Step 1 — Stream time: encode shallowly only

Video units arrive in sequence: $$x_1, x_2, x_3, \dots$$ Each unit passes through the vision encoder to become $$\bm{H}_t^0$$ and goes through only layers 0–4 to produce $$(\bm{K}_t^i, \bm{V}t^i){i=0}^{4}$$. Layers 5–27 are never touched. Per-unit computation thus drops to about $$5/28 \approx 17.9%$$ of the full cost (source: §4.1).

Meanwhile, the keys of the last shallow layer (layer 4) are averaged and normalized into a per-unit descriptor $$\bm{k}_t$$, stored one by one. Now memory accumulates only “shallow KV + descriptors” — far lighter than keeping full-depth KV.

Step 2 — Question arrives: the gate decides

The user asks “Where did I put the keys early in the video?” This question goes into the 10-shot routing prompt and the next-token logits are read. If $$\ell_{\mathrm{ret}} - \ell_{\mathrm{recent}} \geq \tau_g$$, it is classified as A (past needed) and retrieval is activated. In contrast, a question like “What color is the vehicle on screen right now?” is classified as B (recent scene suffices) and only the recent context is used (source: §4.2). This decision costs a single text forward of about 92.7 ms (source: Tab. Appx).

Step 3 — Evidence selection via token voting

Once retrieval is activated, the question tokens attend to all stored history tokens at shallow layers 0–4. In this small example, suppose there are only three candidate units. Drawing the top 64 tokens at each shallow layer and tallying the votes might give the following per-unit counts:

$$x_1$$ receives an overwhelming number of votes. Now max-min diversity selection applies. If several scenes closely resemble $$x_1$$, then after the first pick the method preferentially adds the scene “most different from what has already been chosen.” As a result, complementary scenes are selected instead of duplicates up to $$K=8$$ units (source: the qualitative case in Fig. 6 shows this effect directly).

Step 4 — Re-examine only the selected evidence in depth

Only the inputs $$\bm{H}_t^0$$ of the selected units are gathered and passed together with the question through all of layers 0–27. That is, rather than re-reading the whole stream in depth, only the scenes the retrieval pointed to are recomputed at full depth (source: §4.2).

The key point is this: ordinary stream processing stays cheap at $$P$$ layers, and the expensive $$L$$-layer computation is spent only on the evidence a question has pinpointed.

Performance Evaluation: Key Results

Accuracy — on par with SOTA, but far cheaper

ShallowStream matches the strongest streaming methods on both backbones (source: Tab. 1, Tab. 2).

On OVO-Bench with Qwen3-VL-8B, it surpasses OASIS (67.7), the strongest prior baseline, recording an average of 80.9 on real-time visual perception (Real-Time) and 58.1 on retrospective tracking (Backward Tracing) (source: Tab. 1). The noteworthy benchmark is Backward Tracing: retrospective questions that can only be solved by “recalling past scenes,” it directly tests whether ShallowStream’s shallow retrieval genuinely resurrects past evidence. ShallowStream scores 58.1 on this section, higher even than OASIS (57.2) (source: Tab. 1).

Efficiency — a sharp cut to the continuous stream cost

Accuracy is maintained while costs drop substantially. The key metrics are per-frame prefill (the primary cost of stream processing) and 10-second end-to-end latency (source: Fig. 1).

The concrete numbers are equally clear (source: Tab. Appx):

$$P=5$$ keeps the best accuracy with 31.0% less per-frame prefill than $$P=19$$ (source: Appx). The gate and evidence selection account for only about 11% of the total per-query computation.

From a memory standpoint, enabling long-cluster compression (LC-on) keeps peak GPU memory nearly flat at about 18 GiB over the 64–1024 frame range, whereas uncompressed (LC-off) grows to 21.76 GiB. HERMES and OASIS demand more GPU memory at long prefixes (source: Fig. 4-left, §5).

From a real-time headroom standpoint, at an 80-second question interval ShallowStream spends about 2.6 seconds of total computation, while HERMES spends 6.2 seconds and OASIS spends 50.4 seconds (source: Fig. 4-right, §5). In other words, only ShallowStream operates stably below the real-time boundary.

Contribution of the Gate and Retriever (Ablation)

• The gate activates retrieval selectively, only for retrospective questions. Routing behavior differs clearly between Backward and Real-Time questions, showing that retrieval follows “historical need” rather than being indiscriminate (source: Fig. 5-left, §5).
• Token voting provides a stronger evidence-selection signal than pooled shallow Q-K or a standalone SigLIP encoder, and max-min diversity suppresses duplicate selections for further gains (source: Fig. 5-right, §5).

The qualitative cases make this effect visible too. For the question “Where is the red checkered rug?” (correct answer E), pooled Q-K repeatedly selects similar kitchen views and answers B incorrectly, whereas token voting + diversity selection selects complementary views of the whole room and gets E right (source: Fig. 6, §5).

Our Perspective: Strengths, Limitations, and Why It Matters

Strengths

• Opens a new axis: model depth. Prior efficiency work (token pruning, merging, quantization, retrieval, offloading) all touched “token count” or “cache storage,” but ShallowStream controls when full depth is executed (source: §2, §4). This is orthogonal to existing techniques, so the combination potential is high.
• No additional training at all. The gate merely reuses the logits of the pretrained model; no separate router is trained (source: §4.2). It is a drop-in method that can be attached to any offline MLLM.
• Gate calibration is benchmark-independent. Setting the threshold with synthetic questions rather than evaluation-set labels is an impressive design choice aimed at avoiding evaluation bias (source: §4.2, Appx).

Limitations

• A “Work in Progress” notice indicates the review is still at an unfinished stage (source: title footnote).
• The gate recall on the LLaVA backbone is low at 44.00% (Qwen3-VL achieves 97.00%). That is, on LLaVA, more than half of the questions that should be judged “past needed” are misrouted to “recent suffices.” Still, the precision-first design (precision 97.78%) keeps actual retrieval activation reliable (source: Tab. Appx).
• Long-cluster compression is approximate (lossy). Because only one cluster representative is consumed, member-level fine-grained evidence is not recoverable. When uncompressed fits in memory, this approximation is turned off (source: §4.1). That is, a direct evaluation of how much accuracy drops on very long streams is still missing.
• Evaluation is confined to 1 FPS sampling and fixed 16-token generation, measured on a single GPU (RTX 5090) (source: §5). Behavior at higher frame rates, longer generations, and multi-GPU scaling is unverified.

Why It Matters

Streaming video understanding is a domain where “a frame every second” is routine and “the occasional question” is the event. The first-order term of system cost should then obviously be per-frame prefill, yet prior work has consistently failed to treat it as a first-class cost (source: §1). ShallowStream elevates the empirical finding that “shallow layers suffice for retrieval” into a system design principle, establishing an operating point that cuts the continuous stream cost by more than an order of magnitude with little sacrifice in performance (source: Abstract, §6).

What’s Next?: The Road Ahead

The authors do not explicitly list future directions at length, but reasonable next steps can be read from the paper’s structure and limitations.

• Improving gate robustness — there is room to improve routing on low-recall backbones like LLaVA, or to quantitatively compare the logit-based gate with other on-demand approaches (OASIS, WeaveTime) (source: §2, Appx).
• Quantifying the compression–accuracy trade-off on very long streams — a head-to-head comparison against uncompressed is needed to see how accuracy changes as the approximation loss of long clusters accumulates (source: §4.1).
• Combining with other efficiency techniques — token pruning, quantization, and offloading are orthogonal to ShallowStream’s depth control, so verifying the gains under stacked application is a natural follow-up (source: §2).
• Scaling verification — beyond the 1 FPS / RTX 5090 / fixed 16-token setup, behavior at high frame rates, long generations, multi-GPU, and edge-device deployment needs to be checked (source: §5).

In summary, ShallowStream breaks the implicit assumption that “every frame must be read to the end,” and convincingly shows that deferred depth is a valid way to capture both performance and efficiency in streaming video understanding. Index shallowly, and answer deeply only when needed — intuitive, yet an idea no one had seriously pushed before.

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. Main implementation settings for the two evaluated backbones.

Table 2. Cost summary under the final calibrated Gate and token-vote retriever. Stream-time prefill is averaged over the same five long videos; query components use the matched full-history LC-off setting.

Table 3. Query-logit Gate calibration on the benchmark-independent query-only set.

Table 4. Main results on the OVO-Bench. Baseline results marked with $\ddagger$ are from our controlled reruns. Results marked with $\dagger$ enable long-cluster compression. Dashes indicate unreported entries.

Table 5. Main results on the Real-Time Visual Understanding subset of StreamingBench. Baseline results marked with $\ddagger$ are from our controlled reruns. Results marked with $\dagger$ enable long-cluster compression. Dashes indicate unreported results.

Figures in this post are taken from the original arXiv:2609.02780 (CC BY 4.0). Only size and format were changed.