Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

Paper

Why Are Video LLMs Still So Expensive? — A Four-Stage Guide to Inference Efficiency Mechanisms

TL;DR — This paper classifies 125 VideoLLM inference efficiency studies published between late 2022 and August 2026 into 4 stages of the encoder–connector–LLM pipeline, and performs controlled comparisons only under “the same host model · the same input protocol,” concluding that accuracy is nearly preserved even when only about 25% of visual tokens remain. (source: Abstract, §I)

Core Idea

Video understanding has shifted from task-specific architectures to large-scale pretrained models, and VideoLLMs (video encoder + connector + LLM) have become the mainstream, but their cost grows explosively with the number of frames and the context length. (source: §I)

The central claim of this survey is simple. Efficiency should be classified by which stage of the pipeline the cost is removed from, and numbers reported under different hosts and input conditions cannot be directly compared. To this end, the authors establish two axes:

1. A 4-stage taxonomy — input composition → encoder computation → representation/connector → LLM execution & state. (source: Fig. 4)
2. A controlled comparison protocol — numbers are placed side by side only under “the same host + the same number of frames + the same evaluation harness”; otherwise they are treated merely as “indicative evidence.” (source: §IV-A)

They also specify a system-level definition of “efficient”: reducing parameter count, per-input FLOPs, latency, and memory while maintaining accuracy. (source: §I)

Background: The Problem They Set Out to Solve

VideoLLMs share the standard encoder–connector–LLM architecture. (source: Fig. 2) The problem comes from three places:

• The vision encoder processes hundreds of frames per clip at high resolution, and (source: §I)
• The result floods the LLM context with tens of thousands of visual tokens, and (source: §III-B)
• As the context length $L$ grows, the prefill cost increases quadratically and KV cache memory increases linearly. (source: §III-B)

Such efficiency techniques are usually tied to a single task (captioning, QA, temporal grounding, etc.) and evaluated with different metrics, making it hard to judge “exactly where the cost goes, and which technique is most effective under a given constraint.” (source: §I) Existing surveys were capability- and architecture-centric, covered only a single family of token compression, or omitted frame selection and encoder design. (source: §I) This paper pulls that fragmentation together with a video-focused lens.

New Approach: A 4-Stage Pipeline Taxonomy + Controlled Comparison

This paper’s “method” is not a new model but a classification scheme and a comparison methodology. It restricts VideoLLMs to the Embedder family (encoder–connector–LLM) and assigns each efficiency mechanism to the pipeline stage where it removes cost. (source: §III-A, §IV)

(source: Fig. 4, §IV-B–E)

The key design decision is the principle of controlled comparison. Accuracies from different hosts are never read as a ‘ranking,’ even when placed in the same table. Instead, numbers are compared in the same table only when the following conditions hold: (1) the same host LLM (mostly 7B–8B), (2) the same number of frames and input modality, (3) the same evaluation harness (e.g., LLaVA-OneVision-7B, 32 frames, LMMs-Eval). (source: §IV-A, Tab. V)

How It Works: A Walkthrough with Concrete Examples

1) Where the Tokens Explode

A toy example assuming the standard CLIP ViT-L/14. (source: §III-B)

• Frame resolution $336 \times 336$, patch size $14 \times 14$ → patches per frame $$N_p = \frac{H}{P}\cdot\frac{W}{P} = 24 \times 24 = 576$$
• With 64 frames, the encoder output tokens are $$N_v^{enc} = T \cdot N_p = 64 \times 576 = 36{,}864$$
• The LLM context, adding the text prompt $N_t$, is $$L = N_t + N_{ev} + N_{ea}$$

That is, visual tokens vastly outnumber the few dozen prompt tokens: prefill attention cost grows as $FLOPs_{attn} \propto N d^2 + N^2 d$, and KV memory as $Mem_{KV} \propto 2B \cdot n_{layers} \cdot L \cdot d_{KV} \cdot b$. (source: §III-B)

  flowchart LR
  A["Video/audio input"] --> B["1. Input composition<br/>Frame/patch selection"]
  B --> C["2. Encoder computation<br/>Vision/audio backbone"]
  C --> D["3. Representation & connector<br/>Token compression/resampling"]
  D --> E["4. LLM execution & state<br/>prefill / KV cache"]
  E --> F["Response generation"]

2) What Each of the 4 Stages Saves

The key insight is that each stage saves something different, and the stages interact with one another. (source: §V)

• Stage 1 (frame selection) reduces $T$ before the encoder ever sees the frames, so it cuts encoder cost and LLM context simultaneously. However, frames discarded at this stage cannot be recovered. (source: §IV-B)
• Stage 2 (encoder) makes feature extraction itself cheaper. Lightweight backbones cover the low-compute regime, while pooled-attention transformers cover a wide accuracy range. (source: §IV-C, Fig. 5)
• Stage 3 (connector) reduces only the LLM input after the encoding cost has already been paid. Prefill cost drops, but the encoding cost of the remaining frames stays. (source: §IV-D)
• Stage 4 (inside the LLM) reduces prefill computation (decoder pruning) or decoding memory/latency (KV compression). The two mechanisms combine in a complementary way. (source: §IV-E)

The twist the authors emphasize is that selection does not always happen ‘upstream.’ Frame-Voyager and FlexSelect encode all candidate frames first and only then select, so they reduce the LLM context but save no encoder cost at all; Frame-Voyager was in fact 27.6% slower than uniform sampling. (source: §IV-D, §V)

3) The “Secret Weapon” — Explicitly Model Temporal Redundancy

The clearest pattern in the controlled comparison (Tab. V) is this: at a 25% budget, most techniques stay within 1.5% of the baseline, but at a 10% budget, the choice of technique becomes decisive. (source: §IV-D, Tab. V)

(source: Tab. V)

Why the difference? Because VisionZip keeps only spatially “important” tokens, yet the key redundancy in video is temporal redundancy across frames. PruneVid and HoliTom explicitly merge this temporal redundancy, so they degrade much more gracefully at low budgets. (source: §IV-D) The same logic applies at the decoder stage: HieraVid holds a 0.2–2.1 point gap from the baseline at 24.5% FLOPs, outperforming FastV, which needs a larger budget (39.3% FLOPs). (source: §IV-E, Tab. VII)

Performance Validation: Key Results

Since this is a survey, the results are cross-validated insights rather than “self-reported performance.” The highlights:

① The 25% rule. Multiple method families keep nearly baseline accuracy with only about 25% of visual tokens — pixel-shuffle (InternVL2.5), pooling (PLLaVA), temporal compression (STORM), audio-guided pruning (DASH), and decoder hierarchy (HieraVid) alike. That said, these results come from different hosts, so they do not imply every model can safely drop 75%. (source: §V)

② Diminishing returns in frame selection. Under the same LLaVA-Video-7B · 64-frame protocol, query-aware selectors gain up to 5 points over uniform sampling on LongVideoBench, but the advantage nearly vanishes on Video-MME. The top performer was a policy (TSPO) with a mere 3.5M parameters. (source: §IV-B, Tab. II)

③ Strong 2025–2026 methods combine stages. PruneVid, MeToM, FlashVID, HieraVid, and HoliTom reduce tokens both before and inside the LLM simultaneously. In such cases the end-to-end gain cannot be attributed to any single stage, and only HoliTom reports an ablation that separates the two stages. (source: §V)

④ The retrieval–eviction trade-off in streaming. ReKV’s retrieval-based approach best preserves streaming accuracy but keeps peak GPU memory at the full-cache level, while hard-capped eviction (InfiniPot-V, StreamMem) gives up about 6 points (RVS-Ego) in exchange for lowering the memory ceiling by roughly 10 GB. (source: §IV-E, Tab. VI)

Our Perspective: Strengths, Limitations, and Why This Work Matters

Strengths

The real contribution of this survey is that it places “comparability” at the center of its methodology. When hundreds of papers report GFLOPs and accuracy under different hosts, frame counts, and modalities, the authors neither estimate nor convert numbers — they use only explicitly reported values and draw a boundary so that only matching tables are read as ‘rankings’. (source: §IV-A) In particular, the remark that “a single number like a 25% token retention rate says nothing about where the savings came from” (§V) pinpoints exactly the question we must ask when reading efficiency papers.

Limitations

As the authors themselves acknowledge, the comparison is inherently limited. (source: §IV-E)

• Encoder FLOPs ≠ end-to-end efficiency. An efficient Kinetics backbone is not guaranteed to be efficient for VideoLLM QA either. (source: §V)
• Bias in accuracy metrics. Nearly all comparisons collapse into a single multiple-choice QA task, and not a single captioning, retrieval, or temporal grounding metric appears in the tables. MCQ is often solvable with coarse object and scene cues, so a retention rate that is lossless on MCQ need not be lossless on generation tasks. (source: §V)
• Lack of audiovisual evidence. Audio encoder cost is frequently omitted, and modality ablations are rare, making it hard to separate the benefit and cost of audio-guided selection. (source: §V)
• No energy reporting. None of the surveyed methods reported energy consumption. (source: §IV-A)

Why It Matters

This is a survey that asks “what claims are verifiable” rather than “what is new.” Video LLM efficiency is now a field with new papers every month, and amid the chaos this survey provides the baseline for designing the next experiment — the same host, the same inputs, explicit FLOPs accounting boundaries, and including the selector’s own cost. In particular, the observation that “reinvesting saved compute into more frames can actually let compression raise accuracy” (§V) reframes efficiency not as simple lossy compression but as expanding temporal coverage.

What’s Next?: The Road Ahead

The authors propose a three-pronged research agenda. (source: §V)

1. Establish a common analysis protocol. The same video, prompt, and modality, fixed resolution and decoding settings, a frozen host, and FLOPs accounting that includes the cost of the selection/allocation module itself. As a practical starting point, they propose the LLaVA-OneVision-7B · 32-frame · LMMs-Eval setup. (source: §V)
2. Report per domain and task family. Video-MME annotates content domains and lengths, yet mostly reports aggregate numbers. A budget validated on lecture videos and one validated on sports highlights should be kept separate. (source: §V)
3. Learn when audio should intervene in compression. OmniZip and DASH showed the promise of audio-guided compression, but speech can point to off-screen events and visible events can have no sound. The next step is learned, query- and context-conditional cross-modal budget allocation, and its cost accounting must include the audio encoder. (source: §V)

The authors expect that the LLM-side method family (KV eviction, quantization, sparse attention) will grow the fastest, since it has transferred almost unchanged from text LLMs, and once pre-LLM compression matures, the remaining cost is the decoding memory and cache growth of multi-turn and streaming. (source: §V) In addition, as decoder backbones diversify beyond dense transformers into Mamba–Transformer hybrids, preserving the information of removed tokens in the recurrent state is emerging as a new evaluation challenge. (source: §IV-E)

In summary, this survey draws a map of “how to make Video LLMs cheap” while honestly exposing how trustworthy the numbers on that map are. The question we should now ask when reading an efficiency paper is clear: not “how many percent did it cut,” but “at which stage, under what host and input conditions, and counting the selector’s own cost, was it cut.”

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. VideoLLM benchmarks used in the comparisons of this survey. Mod.\ denotes the modalities provided beyond the question text (V=video, A=audio, T=transcript/subtitles). Dur.: short ($<$1 min), medium (1–10 min), long ($>$10 min). #V = Number of videos, #Q = Number of questions. Fmt.: MCQ = Multiple Choice Question, OE = Open Ended

Table 2. color-frameFrame samplers on LLaVA-Video-7B at a $\sim$64-frame budget. All rows share the same uniform baseline (LongVideoBench 58.9 / Video-MME 64.4) unless marked. $^\dagger$FOCUS reports a 32–64 frame budget, not a fixed 64. $^\ddagger$Own uniform-baseline reproduction differs from the shared one (EFS: 58.8/64.6).

Table 3. color-backboneVision Encoder efficiency methods. GFLOPs$\times$v gives inference GFLOPs per view times the number of temporal$\times$spatial views used for the reported accuracy, as stated by each paper; $^t$ marks papers reporting only the total across views (per-view cost not separately stated). Abbreviations: K-400/600 = Kinetics-400/600 , MiT = Moments in Time , AS = AudioSet , UCF = UCF101 , HMDB = HMDB51 .

Table 4. Reported performance of token reduction methods before and within the LLM. Groups follow the families in Figure ; joint methods may also act at other stages. $^h$ marks a plug-in host size. Retained budgets follow the source and may describe tokens, memory or audio rate. Hosts, inputs, protocols and baselines differ, so these rows are indicative and do not rank methods. NX-QA = NExT-QA, MSR = MSR-VTT-QA, MSVD = MSVD-QA, MVB = MVBench, VME = Video-MME w/o, ES = EgoSchema, ActNet = ActivityNet-QA. HyperCLOVA\textquotesingle s budget describes its documented audio compression; its QA score is a system-level result.

Table 5. Training-free reduction on a shared LLaVA-OneVision-7B host (32 frames, LMMs-Eval). HoliTom re-runs the upper blocks under one harness ; the lower block collects own-paper runs on the same host and frame count. FLOPs are relative LLM-prefill costs, except $^e$, which also includes vision encoding. Avg.\ is relative to the 58.4 baseline mean; $^p$ marks a paper-reported relative average against that paper’s own baseline. $^m$ marks methods whose own MVBench baseline reproduction differs from the shared 58.3 (VidCom2 and FastVID report 56.9, MMG-Vid 57.6).

Table 6. Streaming memory systems, including representation memory and LLM KV state, under the two protocols the literature shares. Top: offline long-video QA on a shared Qwen2-VL-7B backbone against its full-KV baseline. Baseline reproductions drift with frame count (Video-MME w/o 63.3–63.9, MLVU 63.9–65.8, LongVideoBench 55.6–58.8); each method is judged against its own reproduction, so cross-row gaps within a point are not meaningful. Bottom: streaming QA on a shared LLaVA-OneVision-7B backbone (RVS-Ego / RVS-Movie ), reproduced under one protocol by StreamMem ; peak memory is for a 1-hour 0.5-FPS stream where reported.

Table 7. color-llmDecoder-layer visual-token pruning on a shared LLaVA-Video-7B backbone, as re-run by HieraVid at matched ${\sim}30\%$ token budgets (FastV runs at a larger $39.3\%$ FLOPs budget). ``FLOPs’’ is prefilling FLOPs relative to the unpruned model; accuracy is %.