CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs

Paper

CoVeR: Coverage-Based Pruning That Cuts Visual Tokens of Multi-View 3D Reasoning by up to 92% Using Geometry Alone

One-Line Summary (TL;DR)

Rendering a 3D scene into multi-view images and feeding them to a 2D VLM makes tokens explode proportionally to the number of views. CoVeR selects tokens that cover the entire scene uniformly using only the tokens’ 3D coordinates, with no training, retaining 93.5% of the original performance with only about 8% of the tokens, and outperforming the existing SOTA by an average of 3.9%p (source: Abstract, §4.1).

Core Idea

Existing visual token pruning methods fall into two broad families, and the authors argue that neither suits the multi-view 3D setting (source: §1, Tab. 1):

• Learned importance: assigns an importance score to each token via attention maps or encoder features and keeps the top-K. However, multi-view redundancy is inherently geometric (multiple cameras observe the same physical region at the same time), so importance scores that ignore geometry leave near-duplicate tokens from a few salient regions while leaving most of the scene unrepresented (source: §1, §3.2).
• Voxelization: back-projects tokens into 3D and pools the tokens that fall into the same voxel. Spatial coverage is good, but it cannot control the output token count precisely, and as views overlap more, voxel occupancy saturates, stalling retention at about 69% (source: §3.2, Fig. 2).

CoVeR’s central claim is that explicitly optimizing spatial coverage under a fixed budget improves 3D reasoning performance. The authors treat coverage not as a byproduct but directly as the objective, and guarantee an exact per-scene budget with two stages (voxel initialization + FPS expansion) (source: §3.3).

Background: The Problem They Solve

Because 3D-language data is several orders of magnitude scarcer than internet-scale image-text data, approaches that learn 3D representations directly are blocked by data scarcity (source: §1). The practical alternative is to render the scene into multi-view images and reason with a pretrained 2D VLM — and here the bottleneck appears: the number of visual tokens grows linearly with the number of views (source: §1).

Concretely, with LLaVA-OneVision-7B, a 12-view input produces 729 tokens per image × 12 views = 8,748 visual tokens (source: Appx. §1). This substantially raises LLM inference cost, so reducing the token count becomes a prerequisite for scaling 3D reasoning on top of a 2D VLM (source: §1).

Two Decisive Limitations of Existing Methods

① Voxelization cannot hit an exact budget and saturates. With a fixed voxel size $v_s=0.2\,m$, trying to hit a budget of $B=1342$ underfills 56% of scenes and overfills 44% (source: §3.2, Fig. 2(c)). Moreover, about 31% of ScanQA and SQA3D tokens are spatially redundant, so even shrinking the voxel size leaves retention stuck at about 69%, and heavily redundant scenes stall at 46–57% (source: §3.2, Fig. 2(a)(b)(d)).

② Learned importance under-covers the scene. Attention/feature-based scores are model-dependent and blind to geometric redundancy, so they pour the budget into duplicate tokens from a few regions (source: §1, §2).

The reason exact budget control matters is simple: even if the dataset average stays within limits, a single scene can exceed the memory/latency limit (source: §1).

New Approach: CoVeR (Coverage-Based Token Pruning)

CoVeR is a deterministic, training-free, geometry-only selector. It uses no attention, no encoder features, and no auxiliary encoder — only the tokens’ 3D coordinates $\mathbf{t}_i \in \mathbb{R}^3$ (source: §3.1). This means the selection rule transfers across model architectures (source: Tab. 1).

Problem Formulation: Coverage as a k-center Problem

How well a chosen token set $\mathcal{C}$ ($|\mathcal{C}|=B$) covers the original token set $X$ is measured by the directed Hausdorff distance:

That is, the distance from the “least represented token” to the nearest selected token. A small value guarantees that no region is dropped entirely — a guarantee that learned importance cannot provide (source: §3.1). The goal therefore becomes the discrete Euclidean k-center problem:

Two-Stage Structure

A single hyperparameter $\alpha\in(0,1)$ splits the budget. Stage 1 targets $B_{\text{init}}=\max(1,\lfloor\alpha B\rfloor)$, and Stage 2 fills the remainder $B_{\text{expan}}=B-|\mathcal{C}_{\text{init}}|$. Throughout, $\alpha=0.4$ is fixed (source: §3.3, §4.4).

Stage 1 — Coverage Initialization (adaptive voxelization). A voxel size $v_s$ is found per scene via binary search so that the number of occupied voxels $G(v_s;X)$ falls within $\pm\tau$ ($\tau=0.05$) of $B_{\text{init}}$. One representative token per occupied voxel (the token closest to the within-voxel mean, the Eq. medoid) is kept to produce a coarse cover of the whole scene (source: §3.4, Alg. 1). Since the search is a heuristic, a safeguard keeps representatives from the $B$ most-populated voxels if the result exceeds $B$ (source: §3.3).

Stage 2 — Coverage Expansion (FPS). Voxelization alone cannot fill the budget due to saturation, so Farthest Point Sampling (FPS) iteratively adds the token farthest from the current selection. Using Stage 1’s voxel seeds as the starting point, FPS expands directly into uncovered regions instead of re-exploring already-covered ones (source: §3.5). The distance metric is the squared Euclidean distance $\mathcal{D}=\|\mathbf{t}_i-\mathbf{t}_j\|_2^2$ (source: §3.5).

Theoretical Guarantees

Together, the two stages yield the following (source: §3.6, Appx. theoretical analysis):

1. Exact budget: if $B
2. Coverage upper bound: after Stage 1, every token lies within $\sqrt{3}\,v_s$ of some selected token, and this bound is preserved in Stage 2, so $d_H(X,\mathcal{C})\le\sqrt{3}\,v_s$ (source: Lemma 1, 2, Thm. 1).
3. 2-approximation: FPS guarantees within 2× of the k-center optimum, i.e., $d_H(X,\mathcal{C})\le 2\cdot\text{OPT}_{B_{\text{expan}}}(X)$ (source: Prop. 1).

Empirically, even at 9% retention $d_H$ is only 2.3% of the scene diagonal, far tighter than the Stage 1 upper bound (9.6%) (source: Appx., Fig. A).

How It Works: A Concrete Example

Consider a room with one table, three chairs, and one sofa, photographed by four cameras from different angles.

• Since the table is visible to all four cameras, the tokens in the table region appear as four near-duplicate clusters. In contrast, the wall in the room’s corner is only captured by one camera.
• Learned importance, drawn to the salient table, selects table-adjacent tokens in bulk. As the budget shrinks, only the table remains, and the evidence needed to answer questions like “Is there a mirror in the corner?” (corner tokens) disappears.
• Voxelization cannot reduce tokens beyond deduplicating the table, because no matter how small the voxel size, the four cameras’ observations of the table still overlap in the same voxel (saturation).

CoVeR works as follows:

1. Initialization: a binary search finds the voxel size for which “voxel count ≈ $B_{\text{init}}$”. The table’s four duplicate observations merge into the same voxel, leaving one representative, and the corner wall gets its own representative too. The result is a coarse cover across the whole scene.
2. Expansion: now it fills the remaining $B_{\text{expan}}$ slots. Since FPS fills starting from the “token farthest from the current selection”, it reinforces evidence in the relatively empty corner, floor, and ceiling in turn. The table already has a representative, so it is not reselected.

The key point is that the budget lands exactly on $B$ (what voxelization could not do), the whole scene is evenly represented (what learned importance could not do), and the selection is determined by coordinates alone, not by a model. The authors summarize this as Insight 1: “broader spatial coverage accompanies stronger 3D reasoning” (source: §4.2).

Empirical Validation: Key Results

The evaluation spans three axes of 3D reasoning: spatial understanding (ScanQA), situation reasoning based on agent location (SQA3D), and open-vocabulary embodied QA (OpenEQA) (source: §4, Appx. §1).

Main Benchmark: 93.5% Retained with 8% Tokens

Averaged over the three benchmarks, CoVeR preserves 93.5% of the original performance at 8% token retention. Under the same condition, SeGPruner scores 89.6% and VisPruner 85.9% (source: §4.1, Tab. 2).

On ScanQA, CoVeR records EM@1 27.1, CIDEr 81.4, ROUGE-L 41.5 at a 9% budget, beating VisPruner (23.4/66.9/35.6) and SeGPruner (24.5/71.2/37.0) by a wide margin (source: Tab. 2). SQA3D reaches 48.6 EM@1 at 8%, and OpenEQA 53.0 LLM-Match at 8% (source: Tab. 2). Notably, at 23% retention ScanQA exceeds the full-token baseline (28.2 EM@1) with 28.5 (source: Tab. 2).

Coverage Is What Actually Drives Performance

Rather than raw accuracy, the authors validate “how the budget should be spent” with two metrics (source: §4.2, Tab. 3):

• NNI (Nearest Neighbor Index): how evenly the selected tokens are spread. SeGPruner 0.458 vs CoVeR 0.924 — SeGPruner is clustered in a few regions while CoVeR is uniform.
• NND$_{95}$/NND$_{100}$: scene coverage. NND$_{95}$ is 0.967 vs 0.980, and NND$_{100}$ (worst case) is 0.917 vs 0.977, with the largest gap in the worst case (source: Tab. 3). NND$_{100}$ is the normalized complement of the Hausdorff distance that CoVeR minimizes (source: §4.2).

As for the concern that “doesn’t broad coverage throw away the evidence that decides the answer?”, Token Recovery (TR) is small at 0.009 (0.9% of the scene diagonal), showing CoVeR keeps tokens near every region SeGPruner selected, while Token Expansion (TE) is about 2× larger at 0.020, covering roughly 20% additional area that SeGPruner ignored (source: §4.2, Tab. 4, Fig. 4).

Efficiency: Compute and Memory Savings

With LLaVA-OV-7B, at the most aggressive 9% budget (source: §4.3, Tab. 5):

• TFLOPs reduced 13.3×, KV cache reduced 10.7×, inference accelerated 2.9×, with a performance drop of only 1.1 points.
• At 14% retention: TFLOPs reduced 8.6×, KV cache reduced 7×, GPU memory saved 1.4×, 2.5× speedup, with a 1.1% performance drop (source: §1).

Unlike learned-importance pruners, CoVeR achieves top accuracy and the lowest peak memory (17.2 GB) at the same time. VisPruner/SeGPruner must enable attention outputs and hold attention maps, so their memory is higher at 22.1 GB (source: §4.3, Tab. 6). Pruning itself costs only binary search + FPS, i.e., 0.034 seconds — a fraction of the end-to-end inference time (source: Tab. 6).

Generalization: 4 VLMs, Two Model Families

CoVeR transfers to LLaVA-OV-7B, Video-3D LLM, Qwen2.5-VL-7B, and Qwen3-VL-8B without changing the selection rule or $\alpha$ at all. Even though the latter two differ greatly in visual encoder, tokenizer, and resolution handling, they retain at least 96% of ScanQA performance and 95% of SQA3D at retention rates above 20% (source: §4.3, Fig. 5).

The comparison with Geo3DPruner is particularly striking. Geo3DPruner attaches a VGGT-1B encoder and fully retrains the backbone, while CoVeR, with no retraining, shows an advantage at 16 views / 10% budget: ScanQA EM@1 26.5 vs 26.0 and relative retention 93.5% vs 90.7% (source: §4.3, Tab. 7).

Ablation: Why Each Component Is Needed

• Real tokens vs merged features: pruning that keeps encoder-native tokens as-is beats VTC-style merging that averages features within a voxel, raising ScanQA/SQA3D EM@1 by 7.7/18.1 at the tightest budget. Averaging a voxel that mixes heterogeneous objects produces synthetic features (source: §4.4, Fig. 6, Insight 2).
• Spatial vs semantic distance: using semantic (cosine) distance in the FPS metric suppresses similar objects at different locations. Pure 3D distance beats spatial+semantic by 1.7/2.1 and semantic-only by 2.9/4.3 on ScanQA/SQA3D at tight budgets (source: §4.4, Fig. 7, Insight 3).
• Iterative FPS and the Stage 1 seed: dropping the iterative distance update and using Top-K/random selection costs 0.7/1.7 points at 9%, and removing the Stage 1 seed (pure FPS from scratch) costs 1.1/0.2/0.5 points at 23/14/9%. Thanks to the seed, pruning is also about 1.5× faster (source: §4.4, Tab. 8).

Across all variants, higher coverage metrics (NNI, NND$_{95}$, NND$_{100}$) correlated with higher accuracy, and full CoVeR was best on every metric — evidence that coverage is the mechanism behind performance (source: §4.4, Fig. 8).

Our Take: Strengths, Limitations, and Why This Matters

Strengths

1. Close to a free lunch. With nothing but coordinates — no retraining, no extra encoder, no attention outputs — it achieves up to 13.3× TFLOPs reduction. This contrasts with existing pruners that need per-model tuning (source: §4.3).
2. It cleanly redefined the problem. Seeing “token pruning” as “k-center coverage optimization” rather than “importance ranking” is this paper’s core contribution. This yields both the exact budget guarantee and theoretical upper bounds (√3·$v_s$, 2-approximation) (source: §3.6).
3. Rigorous validation. Instead of accuracy alone, the claims are supported by coverage metrics (NNI/NND), directional distances (TR/TE), transfer across 4 backbones, and ablations that peel away seed/distance/initialization one by one (source: §4.2–4.4).

Limitations

1. Depends on depth and camera pose. CoVeR requires RGB-D and poses and is designed for indoor scenes. The authors themselves note that poor geometry-estimation quality can degrade performance (source: §5).
2. The selection mechanism is completely agnostic to visual importance. The small-TR experiment shows it “doesn’t drop important regions”, but that is a post-hoc check. There is no guarantee that spatial coverage preserves small, rare visual cues that are decisive for the answer (e.g., small text, subtle color differences).
3. The heuristic nature of the voxel search. Because voxel grids do not nest across sizes, monotonicity of $G(v_s;X)$ is not guaranteed, and the binary search is a heuristic. The safeguard absorbs this, but the theoretical upper bound (√3·$v_s$) only holds when the safeguard is inactive (source: §3.3, §3.6, Appx. Remark 2).

Why It Matters

Multi-view 3D reasoning is a key path to robotics and embedded AI, yet its entry barrier was not 3D data scarcity but the inference cost of recycling a 2D VLM. CoVeR offers a simple yet theoretically grounded solution that lowers this bottleneck “with a single coordinate and no training”. At a time when 3D-language data remains scarce, this near-free efficiency gain has substantial practical impact.

What’s Next?: The Road Ahead

The authors propose as future directions coupling with reliable depth/pose estimation and hierarchical/streaming selection for outdoor scenes (source: §5). In addition, the following extensions are natural:

• Robustness under uncertain depth/pose: how selection degrades when coordinates are noisy, or a formulation that reflects coordinate uncertainty in the selection.
• Dynamic scenes and video: extending coverage along the temporal axis (spatiotemporal coverage) beyond static indoor scenes would apply directly to embedded video understanding.
• Generalization to other modalities: the idea of “uniform coverage from coordinates alone” can be carried over to token/point downsampling in point-cloud and LiDAR-based models.

One more thing worth noting: since CoVeR uses neither attention nor features, there is ample room to push pruning itself off the GPU (as a preprocessing step). The authors already showed that “pruning cost is a fraction of end-to-end inference”, but bringing it down to the edge-device level could be the next practical step.

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. Comparison with token pruning methods. Avoiding learned signals (attention, visual features, auxiliary encoders) removes model-specific dependence, while deterministic selection and exact per-scene budget give precise control over token count.

Table 2. Performance Comparison. CoVeR compared to ExpCTextVoxelization and ExpDTextLearned Importance methods on 12-view ScanQA , OpenEQA , and SQA3D . Avg. is over benchmarks, while Rel. is avg. % of performance maintained. Higher is better.

Table 3. Coverage and Performance. CoVeR achieves substantially better spatial coverage with higher downstream 3D scene understanding performance compared to SeGPruner . Higher $\uparrow$ is better on all columns.

Table 4. Directed distance. CoVeR remains close to regions selected by SeGPruner while expanding into additional regions.

Table 5. Efficiency at varying token retention. Pruning' denotes the average time which CoVeR needs to select tokens, while Time’ reports end-to-end inference latency (in sec). CoVeR substantially reduces TFLOPs, KV cache (MB), memory (GB) while remaining competitive across token budgets. Results are relative to LLaVA-OV-7B on ScanQA.

Table 6. Efficiency comparison. CoVeR attains the highest accuracy at the lowest peak memory. Its selection cost remains a fraction of end-to-end inference. Scores report pruning on LLaVA-OV-7B at 9% token retention for ScanQA.

Table 7. Generalization with Video-3D LLM backbone. Geo3DPruner introduces a VGGT-1B encoder and requires full backbone retraining, while CoVeR is training-free. Scores are EM@1 $\uparrow$ on 16-view at 10% budget.

Table 8. Design ablations. Iterative FPS with stage 1 initialization consistently performs best. Results on ScanQA, averaged over EM@1, CIDEr, and ROUGE-L. Pruning time is reported in sec.