Efficient VLM
22 notes in this category
-
Efficient VLM — Overview
Cutting visual tokens to make VLMs cheaper — mapped on two axes (where: encoder·bridge·LLM, and what criterion: importance·diversity·duplication·sensitivity·spatial), the field's evolution, and a one-glance table of 20 methods.
-
[VLM-Pruner] Buffering for Spatial Sparsity in a Centrifugal Token Pruning Paradigm
Importance-only pruning keeps duplicate tokens; redundancy/diversity-aware pruning ignores spatial layout and scatters the retained tokens so they miss object regions. VLM-Pruner balances both with a 'centrifugal' near-to-far selection: start from pivot tokens, expand to spatially adjacent low-redundancy tokens (Buffering for Spatial Sparsity), then recover discarded tokens via similarity-weighted aggregation. Training-free; beats baselines across 5 VLMs at 88.9% pruning.
-
[ZOO-Prune] Training-Free Token Pruning via Zeroth-Order Gradient Estimation in VLMs
Attention scores are unstable and keep redundant tokens; diversity selection can drop the task-relevant region. ZOO-Prune scores tokens by sensitivity — how much a small perturbation changes the model output — estimated with zeroth-order gradients at the lightweight projection layer (no backprop, no labels). It then fuses sensitivity with diversity (Score = Div × Sens). Training-free; prunes up to 94.4% of tokens with no accuracy loss and up to 2.30× faster inference.
-
[CoViPAL] Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
Training-free pruners struggle in shallow layers — there isn't enough context yet to tell which visual tokens are redundant. CoViPAL trains a lightweight, model-agnostic Plug-and-Play Pruning Module (PPM) that predicts and removes redundant vision tokens before the LVLM sees them, using contextual signals so pruning works even in shallow layers. Outperforms training-free pruners at equal budgets and training-based ones at comparable supervision.
-
[CDPruner] Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
Attention-based pruning keeps duplicates; similarity-based pruning ignores the instruction. CDPruner does both right: it defines visual-token similarity conditioned on the instruction, then uses a Determinantal Point Process (DPP) to pick the subset that maximizes conditional diversity — diverse AND question-relevant. Training-free, model-agnostic; on LLaVA, 95% fewer FLOPs and 78% lower latency at 94% accuracy.
-
[TopV] Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory VLM
Most LLM-side pruners rank visual tokens by attention score — which is a greedy heuristic AND breaks FlashAttention and the KV cache. TopV instead formulates pruning as an optimization problem with a vision-aware cost (feature similarity + spatial + central distance), solves it with Sinkhorn, and prunes once at prefilling — staying compatible with FlashAttention and KV cache. Training-free.
-
[DivPrune] Diversity-based Visual Token Pruning for Large Multimodal Models
Reframes visual token pruning as a Max-Min Diversity Problem — pick the subset of visual tokens whose mutual diversity is maximal (instead of keeping the 'most important', which are often redundant). Training-free, done once before the first LLM layer; SOTA across 16 image/video datasets with lower latency and memory.
-
[DART] Stop Looking for "Important Tokens": Duplication Matters More
A contrarian take: token importance is a bad pruning criterion — importance-based methods often do worse than random pruning and break efficient attention. DART (Duplication-Aware Reduction of Tokens) instead picks a tiny set of pivot tokens and keeps the tokens least duplicated with them. Training-free, Flash-Attention-friendly; prunes 88.9% of vision tokens with comparable accuracy and 2.99× faster prefilling.
-
[G-Prune] What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning from the Perspective of Graph
ViT pruning keeps only 'important' foreground tokens — but for MLLMs, G-Prune shows both foreground AND background tokens matter (their 2-norm distributions overlap). It builds a graph over visual tokens (nodes = tokens, edges = semantic similarity), propagates information iteratively to score importance, and keeps the top-k — which can be foreground or background. Training-free, not bound to the LLM's computation; −63.57% FLOPs on LLaVA-NeXT with <2.5% drop.
-
[FastVLM] Efficient Vision Encoding for Vision Language Models
Instead of pruning tokens after the encoder, FastVLM fixes the encoder itself. FastViTHD — a hybrid (conv + transformer) vision encoder — outputs far fewer tokens and encodes high-resolution images much faster, so the right token-count/resolution balance comes simply from scaling the input image, no token pruning needed. 3.2× faster time-to-first-token at similar accuracy.
-
[VisionZip] Longer is Better but Not Necessary in Vision Language Models
Vision encoders (CLIP/SigLIP) emit highly redundant visual tokens — VisionZip keeps only a few dominant tokens (high attention) plus merged contextual tokens, text-agnostic and training-free. 8× faster prefilling at 95% performance; shines in multi-turn dialogue where text-guided pruners fail.
-
[ATP-LLaVA] Adaptive Token Pruning for Large Vision Language Models
Fixed pruning ratios are suboptimal — the right amount to cut varies by LLM layer and by instance (image-prompt). ATP-LLaVA learns an Adaptive Token Pruning module that sets an instance- and layer-specific ratio, plus a Spatial Augmented Pruning strategy. Cuts ~75% of tokens with only 1.9% drop.
-
[PyramidDrop] Accelerating Large Vision-Language Models via Pyramid Visual Redundancy Reduction
Empirically shows visual tokens are all needed in shallow LLM layers but grow redundant in deeper ones — so it splits the LLM into stages and drops a fixed ratio of image tokens at the end of each stage (pyramid). Accelerates both training (−40%) and inference (−55% FLOPs).
-
[SparseVLM] Visual Token Sparsification for Efficient Vision-Language Model Inference
Training-free, text-guided visual token sparsification inside the LLM — relevant text tokens act as 'raters' to score visual tokens via self-attention, a rank-based rule sets the per-layer ratio, and pruned tokens are recycled by clustering. 4.5× compression keeping 97% on LLaVA; beats FastV.
-
[IVTP] Instruction-guided Visual Token Pruning for Large Vision-Language Models
Two-stage visual token pruning for LVLMs — a Group-wise Token Pruning (attention rollout) inside the frozen ViT, then an instruction-guided filter inside the LLM using a pseudo CLS token. Training-free: cuts 88.9% of visual tokens (FLOPs −46%) with only ~1% drop on LLaVA-1.5.
-
[VLTP] Vision-Language Guided Token Pruning for Task-Oriented Segmentation
Accelerates ViT-based segmentation by pruning image tokens that aren't relevant to the task — a prune decoder uses MLLM guidance to score each token's task-relevance, keeping only relevant tokens in deeper ViT layers. ~25% ViT FLOPs cut with no drop (40% with 1%).
-
[Recoverable Compression] A Multimodal Vision Token Recovery Mechanism Guided by Text Information
Beyond pruning: after an initial CLS-based filter, it recovers visual tokens that are similar to the question text (reclaiming wrongly-dropped but answer-relevant ones) and merges the rest — training-free, compressing visual tokens to ~10% with competitive accuracy on LLaVA-1.5.
-
[CrossGET] Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers
Reduces tokens inside vision-language Transformers by ensembling (merging) them, guided by cross-modal importance — works on both modality-independent (CLIP) and modality-dependent (BLIP-2) models via learnable cross tokens and a parallelizable complete-graph soft matching.
-
[LLaVA-PruMerge] Adaptive Token Reduction for Efficient Large Multimodal Models
The CLS-to-patch attention in the vision encoder is sparse — only a few visual tokens matter. PruMerge exploits this: it uses IQR outlier detection to adaptively keep the important tokens (more on text-rich images, fewer on simple ones), then merges the rest into them via k-NN weighted averaging. Training-free; ~5.5% of tokens (~32 of 576) keeps LLaVA-1.5 performance.
-
[FastV] An Image is Worth 1/2 Tokens After Layer 2
Observes that in deep LLM layers of LVLMs, visual tokens receive almost no attention — so after an early layer (e.g., layer 2) it prunes low-attention visual tokens. Training-free plug-and-play: 45% FLOPs cut on LLaVA-1.5-13B with no performance loss.
-
[MADTP] Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
Prunes tokens inside vision-language Transformers, but guided by cross-modal alignment (MAG) so a token isn't cut in one branch while still vital in the other — plus per-layer, per-instance dynamic ratios (DTP). 80% fewer GFLOPs on BLIP/NLVR2 with <4% drop.
-
[ToMe] Token Merging: Your ViT But Faster
Combine similar tokens (not prune) via bipartite soft matching, fast as pruning, works even without training.
