2025
12 notes from 2025
-
[VLM-Pruner] Buffering for Spatial Sparsity in a Centrifugal Token Pruning Paradigm
Importance-only pruning keeps duplicate tokens; redundancy/diversity-aware pruning ignores spatial layout and scatters the retained tokens so they miss object regions. VLM-Pruner balances both with a 'centrifugal' near-to-far selection: start from pivot tokens, expand to spatially adjacent low-redundancy tokens (Buffering for Spatial Sparsity), then recover discarded tokens via similarity-weighted aggregation. Training-free; beats baselines across 5 VLMs at 88.9% pruning.
-
[Qwen3-VL] Technical Report
The most capable Qwen VLM yet — native 256K interleaved context, dense (2/4/8/32B) and MoE (30B-A3B/235B-A22B) variants, and three architecture upgrades: interleaved-MRoPE, DeepStack multi-level ViT fusion, and text-based timestamp alignment for video.
-
[ZOO-Prune] Training-Free Token Pruning via Zeroth-Order Gradient Estimation in VLMs
Attention scores are unstable and keep redundant tokens; diversity selection can drop the task-relevant region. ZOO-Prune scores tokens by sensitivity — how much a small perturbation changes the model output — estimated with zeroth-order gradients at the lightweight projection layer (no backprop, no labels). It then fuses sensitivity with diversity (Score = Div × Sens). Training-free; prunes up to 94.4% of tokens with no accuracy loss and up to 2.30× faster inference.
-
[InternVL3.5] Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
Builds on InternVL3 with Cascade RL (offline→online) for reasoning, a Visual Resolution Router (ViR) and Decoupled Vision-Language Deployment (DvD) for efficiency — +16% reasoning and 4.05× faster inference, plus GUI/embodied agency.
-
[CoViPAL] Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
Training-free pruners struggle in shallow layers — there isn't enough context yet to tell which visual tokens are redundant. CoViPAL trains a lightweight, model-agnostic Plug-and-Play Pruning Module (PPM) that predicts and removes redundant vision tokens before the LVLM sees them, using contextual signals so pruning works even in shallow layers. Outperforms training-free pruners at equal budgets and training-based ones at comparable supervision.
-
[CDPruner] Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
Attention-based pruning keeps duplicates; similarity-based pruning ignores the instruction. CDPruner does both right: it defines visual-token similarity conditioned on the instruction, then uses a Determinantal Point Process (DPP) to pick the subset that maximizes conditional diversity — diverse AND question-relevant. Training-free, model-agnostic; on LLaVA, 95% fewer FLOPs and 78% lower latency at 94% accuracy.
-
[InternVL3] Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
Brings native multimodal pre-training to the open InternVL line — jointly learning language and vision in a single stage instead of retrofitting a text-only LLM — plus V2PE, mixed preference optimization (MPO), and test-time scaling. InternVL3-78B sets a new open MMMU SOTA (72.2).
-
[TopV] Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory VLM
Most LLM-side pruners rank visual tokens by attention score — which is a greedy heuristic AND breaks FlashAttention and the KV cache. TopV instead formulates pruning as an optimization problem with a vision-aware cost (feature similarity + spatial + central distance), solves it with Sinkhorn, and prunes once at prefilling — staying compatible with FlashAttention and KV cache. Training-free.
-
[DivPrune] Diversity-based Visual Token Pruning for Large Multimodal Models
Reframes visual token pruning as a Max-Min Diversity Problem — pick the subset of visual tokens whose mutual diversity is maximal (instead of keeping the 'most important', which are often redundant). Training-free, done once before the first LLM layer; SOTA across 16 image/video datasets with lower latency and memory.
-
[Qwen2.5-VL] Technical Report
Refines Qwen2-VL: a from-scratch native-resolution ViT with window attention (linear cost), dynamic FPS sampling, and MRoPE aligned to absolute time — pushing document parsing, object grounding, hour-long video, and computer/mobile agents.
-
[DART] Stop Looking for "Important Tokens": Duplication Matters More
A contrarian take: token importance is a bad pruning criterion — importance-based methods often do worse than random pruning and break efficient attention. DART (Duplication-Aware Reduction of Tokens) instead picks a tiny set of pivot tokens and keeps the tokens least duplicated with them. Training-free, Flash-Attention-friendly; prunes 88.9% of vision tokens with comparable accuracy and 2.99× faster prefilling.
-
[G-Prune] What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning from the Perspective of Graph
ViT pruning keeps only 'important' foreground tokens — but for MLLMs, G-Prune shows both foreground AND background tokens matter (their 2-norm distributions overlap). It builds a graph over visual tokens (nodes = tokens, edges = semantic similarity), propagates information iteratively to score importance, and keeps the top-k — which can be foreground or background. Training-free, not bound to the LLM's computation; −63.57% FLOPs on LLaVA-NeXT with <2.5% drop.
