bridge
7 notes tagged “bridge”
-
[ZOO-Prune] Training-Free Token Pruning via Zeroth-Order Gradient Estimation in VLMs
Attention scores are unstable and keep redundant tokens; diversity selection can drop the task-relevant region. ZOO-Prune scores tokens by sensitivity — how much a small perturbation changes the model output — estimated with zeroth-order gradients at the lightweight projection layer (no backprop, no labels). It then fuses sensitivity with diversity (Score = Div × Sens). Training-free; prunes up to 94.4% of tokens with no accuracy loss and up to 2.30× faster inference.
-
[CoViPAL] Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
Training-free pruners struggle in shallow layers — there isn't enough context yet to tell which visual tokens are redundant. CoViPAL trains a lightweight, model-agnostic Plug-and-Play Pruning Module (PPM) that predicts and removes redundant vision tokens before the LVLM sees them, using contextual signals so pruning works even in shallow layers. Outperforms training-free pruners at equal budgets and training-based ones at comparable supervision.
-
[CDPruner] Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
Attention-based pruning keeps duplicates; similarity-based pruning ignores the instruction. CDPruner does both right: it defines visual-token similarity conditioned on the instruction, then uses a Determinantal Point Process (DPP) to pick the subset that maximizes conditional diversity — diverse AND question-relevant. Training-free, model-agnostic; on LLaVA, 95% fewer FLOPs and 78% lower latency at 94% accuracy.
-
[DivPrune] Diversity-based Visual Token Pruning for Large Multimodal Models
Reframes visual token pruning as a Max-Min Diversity Problem — pick the subset of visual tokens whose mutual diversity is maximal (instead of keeping the 'most important', which are often redundant). Training-free, done once before the first LLM layer; SOTA across 16 image/video datasets with lower latency and memory.
-
[G-Prune] What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning from the Perspective of Graph
ViT pruning keeps only 'important' foreground tokens — but for MLLMs, G-Prune shows both foreground AND background tokens matter (their 2-norm distributions overlap). It builds a graph over visual tokens (nodes = tokens, edges = semantic similarity), propagates information iteratively to score importance, and keeps the top-k — which can be foreground or background. Training-free, not bound to the LLM's computation; −63.57% FLOPs on LLaVA-NeXT with <2.5% drop.
-
[Recoverable Compression] A Multimodal Vision Token Recovery Mechanism Guided by Text Information
Beyond pruning: after an initial CLS-based filter, it recovers visual tokens that are similar to the question text (reclaiming wrongly-dropped but answer-relevant ones) and merges the rest — training-free, compressing visual tokens to ~10% with competitive accuracy on LLaVA-1.5.
-
[LLaVA-PruMerge] Adaptive Token Reduction for Efficient Large Multimodal Models
The CLS-to-patch attention in the vision encoder is sparse — only a few visual tokens matter. PruMerge exploits this: it uses IQR outlier detection to adaptively keep the important tokens (more on text-rich images, fewer on simple ones), then merges the rest into them via k-NN weighted averaging. Training-free; ~5.5% of tokens (~32 of 576) keeps LLaVA-1.5 performance.
