2024
16 notes from 2024
-
[FastVLM] Efficient Vision Encoding for Vision Language Models
Instead of pruning tokens after the encoder, FastVLM fixes the encoder itself. FastViTHD — a hybrid (conv + transformer) vision encoder — outputs far fewer tokens and encodes high-resolution images much faster, so the right token-count/resolution balance comes simply from scaling the input image, no token pruning needed. 3.2× faster time-to-first-token at similar accuracy.
-
[DeepSeek-VL2] Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
Upgrades DeepSeek-VL with a dynamic tiling vision encoder (single SigLIP, any aspect ratio) and a DeepSeekMoE LLM with Multi-head Latent Attention — matching or beating dense/MoE models with only 1.0/2.8/4.5B activated parameters.
-
[InternVL 2.5] Expanding Performance Boundaries with Model, Data, and Test-Time Scaling
Keeps InternVL 2.0's ViT-MLP-LLM architecture but pushes three scaling axes — model (6B vision encoder), data (doubled + strictly filtered), and test-time (CoT + voting) — becoming the first open MLLM to pass 70% on MMMU.
-
[VisionZip] Longer is Better but Not Necessary in Vision Language Models
Vision encoders (CLIP/SigLIP) emit highly redundant visual tokens — VisionZip keeps only a few dominant tokens (high attention) plus merged contextual tokens, text-agnostic and training-free. 8× faster prefilling at 95% performance; shines in multi-turn dialogue where text-guided pruners fail.
-
[ATP-LLaVA] Adaptive Token Pruning for Large Vision Language Models
Fixed pruning ratios are suboptimal — the right amount to cut varies by LLM layer and by instance (image-prompt). ATP-LLaVA learns an Adaptive Token Pruning module that sets an instance- and layer-specific ratio, plus a Spatial Augmented Pruning strategy. Cuts ~75% of tokens with only 1.9% drop.
-
[PyramidDrop] Accelerating Large Vision-Language Models via Pyramid Visual Redundancy Reduction
Empirically shows visual tokens are all needed in shallow LLM layers but grow redundant in deeper ones — so it splits the LLM into stages and drops a fixed ratio of image tokens at the end of each stage (pyramid). Accelerates both training (−40%) and inference (−55% FLOPs).
-
[SparseVLM] Visual Token Sparsification for Efficient Vision-Language Model Inference
Training-free, text-guided visual token sparsification inside the LLM — relevant text tokens act as 'raters' to score visual tokens via self-attention, a rank-based rule sets the per-layer ratio, and pruned tokens are recycled by clustering. 4.5× compression keeping 97% on LLaVA; beats FastV.
-
[IVTP] Instruction-guided Visual Token Pruning for Large Vision-Language Models
Two-stage visual token pruning for LVLMs — a Group-wise Token Pruning (attention rollout) inside the frozen ViT, then an instruction-guided filter inside the LLM using a pseudo CLS token. Training-free: cuts 88.9% of visual tokens (FLOPs −46%) with only ~1% drop on LLaVA-1.5.
-
[Qwen2-VL] Enhancing Vision-Language Model's Perception of the World at Any Resolution
Upgrades Qwen-VL with Naive Dynamic Resolution (any resolution → variable visual tokens) and Multimodal RoPE (M-RoPE: temporal·height·width), unifying image and video under one spatiotemporal position scheme — scaled to 2B/7B/72B.
-
[VLTP] Vision-Language Guided Token Pruning for Task-Oriented Segmentation
Accelerates ViT-based segmentation by pruning image tokens that aren't relevant to the task — a prune decoder uses MLLM guidance to score each token's task-relevance, keeping only relevant tokens in deeper ViT layers. ~25% ViT FLOPs cut with no drop (40% with 1%).
-
[Recoverable Compression] A Multimodal Vision Token Recovery Mechanism Guided by Text Information
Beyond pruning: after an initial CLS-based filter, it recovers visual tokens that are similar to the question text (reclaiming wrongly-dropped but answer-relevant ones) and merges the rest — training-free, compressing visual tokens to ~10% with competitive accuracy on LLaVA-1.5.
-
[LLaVA-OneVision] Easy Visual Task Transfer
The first single open LMM strong across single-image, multi-image, and video — with cross-scenario task transfer (video understanding emerges from image training) via a balanced AnyRes token budget.
-
[CrossGET] Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers
Reduces tokens inside vision-language Transformers by ensembling (merging) them, guided by cross-modal importance — works on both modality-independent (CLIP) and modality-dependent (BLIP-2) models via learnable cross tokens and a parallelizable complete-graph soft matching.
-
[LLaVA-PruMerge] Adaptive Token Reduction for Efficient Large Multimodal Models
The CLS-to-patch attention in the vision encoder is sparse — only a few visual tokens matter. PruMerge exploits this: it uses IQR outlier detection to adaptively keep the important tokens (more on text-rich images, fewer on simple ones), then merges the rest into them via k-NN weighted averaging. Training-free; ~5.5% of tokens (~32 of 576) keeps LLaVA-1.5 performance.
-
[FastV] An Image is Worth 1/2 Tokens After Layer 2
Observes that in deep LLM layers of LVLMs, visual tokens receive almost no attention — so after an early layer (e.g., layer 2) it prunes low-attention visual tokens. Training-free plug-and-play: 45% FLOPs cut on LLaVA-1.5-13B with no performance loss.
-
[MADTP] Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
Prunes tokens inside vision-language Transformers, but guided by cross-modal alignment (MAG) so a token isn't cut in one branch while still vital in the other — plus per-layer, per-instance dynamic ratios (DTP). 80% fewer GFLOPs on BLIP/NLVR2 with <4% drop.
