Efficient VLM (22)

  • Efficient VLM — Overview

    Cutting visual tokens to make VLMs cheaper — mapped on two axes (where: encoder·bridge·LLM, and what criterion: importance·diversity·duplication·sensitivity·spatial), the field's evolution, and a one-glance table of 20 methods.

    Overview

    image

Key papers

SparseVLM

LLM ICML 2025

Training-free, text-guided visual token sparsification inside the LLM — relevant text tokens act as 'raters' to score visual tokens via self-attention, a rank-based rule sets the per-layer ratio, and pruned tokens are recycled by clustering. 4.5× compression keeping 97% on LLaVA; beats FastV.

ZOO-Prune

Bridge CVPR 2026

Attention scores are unstable and keep redundant tokens; diversity selection can drop the task-relevant region. ZOO-Prune scores tokens by sensitivity — how much a small perturbation changes the model output — estimated with zeroth-order gradients at the lightweight projection layer (no backprop, no labels). It then fuses sensitivity with diversity (Score = Div × Sens). Training-free; prunes up to 94.4% of tokens with no accuracy loss and up to 2.30× faster inference.

View all 22 Efficient VLM notes →

VLM (24)

  • VLM — Overview

    How vision-language / multimodal LLMs are built — the 5-component architecture (encoder · projector · LLM · output projector · generator), the understanding-vs-generation taxonomy, and key models from Flamingo to Qwen2.5-VL.

    Overview

    image

Key papers

LLaVA-1.5

VLM CVPR 2024

A systematic study of LLaVA's design choices — an MLP connector, a 336px CLIP encoder, and academic-task VQA data with response-format prompts — sets SOTA on 11 benchmarks with only public data.

InternVL

VLM CVPR 2024

Argues the vision encoder is too small next to the LLM, so it scales the encoder to 6B (InternViT-6B) and bridges it with an 8B language middleware (QLLaMA) via progressive contrastive→generative alignment.

Qwen2-VL

VLM arXiv 2024

Upgrades Qwen-VL with Naive Dynamic Resolution (any resolution → variable visual tokens) and Multimodal RoPE (M-RoPE: temporal·height·width), unifying image and video under one spatiotemporal position scheme — scaled to 2B/7B/72B.

View all 24 VLM notes →

Token Reduction in ViTs (18)

Key papers

DynamicViT

Pruning NeurIPS 2021

Dynamically drops redundant tokens per input to speed up ViTs.

EViT

Pruning ICLR 2022

Keep top-k attentive tokens by CLS attention, fuse the rest into one.

ToMe

MergingEncoder ICLR 2023

Combine similar tokens (not prune) via bipartite soft matching, fast as pruning, works even without training.

View all 18 Token Reduction in ViTs notes →