2023
24 notes from 2023
-
[InternVL] Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Argues the vision encoder is too small next to the LLM, so it scales the encoder to 6B (InternViT-6B) and bridges it with an 8B language middleware (QLLaMA) via progressive contrastive→generative alignment.
-
[Gemini] A Family of Highly Capable Multimodal Models
Google's frontier family (Ultra/Pro/Nano) trained jointly across image, audio, video, and text from the beginning — natively multimodal rather than bolting a vision encoder onto a pretrained LLM.
-
[Token Cropr] Token Cropr: Faster ViTs for Quite a Few Tasks
Prunes tokens by task relevance using auxiliary cross-attention heads that are thrown away after training, plus Last Layer Fusion to revive pruned tokens for dense tasks.
-
[Frequency-Aware TR] Frequency-Aware Token Reduction for Efficient Vision Transformer
Reads token reduction through a frequency lens: keeps high-frequency tokens (which fight rank collapse) and squeezes the low-frequency rest into a compact DC token.
-
[LLaVA-1.5] Improved Baselines with Visual Instruction Tuning
A systematic study of LLaVA's design choices — an MLP connector, a 336px CLIP encoder, and academic-task VQA data with response-format prompts — sets SOTA on 11 benchmarks with only public data.
-
[MCTF] Multi-criteria Token Fusion with One-step-ahead Attention for Efficient Vision Transformers
Fuses tokens by a product of three criteria — similarity, informativeness, size — with one-step-ahead attention and bidirectional bipartite matching, beating the base model while cutting FLOPs.
-
[STAR] Synergistic Patch Pruning for Vision Transformer: Unifying Intra- & Inter-Layer Patch Importance
Fuses online intra-layer [CLS] attention with offline inter-layer LRP importance, and auto-tunes per-layer retention rates from patch similarity.
-
[Qwen-VL] A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Adds vision to Qwen-7B via a ViT + position-aware cross-attention adapter (256 query tokens), trained in 3 stages — and adds grounding and text-reading via box/ref special tokens.
-
[Token Fusion / ToFu] Bridging the Gap between Token Pruning and Token Merging
Switches between pruning (early layers) and merging (later layers) by each layer's functional linearity, with a norm-preserving MLERP merge.
-
[P-Former] Bootstrapping Vision-Language Learning with Decoupled Language Pre-training
Flips the usual connector training: instead of asking 'which visual features make a good prompt', P-Former first learns — from text only — the ideal 'reference prompt' a frozen LLM needs, then aligns visual features to it. A training-only module that boosts BLIP-2's data efficiency.
-
[DTEM] Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers
Learns a lightweight embedding dedicated to merging — decoupled from the ViT forward pass — via a continuously relaxed (differentiable) token merging.
-
[Zero-TPrune] Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers
Treats the attention matrix as a directed graph and ranks tokens with a Weighted PageRank — pruning without any fine-tuning.
-
[InstructBLIP] Towards General-purpose Vision-Language Models with Instruction Tuning
Instruction-tunes BLIP-2 on 26 datasets, and makes the Q-Former instruction-aware — feeding the instruction to the Q-Former so it extracts visual features tailored to the task.
-
[Token Pooling] Token Pooling in Vision Transformers
Reframes token downsampling as minimizing reconstruction error, and solves it with simple, parameter-free clustering (K-Means / K-Medoids).
-
[MiniGPT-4] Enhancing Vision-Language Understanding with Advanced Large Language Models
Aligns a frozen vision encoder (BLIP-2's ViT+Q-Former) and a frozen Vicuna with a single linear projection layer — showing GPT-4-like abilities emerge from minimal alignment of a strong LLM.
-
[LLaVA] Visual Instruction Tuning
Uses language-only GPT-4 to generate multimodal instruction-following data, then connects a frozen CLIP encoder to Vicuna with a single linear projection and instruction-tunes end-to-end.
-
[TPS] Joint Token Pruning & Squeezing Towards More Aggressive Compression of Vision Transformers
Instead of throwing pruned tokens away, squeezes their information into the surviving 'host' tokens — parameter-free matching + similarity-based fusing.
-
[EVA-CLIP] Improved Training Techniques for CLIP at Scale
Makes CLIP training cheaper and more stable at scale — EVA initialization, the LAMB optimizer, random token dropping, and flash attention — reaching 82.0% zero-shot ImageNet with far fewer samples. The vision encoder many VLMs (e.g., BLIP-2) reuse.
-
[SigLIP] Sigmoid Loss for Language Image Pre-Training
Replaces CLIP's batch-wise softmax contrastive loss with a simple pairwise sigmoid loss — each image-text pair is an independent binary classification, so no global normalization (no all-gather, no full N×N matrix). Memory-efficient, batch-size-decoupled, strong at small and huge batches. SigLIP-SO400M became a default VLM vision encoder.
-
[DiffRate] Differentiable Compression Rate for Efficient Vision Transformers
Makes the per-layer compression rate differentiable, and prunes + merges in one unified framework.
-
[Kosmos-1] Language Is Not All You Need: Aligning Perception with Language Models
A multimodal LLM trained from scratch on web-scale interleaved image-text — perceiving general modalities, learning in context (few-shot), and following instructions (zero-shot), without a frozen LLM.
-
[ToMe] Token Merging: Your ViT But Faster
Combine similar tokens (not prune) via bipartite soft matching, fast as pruning, works even without training.
-
[BLIP-2] Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Bridges a frozen image encoder and a frozen LLM with a lightweight Querying Transformer (Q-Former), pre-trained in two stages — representation learning then generative learning.
-
[AS-ViT] Adaptive Sparse ViT: Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
Learnable thresholds replace fixed keep-ratios, scoring tokens for free from MHSA's own intermediate results.
