VLM
24 notes in this category
-
VLM — Overview
How vision-language / multimodal LLMs are built — the 5-component architecture (encoder · projector · LLM · output projector · generator), the understanding-vs-generation taxonomy, and key models from Flamingo to Qwen2.5-VL.
-
[Qwen3-VL] Technical Report
The most capable Qwen VLM yet — native 256K interleaved context, dense (2/4/8/32B) and MoE (30B-A3B/235B-A22B) variants, and three architecture upgrades: interleaved-MRoPE, DeepStack multi-level ViT fusion, and text-based timestamp alignment for video.
-
[InternVL3.5] Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
Builds on InternVL3 with Cascade RL (offline→online) for reasoning, a Visual Resolution Router (ViR) and Decoupled Vision-Language Deployment (DvD) for efficiency — +16% reasoning and 4.05× faster inference, plus GUI/embodied agency.
-
[InternVL3] Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
Brings native multimodal pre-training to the open InternVL line — jointly learning language and vision in a single stage instead of retrofitting a text-only LLM — plus V2PE, mixed preference optimization (MPO), and test-time scaling. InternVL3-78B sets a new open MMMU SOTA (72.2).
-
[Qwen2.5-VL] Technical Report
Refines Qwen2-VL: a from-scratch native-resolution ViT with window attention (linear cost), dynamic FPS sampling, and MRoPE aligned to absolute time — pushing document parsing, object grounding, hour-long video, and computer/mobile agents.
-
[DeepSeek-VL2] Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
Upgrades DeepSeek-VL with a dynamic tiling vision encoder (single SigLIP, any aspect ratio) and a DeepSeekMoE LLM with Multi-head Latent Attention — matching or beating dense/MoE models with only 1.0/2.8/4.5B activated parameters.
-
[InternVL 2.5] Expanding Performance Boundaries with Model, Data, and Test-Time Scaling
Keeps InternVL 2.0's ViT-MLP-LLM architecture but pushes three scaling axes — model (6B vision encoder), data (doubled + strictly filtered), and test-time (CoT + voting) — becoming the first open MLLM to pass 70% on MMMU.
-
[Qwen2-VL] Enhancing Vision-Language Model's Perception of the World at Any Resolution
Upgrades Qwen-VL with Naive Dynamic Resolution (any resolution → variable visual tokens) and Multimodal RoPE (M-RoPE: temporal·height·width), unifying image and video under one spatiotemporal position scheme — scaled to 2B/7B/72B.
-
[LLaVA-OneVision] Easy Visual Task Transfer
The first single open LMM strong across single-image, multi-image, and video — with cross-scenario task transfer (video understanding emerges from image training) via a balanced AnyRes token budget.
-
[InternVL] Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Argues the vision encoder is too small next to the LLM, so it scales the encoder to 6B (InternViT-6B) and bridges it with an 8B language middleware (QLLaMA) via progressive contrastive→generative alignment.
-
[Gemini] A Family of Highly Capable Multimodal Models
Google's frontier family (Ultra/Pro/Nano) trained jointly across image, audio, video, and text from the beginning — natively multimodal rather than bolting a vision encoder onto a pretrained LLM.
-
[LLaVA-1.5] Improved Baselines with Visual Instruction Tuning
A systematic study of LLaVA's design choices — an MLP connector, a 336px CLIP encoder, and academic-task VQA data with response-format prompts — sets SOTA on 11 benchmarks with only public data.
-
[Qwen-VL] A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Adds vision to Qwen-7B via a ViT + position-aware cross-attention adapter (256 query tokens), trained in 3 stages — and adds grounding and text-reading via box/ref special tokens.
-
[P-Former] Bootstrapping Vision-Language Learning with Decoupled Language Pre-training
Flips the usual connector training: instead of asking 'which visual features make a good prompt', P-Former first learns — from text only — the ideal 'reference prompt' a frozen LLM needs, then aligns visual features to it. A training-only module that boosts BLIP-2's data efficiency.
-
[InstructBLIP] Towards General-purpose Vision-Language Models with Instruction Tuning
Instruction-tunes BLIP-2 on 26 datasets, and makes the Q-Former instruction-aware — feeding the instruction to the Q-Former so it extracts visual features tailored to the task.
-
[MiniGPT-4] Enhancing Vision-Language Understanding with Advanced Large Language Models
Aligns a frozen vision encoder (BLIP-2's ViT+Q-Former) and a frozen Vicuna with a single linear projection layer — showing GPT-4-like abilities emerge from minimal alignment of a strong LLM.
-
[LLaVA] Visual Instruction Tuning
Uses language-only GPT-4 to generate multimodal instruction-following data, then connects a frozen CLIP encoder to Vicuna with a single linear projection and instruction-tunes end-to-end.
-
[EVA-CLIP] Improved Training Techniques for CLIP at Scale
Makes CLIP training cheaper and more stable at scale — EVA initialization, the LAMB optimizer, random token dropping, and flash attention — reaching 82.0% zero-shot ImageNet with far fewer samples. The vision encoder many VLMs (e.g., BLIP-2) reuse.
-
[SigLIP] Sigmoid Loss for Language Image Pre-Training
Replaces CLIP's batch-wise softmax contrastive loss with a simple pairwise sigmoid loss — each image-text pair is an independent binary classification, so no global normalization (no all-gather, no full N×N matrix). Memory-efficient, batch-size-decoupled, strong at small and huge batches. SigLIP-SO400M became a default VLM vision encoder.
-
[Kosmos-1] Language Is Not All You Need: Aligning Perception with Language Models
A multimodal LLM trained from scratch on web-scale interleaved image-text — perceiving general modalities, learning in context (few-shot), and following instructions (zero-shot), without a frozen LLM.
-
[BLIP-2] Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Bridges a frozen image encoder and a frozen LLM with a lightweight Querying Transformer (Q-Former), pre-trained in two stages — representation learning then generative learning.
-
[Flamingo] Flamingo: a Visual Language Model for Few-Shot Learning
Bridges a frozen vision encoder and a frozen LLM with a Perceiver Resampler + gated cross-attention, unlocking GPT-3-style few-shot in-context learning on image/video tasks.
-
[BLIP] Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
A unified vision-language model (MED) that handles both understanding and generation, plus CapFilt — a captioner+filter that bootstraps noisy web captions into cleaner training data.
-
[CLIP] Learning Transferable Visual Models From Natural Language Supervision
Trains an image encoder and a text encoder to match images with their captions (contrastive) on 400M web pairs — enabling open-vocabulary zero-shot transfer, and becoming the vision encoder most VLMs freeze and reuse.
