modality-encoder
3 notes tagged “modality-encoder”
-
[EVA-CLIP] Improved Training Techniques for CLIP at Scale
Makes CLIP training cheaper and more stable at scale — EVA initialization, the LAMB optimizer, random token dropping, and flash attention — reaching 82.0% zero-shot ImageNet with far fewer samples. The vision encoder many VLMs (e.g., BLIP-2) reuse.
-
[SigLIP] Sigmoid Loss for Language Image Pre-Training
Replaces CLIP's batch-wise softmax contrastive loss with a simple pairwise sigmoid loss — each image-text pair is an independent binary classification, so no global normalization (no all-gather, no full N×N matrix). Memory-efficient, batch-size-decoupled, strong at small and huge batches. SigLIP-SO400M became a default VLM vision encoder.
-
[CLIP] Learning Transferable Visual Models From Natural Language Supervision
Trains an image encoder and a text encoder to match images with their captions (contrastive) on 400M web pairs — enabling open-vocabulary zero-shot transfer, and becoming the vision encoder most VLMs freeze and reuse.
