OpenVision - A family of open-source visual encoders from the University of California

What is OpenVision?

OpenVision is a fully open, efficient, and flexible family of advanced visual encoders from the University of California, Santa Cruz (UCSC), focusing on multimodal learning. It offers models of various sizes, from 5.9M to 632.1M parameters, suitable for diverse scenarios ranging from edge devices to high-performance servers. OpenVision employs a progressive multi-stage resolution training strategy, achieving training efficiency 2 to 3 times faster than comparable proprietary models. It performs exceptionally well in multimodal benchmarks, comparable to OpenAI's CLIP and SigLIP models. OpenVision supports variable-size patches of 8×8 and 16×16, flexibly addressing the needs for detailed visual understanding or efficient processing.

Main functions of OpenVision

• Fully openThe dataset, training recipes, and model checkpoints are all publicly available and open-source under the Apache 2.0 license, promoting reproducibility and transparency in multimodal research.
• Models of varying sizesIt offers visual encoders with parameters ranging from 5.9M to 632.1M, with a total of 26 different models, covering a wide range of deployment needs from edge devices to high-performance servers.
• Excellent performanceIn multimodal benchmarks, OpenVision performs on par with, and in some cases surpasses, proprietary visual encoders such as OpenAI’s CLIP and SigLIP.
• High training efficiencyThrough a progressive multi-stage resolution training strategy, OpenVision is 2 to 3 times faster than its proprietary competitors in training efficiency.
• Flexible configurationSupports variable-size patches of 8×8 and 16×16, and supports detailed visual understanding or efficient processing according to actual needs.

OpenVision's technical principles

• Progressive resolution training strategyOpenVision employs a training approach that starts with low resolution (e.g., 84×84) and gradually increases to high resolution (e.g., 336×336 or 384×384). This significantly improves training efficiency, being 2 to 3 times faster than CLIP and SigLIP without sacrificing downstream performance.
• Visual encoder pre-trainingDuring the pre-training phase, each OpenVision encoder is trained at three consecutive resolution stages. Specifically, different variants of the model are trained at different resolutions depending on their size; for example, the Large, SoViT-400M, and Huge variants are trained at 84×84, 224×224, and finally at 336×336 or 384×384, respectively. After pre-training, the text tower and decoder are discarded, retaining only the visual backbone.
• Multimodal learning architectureOpenVision's model architecture primarily consists of a visual encoder and a text encoder. The visual encoder is responsible for extracting features from images, while the text encoder extracts features from natural language sequences. During training, the model learns by comparing image-text pairs, maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs.
• Optimize lightweight systems and edge computing applicationsOpenVision is effectively combined with small language models to build multimodal models with low parameter count.

OpenVision project address

• Project official website:https://ucsc-vlaa.github.io/OpenVision/
• Github repository:https://github.com/UCSC-VLAA/OpenVision
• HuggingFace model library:https://huggingface.co/collections/UCSC-VLAA/openvision
• arXiv technical paper:https://arxiv.org/pdf/2505.04601

Application scenarios of OpenVision

• Multimodal learningOpenVision can be integrated into multimodal frameworks, such as LLaVA, for tasks such as image recognition, video analysis, and natural language processing.
• Industrial testingOpenVision's high-resolution image sensors and powerful processing capabilities are suitable for industrial inspection, such as defect detection and dimensional measurement.
• Robot VisionBy integrating high-performance image sensors and processing chips, OpenVision can provide robots with real-time visual perception capabilities, supporting tasks such as path planning and object recognition.
• autonomous drivingIn the field of autonomous driving, OpenVision can serve as an in-vehicle vision system, processing image data from multiple cameras to perform environmental perception and decision-making.
• Scientific Research and EducationIts open-source nature makes OpenVision an ideal platform for researchers and educational institutions to conduct research and teaching in visual computing.