LLaVA-OneVision-1.5 - EvolvingLMMS-Lab open-source multimodal model - AiBoss

What is LLaVA-OneVision-1.5?

LLaVA-OneVision-1.5 is an open-source multimodal model that achieves high performance, low cost, and strong reproducibility through efficient training and high-quality data. It employs the self-developed RICE-ViT as the visual encoder, combining 2D rotational position encoding and a region-aware attention mechanism to support variable input resolution and enhance object and OCR capabilities. The language model is based on Qwen3 and optimized through a three-stage training process (language-image alignment, high-quality knowledge mid-term pre-training, and visual instruction alignment). Offline parallel data packaging and a hybrid parallel strategy are used during training to improve computational power and memory efficiency. For data, an 85M pre-training dataset is constructed, employing a "conceptual balance" strategy and covering multiple sources; 22M instruction data covers eight categories, aggregated from multiple sources and formatted uniformly. LLaVA-OneVision-1.5 performs excellently on multimodal benchmarks, is cost-effective, and is transparent and open throughout the entire chain, providing code, data, and model resources to help the community achieve low-cost reproduction and expansion.

Main functions of LLaVA-OneVision-1.5

Multimodal understanding and generationIt can process and understand information in multiple modalities such as images and text, and generate high-quality text descriptions, answer questions, or perform reasoning.
Visual Question Answering (VQA)It answers questions based on the content in images and supports a wide range of visual tasks, such as object recognition and scene understanding.
Image description generationIt generates accurate and detailed descriptive text for input images, helping users better understand the image content.
Instruction Compliance and ExecutionIt executes corresponding tasks based on user-provided instructions, such as image editing and information extraction, and has good command generalization ability.
Cross-modal retrievalIt supports querying images based on text or querying text based on images, enabling cross-modal information retrieval.
Long-tail recognition capabilityIt can effectively identify and understand categories or concepts that appear less frequently in the data, thereby improving the model's generalization ability.
Multilingual supportIt supports input and output in multiple languages and has a certain degree of cross-language understanding and generation capabilities.
Knowledge EnhancementBy pre-training with high-quality knowledge data, the model acquires richer world knowledge and can better handle complex multimodal tasks.
Efficient training and reproductionIt employs optimized training strategies and data packaging techniques to achieve an efficient training process, and provides complete code and data resources to facilitate community reproduction and expansion.

Technical Principles of LLaVA-OneVision-1.5

Visual encoderIt adopts the self-developed RICE-ViT (Region-aware Cluster Discrimination Vision Transformer) as the visual backbone, and enhances the semantic understanding of local regions in the image through the region-aware attention mechanism and the unified region cluster discrimination loss, while supporting variable input resolution.
Projector DesignBy using a multilayer perceptron (MLP) to map visual features to the text embedding space of a language model, effective alignment between visual and linguistic features can be achieved.
Language ModelBased on Qwen3 as the language backbone, it provides powerful language generation and understanding capabilities, and supports text processing in multimodal tasks.
Three-stage training processThis includes language-image alignment, high-quality knowledge mid-term pre-training, and visual instruction alignment, gradually improving the model's multimodal alignment capability and task generalization capability.
Offline parallel data packagingThe pre-training dataset is constructed using a feature-driven "concept balancing" strategy, and offline parallel data packaging technology is used to reduce padding waste and improve training efficiency.
Hybrid Parallelism and Long Context OptimizationDuring training, hybrid parallelism (tensor parallelism, pipeline parallelism, and sequence parallelism) and long context optimization techniques are used to improve computing power utilization and GPU memory efficiency.
Data Construction and OptimizationWe constructed a large-scale pre-trained dataset and instruction fine-tuning dataset, and ensured the high quality and diversity of the data through multi-source aggregation, format unification and safe screening.

LLaVA-OneVision-1.5 project address

Github address: https://github.com/EvolvingLMMs-Lab/LLaVA-OneVision-1.5
HuggingFace model library: https://huggingface.co/collections/lmms-lab/llava-onevision-15-68d385fe73b50bd22de23713
arXiv technical paper: https://arxiv.org/pdf/2509.23661
Experience the demo online: https://huggingface.co/spaces/lmms-lab/LLaVA-OneVision-1.5

Application Scenarios of LLaVA-OneVision-1.5

Intelligent Customer ServiceBy understanding the images or text information uploaded by users, it provides automated customer support, answers questions, and offers solutions.
Content creationIt helps creators generate image descriptions, creative copy, or stories, improving creative efficiency and quality.
Educational SupportIn the field of education, it is used to interpret the content in images, assist teaching, and help students better understand complex visual information.
Medical image analysisIt assists doctors in interpreting medical images, providing preliminary diagnostic suggestions, or generating image reports.
Intelligent drivingIn autonomous driving systems, it is used to understand road scenarios, assist in decision-making, and improve driving safety.
Image Editing and DesignIt allows users to edit, crop, and add effects to images based on their instructions, improving the convenience of image processing.