AB
AB
AiBoss

All Categories

project

Ovis2 - A multimodal large language series model launched by Alibaba International.

Ovis2 is a new multimodal language model developed by Alibaba's international team. It addresses the differences between visual and textual modalities based on structured embedding alignment. Ovis2 inherits and optimizes the Ovis series architecture, enhancing the performance of small-scale models...

What is Ovis2?

Ovis2 is a new multimodal large language model launched by Alibaba's international team. It addresses the differences between visual and textual modalities based on structured embedding alignment. Ovis2 inherits and optimizes the Ovis series architecture, enhancing the capability density of small-scale models and significantly improving CoT (Conceptual Chain) reasoning capabilities based on instruction fine-tuning and preference learning. Ovis2 introduces video and multi-image processing capabilities, enhancing multilingual support and OCR capabilities in complex scenarios. The Ovis2 series includes six model versions with different parameter scales: 1B, 2B, 4B, 8B, 16B, and 34B. All versions demonstrate excellent performance in the OpenCompass multimodal benchmark, excelling in mathematical reasoning and video understanding. The open-source nature of Ovis2 provides new directions and tools for the research and application of multimodal large models.

Main features of Ovis2

  • Multimodal understanding and generationIt can process multiple input modalities such as text, images, and videos, generate high-quality text output, and support visual and language tasks in complex scenarios.
  • Strengthen reasoning abilityBased on the enhanced reasoning ability of the Chain of Reasoning (CoT), it solves complex logic and mathematical problems and provides step-by-step reasoning solutions.
  • Video and Multi-Image ProcessingIt introduces video understanding capabilities, supports keyframe selection and multiple image inputs, and processes complex visual information across frames.
  • Multilingual support and OCR capabilitiesIt supports text processing in multiple languages and extracts structured data from complex visual elements such as tables and charts.
  • Small model optimizationBased on optimized training strategies, small-scale models can achieve high capability density to meet the needs of different application scenarios.

Ovis2's technical principles

  • Structured Embedding AlignmentThe system segments images into patches using a visual tokenizer, extracts features, and maps them to "visual words" to form probabilistic visual tokens. These visual tokens, along with text tokens, are input into an LLM to achieve structured alignment between modalities.
  • Four-stage training strategy:
    • Phase 1Freeze the LLM, train the visual module, and learn the transformation from visual features to embeddings.
    • Phase TwoFurther train the vision module to enhance high-resolution image understanding and multilingual OCR capabilities.
    • Phase ThreeUse visual data in the form of dialogue to align visual embeddings with the dialogue format of LLM.
    • Phase 4: Conduct multimodal instruction training and preference learning to improve the model's ability to follow user instructions and the quality of its output.
  • Video understanding enhancementThe MDP3 algorithm (based on the correlation between frames and text, combinatorial diversity, and sequence) is used to select key frames, thereby improving video understanding capabilities.
  • Based on Transformer architectureIt combines powerful visual encoders (such as ViT) and language models (such as Qwen) to achieve efficient multimodal fusion and generation.

Ovis2 project address

Application scenarios of Ovis2

  • Researchers and developersProfessionals engaged in research on artificial intelligence and multimodal technologies, as well as developers who need to develop intelligent applications, to optimize models, improve algorithms, or develop multimodal applications.
  • Content creatorsFor professionals in news media, advertising, marketing, and other industries, this tool allows for the rapid generation of descriptions, text, and headlines for images or videos, improving creative efficiency.
  • Educators and studentsTeachers generate explanatory text for images or videos to help students understand complex content; students then solve problems in their learning through visual question-and-answer functionality.
  • Enterprise usersProfessionals in industries such as finance, law, and healthcare can process complex documents, images, or video data, extract key information, and assist in decision-making.
  • ordinary users and technology enthusiastsFor those interested in artificial intelligence, simple multimodal tasks can be performed, such as generating image descriptions or performing visual question answering, to explore the application of the technology in daily life.
Back