Ovis-U1 - A unified multimodal model launched by Alibaba

What is Ovis-U1?

Ovis-U1 is a multimodal unified model developed by Alibaba Group's Ovis team, boasting 3 billion parameters. The model integrates three core capabilities: multimodal understanding, text-to-image generation, and image editing. Based on an advanced architecture and a collaborative unified training method, it achieves high-fidelity image synthesis and efficient text-visual interaction. Ovis-U1 has achieved leading results in multiple academic benchmark tests in multimodal understanding, generation, and editing, demonstrating strong generalization ability and outstanding performance.

Main functions of Ovis-U1

• Multimodal understandingIt supports understanding complex visual scenes and text content, answering questions about images, performing visual question answering (VQA) tasks, and generating image descriptions.
• Text to Image GenerationGenerates high-quality images based on text descriptions, supporting various styles and complex scene descriptions.
• Image editingIt allows for precise editing of images based on text instructions, including adding, adjusting, replacing, and deleting elements in the image, as well as style conversion.

Ovis-U1 Technical Principles

• Architecture Design:
  • Visual Decoder: A diffusion-based Transformer architecture (MMDiT) generates high-quality images from text embeddings.
  • Bidirectional Token RefinerEnhance the interaction between text and visual embeddings, and improve the performance of text-to-image synthesis and image editing tasks.
  • Visual EncoderFine-tuning based on pre-trained visual encoders (such as Aimv2-large-patch14-448) to adapt to multimodal tasks.
  • AdapterConnect the visual encoder and the multimodal large language model (MLLM) to align visual and text embeddings.
  • Multimodal Large Language Model (MLLM)As the core of the model, it processes textual and visual information and supports a variety of multimodal tasks.
• Unified training methodsOvis-U1 is trained simultaneously on multimodal understanding, text-to-image generation, and image editing tasks, improving the model's generalization ability based on shared knowledge. The training process consists of six stages, progressively optimizing the model's performance on different tasks. Each stage has a specific task and training objective, gradually improving the model's multimodal capabilities.
• Data composition:
  • Multimodal understanding of dataThis includes publicly available datasets (such as COYO, Wukong, Laion, ShareGPT4V, CC3M) and internally developed data.
  • Text to Image Generation DataUsing the Laion5B and JourneyDB datasets, we generate detailed image descriptions based on a pre-trained model.
  • Image + Text to Image Data GenerationIt covers a variety of tasks, including image editing, reference image-driven image generation, and pixel-level controlled image generation.
• Performance optimizationIn image editing tasks, the guided coefficients (CFG) of text and images are adjusted to achieve precise control over editing instructions. The multimodal capabilities of the model are comprehensively evaluated using multiple benchmarks (such as OpenCompass, GenEval, DPG-Bench, ImgEdit-Bench, and GEdit-Bench-EN).

Ovis-U1 project address

• GitHub repositoryhttps://github.com/AIDC-AI/Ovis-U1
• HuggingFace model libraryhttps://huggingface.co/AIDC-AI/Ovis-U1-3B
• Technical Papers: https://github.com/AIDC-AI/Ovis-U1/blob/main/docs/Ovis_U1_Report.pdf
• Experience the demo onlinehttps://huggingface.co/spaces/AIDC-AI/Ovis-U1-3B

Application scenarios of Ovis-U1

• Content creationOvis-U1 generates high-quality image and video frame sequences based on text descriptions, providing artists and video editors with an efficient tool for creative brainstorming and content creation, significantly improving creative efficiency.
• Advertising and MarketingThe model generates attractive advertising images and promotional posters based on product characteristics and target audience descriptions, creates image and video content for social media marketing, and helps brands enhance their communication effectiveness and attract more user attention.
• Game developmentOvis-U1 generates images of game scenes, characters, and props based on game backgrounds and character descriptions, providing creative inspiration and initial materials for game design.
• Architectural DesignOvis-U1 generates architectural concept drawings, interior scenes, and furniture layout images based on architectural style and surrounding environment descriptions, helping clients quickly understand design intent, assisting designers in efficiently presenting design solutions, and improving design communication efficiency.
• Scientific researchThe model can generate visualizations of complex scientific phenomena and data, as well as images of experimental scenarios and equipment, helping researchers better understand and present their research findings.