MindOmni - A multimodal large language model launched by Tencent in collaboration with Tsinghua University and other institutions.
MindOmni is a multimodal large-scale language model developed by Tencent ARC Lab in collaboration with Tsinghua University Shenzhen International Graduate School, the Chinese University of Hong Kong, and the University of Hong Kong. Based on the reinforcement learning algorithm (RGPO), it significantly improves visual language models...
What is MindOmni?
MindOmni is a multimodal large-scale language model developed by Tencent ARC Lab in collaboration with Tsinghua University Shenzhen International Graduate School, the Chinese University of Hong Kong, and the University of Hong Kong. Based on the reinforcement learning algorithm (RGPO), it significantly enhances the reasoning and generation capabilities of visual language models. The model employs a three-stage training strategy: first, a unified visual language model is constructed; then, supervised fine-tuning is performed using Chained Thinking (CoT) data; and finally, the RGPO algorithm is used to optimize reasoning generation. MindOmni demonstrates outstanding performance in multimodal understanding and generation tasks, showcasing powerful reasoning and generation capabilities in complex scenarios such as mathematical reasoning, thus paving a new path for the development of multimodal AI.
MindOmni's main functions
- Visual understandingIt supports understanding and interpreting image content and answering image-related questions.
- Text to Image GenerationGenerate high-quality images based on text descriptions.
- Inference generationIt can perform complex logical reasoning and generate images that contain the reasoning process.
- Visual editingEdit existing images, such as adding, deleting, or modifying elements in the image.
- Multimodal input processingIt supports simultaneous processing of text and image inputs and generates corresponding outputs.
MindOmni's technical principles
- Model Architecture:
- Visual Language Model (VLM)Image features are extracted based on pre-trained ViT (Vision Transformer), and text input is converted into discrete text tags using a text encoder.
- Lightweight connectorsUsed to connect VLM and diffusion decoder to ensure effective transfer of features between different modules.
- Text header: Process text input and generate text output.
- Decoder diffusion module: Responsible for generating images, converting potential noise into actual images based on the denoising process.
- Three-stage training strategy:
- Phase 1Pre-training enables the model to have basic text-to-image generation and editing capabilities. Image-text pairs and X2I data pairs are used to train the connector, ensuring the diffusion decoder can seamlessly process the semantic representation of the VLM. Diffusion loss and KL divergence loss are used as the optimization objective functions.
- Phase TwoThe model is further optimized based on Chain Thinking (CoT) instruction data to generate logical reasoning processes. A series of coarse-to-fine CoT instruction data is constructed, and the model is supervised and fine-tuned using the instruction data.
- Phase ThreeThis study further enhances the model's inference and generation capabilities based on reinforcement learning, ensuring the quality and accuracy of generated content. It introduces the Inference Generation Policy Optimization (RGPO) algorithm, using multimodal feedback signals (including image and text features) to guide policy updates. Format reward functions and consistency reward functions are introduced to evaluate visual-language alignment. A KL divergence regularizer is used to stabilize the training process and prevent knowledge forgetting.
MindOmni's project address
- Project official websitehttps://mindomni.github.io/
- GitHub repositoryhttps://github.com/TencentARC/MindOmni
- arXiv technical paper: https://arxiv.org/pdf/2505.13031
- Experience the demo onlinehttps://huggingface.co/spaces/stevengrove/MindOmni
MindOmni Application Scenarios
- Content creationGenerate high-quality images based on text descriptions for use in visual content creation in industries such as advertising, gaming, and film, accelerating the creative design process.
- EducationGenerate images and explanations related to the teaching content to assist teaching, help students better understand and remember complex concepts, and improve learning outcomes.
- Entertainment industryGenerate characters, scenes, and props in game development to accelerate the development process; provide storyboards and concept art for film and television production to enrich creative expression.
- Advertising industryGenerate attractive advertising images and videos to improve advertising effectiveness.
- Smart AssistantIt combines voice, text, and image input to provide a more natural and intelligent interactive experience, meeting diverse user needs.