Mogao - A unified architecture for multimodal understanding and generation launched by ByteDance's Seed team.

What is Mogao?

Mogao is a comprehensive interleaved multimodal generation model developed by ByteDance's Seed team. Its architecture employs dual visual encoders, combining a variational autoencoder (VAE) and a visual transformer (ViT) to improve visual understanding and context alignment in image generation. Mogao introduces Interleaved Rotational Position Embedding (IL-RoPE) to capture the two-dimensional spatial location information of images and the temporal positional relationships of multimodal data, further enhancing generation quality and consistency through multimodal classifier-free guidance technology.

Mogao's main functions

• Multimodal understanding and generationMogao can handle interleaved sequences of text and images, achieving high-quality multimodal understanding and generation. It can generate high-quality images given a text description and generate related text content given an image. In multimodal understanding tasks, text tagging focuses on visual transformer (ViT) tags and text tags in the historical sequence to better understand image content.
• Zero-shot image editing and combination generationMogao demonstrates powerful zero-shot image editing capabilities, enabling the editing and modification of images without additional training. It also possesses combinatorial generation capabilities, combining different elements to generate new images with strong consistency and coherence.
• High-quality image generationMogao excels in image generation, performing exceptionally well across multiple style categories including realism, graphic design, animation, and illustration. It supports image generation up to 2K resolution, producing images with high detail and quality.
• Text rendering capabilitiesMogao has significantly improved text rendering, with a text usability rate of up to 94%, effectively solving the problem of Chinese character rendering in previous image generation.

Mogao's technical principles

• Dual vision encoderMogao uses a variational autoencoder (VAE) and a visual transformer (ViT) as the visual encoder. When an image is used as conditional input, visual features from both the VAE and ViT are extracted and appended to the historical sequence. For multimodal understanding tasks, text tagging focuses only on ViT tags and text tags; for multimodal generation tasks, noisy VAE tags focus on all tags in the historical sequence.
• Deeply integrated architectureBased on a pre-trained Large Language Model (LLM), Mogao uses a unified self-attention layer to process visual and text sequences simultaneously, and uses different multilayer perceptrons (MLPs) in the feedforward network (FFN) to process visual and text modalities separately.
• Interleaved Rotational Position Embedding (IL-RoPE)It is used to capture the two-dimensional spatial location information of images and the temporal location relationship of multimodal data. The model can better handle interlaced text and image sequences.
• Mixed resolution trainingPre-training and fine-tuning are performed on images with different aspect ratios and resolutions, from low resolution (e.g., 256²) to high resolution (e.g., 2048²), and size embedding is introduced to enable the model to perceive the target resolution.
• Cross-modal RoPETreating text tokens as two-dimensional tokens and applying two-dimensional RoPE further enhances the alignment between visual and text tokens.
• Post-training phaseThis includes Continuous Training (CT), Supervised Fine-tuning (SFT), Human Feedback Alignment (RLHF), and Cueing Engineering (PE) to improve model performance and controllability.
• Defect-aware training paradigmA defect detector is introduced to accurately locate defective regions, and the training dataset is effectively expanded through mask latent space optimization.
• Hyper-SD and RayFlow: Optimize the generation path, guide each data point to the target distribution of a specific instance, reduce path collisions, and improve generation stability and sample diversity.
• Importance sampling mechanismThe learning process focuses on the most critical time steps during training, supporting efficient low-step sampling without affecting the generation quality.

Mogao's project address

• arXiv technical paper:https://arxiv.org/pdf/2505.05472

Application scenarios of Mogao

• Content creationMogao can generate high-quality images from text descriptions, and it can also generate relevant text descriptions from images.
• Smart AssistantMogao can combine multiple modalities such as voice, image, and text to achieve more natural and intelligent human-computer interaction.
• Mutual retrieval of images and textUsers can find relevant images by entering a text description, or obtain relevant text descriptions by uploading images.
• Virtual Reality and Augmented RealityMogao can be used to generate virtual environments and interactive elements, enhancing the user experience of virtual reality and augmented reality.
• Medical image analysisMogao can combine medical images of different modalities (such as MRI, CT, ultrasound, etc.) with text descriptions to improve the accuracy of disease diagnosis and the ability to detect diseases early.