OneCAT - A unified multimodal model jointly launched by Meituan and Shanghai Jiao Tong University

What is OneCAT?

OneCAT is a new unified multimodal model launched by Meituan. It employs a pure decoder architecture, seamlessly integrating multimodal understanding, text-to-image generation, and image editing functions. The model abandons the traditional multimodal model design that relies on external visual encoders and word segmenters, achieving efficient multimodal processing through a modal-specific expert mixture (MoE) structure and a multi-scale autoregressive mechanism. It performs exceptionally well when processing high-resolution image input and output. Through an innovative scale-aware adapter and a multimodal multi-functional attention mechanism, it further enhances visual generation capabilities and cross-modal alignment capabilities.

OneCAT's main functions

• Multimodal understandingIt can efficiently handle multimodal understanding tasks of images and text, without the need for external visual encoders or word segmenters, and can achieve deep understanding of image and text content directly in a pure decoder architecture.
• Text to Image GenerationIt generates high-quality images based on text descriptions and predicts visual labels from low to high resolution step by step through a multi-scale autoregressive mechanism. The generation process is efficient and produces excellent results.
• Image editingIt supports instruction-based image editing, conditionally applying the visual generation process to reference images and editing instructions. This enables powerful conditional generation capabilities without requiring additional architectural modifications, allowing for precise local and global adjustments to images.

OneCAT's technical principles

• Pure decoder architectureOneCAT uses a pure decoder autoregressive Transformer model, which eliminates the need for external vision components such as visual transformers (ViT) or visual taggers. This significantly simplifies the model structure and reduces computational overhead, with a clear efficiency advantage, especially when processing high-resolution inputs.
• Modality-Specific Expert Hybrid (MoE) StructureThe model incorporates three dedicated feedforward network (FFN) experts to handle text tokens, continuous visual tokens, and discrete visual tokens, respectively, for language understanding, multimodal understanding, and image synthesis. All queries, key-value pairs (QKV), and attention layers are shared across different modalities and tasks, improving parameter efficiency and enhancing cross-modal alignment capabilities.
• Multiscale visual autoregressive mechanismIntroduced in Large Language Models (LLMs), it generates images in a coarse-to-fine, hierarchical manner, progressively predicting visual tokens from the lowest to the highest resolution, significantly reducing decoding steps while maintaining state-of-the-art performance.
• Multimodal multifunctional attention mechanismBased on PyTorch FlexAttention, the model can flexibly adapt to multiple modalities and tasks. Text labeling uses causal attention, continuous visual labels are processed through full attention, and multi-scale discrete visual labels are processed through block-based causal attention.

OneCAT's project address

• Project official websitehttps://onecat-ai.github.io/
• Github repositoryhttps://github.com/onecat-ai/onecat
• HuggingFace model libraryhttps://huggingface.co/onecat-ai/OneCAT-3B
• arXiv technical paper: https://arxiv.org/pdf/2509.03498

OneCAT application scenarios

• Intelligent customer service and content moderationOneCAT's multimodal understanding capabilities can efficiently process image and text content. It can be used in intelligent customer service systems to understand user-uploaded image and text information and provide accurate responses. It can also be used in the field of content moderation to automatically identify and filter illegal image and text content.
• Creative Design and Digital Content CreationIts text-to-image generation function can generate high-quality images based on text descriptions, providing designers and creators with creative inspiration and quickly generating image content that meets their needs. It can be used for early concept design in fields such as advertising design, film and television special effects production, and game development.
• Advertising Design and MarketingIn advertising design, OneCAT can quickly generate corresponding image assets based on advertising copy, improving design efficiency. It can also be used to generate personalized advertising content, producing advertising images tailored to the preferences of different target audiences.
• Film and television post-productionOneCAT's image editing functions can be used for tasks such as image restoration, style transfer, and special effects addition in film and television post-production, helping film and television producers quickly realize creative effects and improve production efficiency.
• Education and LearningIn the field of education, OneCAT can generate images related to teaching content to help students better understand and memorize knowledge. For example, it can generate corresponding diagrams based on scientific concepts or generate relevant scene images based on descriptions of historical events.