Nexus-Gen - A multimodal image generation model open-sourced by Moda in collaboration with institutions such as East China Normal University.
Nexus-Gen is an open-source, all-around image generation model developed by the Moda team, East China Normal University, and other institutions. It supports simultaneous image understanding, generation, and editing tasks. Nexus-Gen combines the powerful capabilities of language models and diffusion models...
What is Nexus-Gen?
Nexus-Gen is an open-source, all-around image generation model developed by the Moda team, East China Normal University, and other institutions. It supports simultaneous image understanding, generation, and editing tasks. Nexus-Gen integrates the powerful capabilities of language and diffusion models, and addresses the problem of image embedding error accumulation in traditional methods based on a pre-filled autoregressive strategy. The model achieves image quality and editing capabilities comparable to GPT-4o, driving the development of the multimodal model field.
Main functions of Nexus-Gen
- Image understandingAnalyze image content, generate descriptive text, and answer image-related questions.
- Image generationGenerates high-quality images based on text descriptions, supporting the generation of complex scenes and styles.
- Image editingIt offers a variety of editing functions, including color adjustment, object addition/deletion, style conversion, etc.
Nexus-Gen's technical principles
- Architecture DesignThe input text and images are transformed into embedding vectors using a text tokenizer and a vision encoder, respectively. These vectors are then fed into an autoregressive transformer to generate the output text tokens and image embeddings. The image embeddings are aligned to the same feature space as the input using a vision projector and decoded into pixel-level images using a diffusion model (vision decoder).
- Pre-filled autoregressive strategyA pre-filled autoregressive strategy is introduced. During training, the model fills the image embedding locations with learnable special tokens. During inference, the model directly pre-fills the special tokens based on the predicted image starting token (BOI), unifying training and inference behaviors and avoiding error accumulation.
- Unified task representationBased on a unified task representation format, image understanding, generation, and editing tasks are integrated into a single framework. Data for all tasks is organized using a similar message format, and the model learns the synergistic effects between different tasks, improving the ability to handle multimodal tasks.
- Two-stage alignment training
- Autoregressive Transformer TrainingThe model is gradually endowed with image generation and editing capabilities through three-stage training, and finally the generation quality is improved with high-quality data.
- Diffusion model trainingBased on single-stage training, the input conditions are changed from text to image embedding to achieve high-quality image reconstruction.
Nexus-Gen project address
- GitHub repository:https://github.com/modelscope/Nexus-Gen
- HuggingFace model library:https://huggingface.co/modelscope/Nexus-Gen
- arXiv technical paper:https://arxiv.org/pdf/2504.21356
Application scenarios of Nexus-Gen
- Creative DesignQuickly generate artwork, illustrations, or concept art, support style conversion, and improve creative efficiency.
- Content creationGenerate article images and video materials, and edit images to match the content theme.
- Advertising and MarketingCreate advertising materials that match the brand style and quickly generate product display images.
- Education and LearningGenerate images of historical, scientific, or literary scenes to enhance the intuitiveness of learning.
- Game developmentQuickly generate game scenes, characters, and items, supporting virtual reality (VR) and augmented reality (AR) applications.