WeGen - A unified multimodal generative model jointly developed by the University of Science and Technology of China and Shanghai Jiao Tong University.
WeGen is a unified multimodal generative model developed by the University of Science and Technology of China in collaboration with Shanghai Jiao Tong University, the WeChat team, the Chinese Academy of Sciences, and other institutions. It enables diverse visual generation tasks based on natural dialogue. WeGen combines multimodal...
What is WeGen?
WeGen is a unified multimodal generative model developed by the University of Science and Technology of China in collaboration with Shanghai Jiao Tong University, the WeChat team, and the Chinese Academy of Sciences. It enables diverse visual generation tasks based on natural dialogue. WeGen combines a multimodal large language model (MLLM) and a diffusion model to handle various tasks such as text-to-image generation, conditional generation, image editing, and style transfer. WeGen's core advantage lies in its ability to provide diverse creative outputs when user instructions are ambiguous, while maintaining consistency between the generated results and the instructions and reference images when users have specific needs. Based on a Dynamic Instance Consistency (DIIC) data pipeline and a Prompt Self-Rewrite (PSR) mechanism, WeGen addresses the two major challenges of instance identity consistency and generation diversity, demonstrating its potential as a user-friendly design assistant.
WeGen's main functions
- Text to Image GenerationGenerate high-quality images based on text descriptions.
- Condition-driven generationGenerate images based on specific conditions (such as edge map, depth map, pose map).
- Image editing and restorationModify, repair, or expand existing images.
- Style transferApplying the style of one image to another.
- Multi-subject generation: Preserve key features of multiple reference objects when generating images.
- Interactive generationBased on natural dialogue and user interaction, the generated results are gradually optimized.
- Creative Design AssistanceIt provides users with diverse generation options to inspire creativity.
WeGen's technical principles
- Combining Multimodal Large Language Model (MLLM) with Diffusion ModelCLIP is used as a visual encoder to transform images into semantic features; diffusion models (such as SDXL) are used as decoders to generate high-quality images. LLMs (such as LLaMA) process natural language instructions, achieving the fusion of text and visual information.
- Dynamic Instance Consistency (DIIC)The DIIC data pipeline tracks natural changes in objects using video sequences, maintaining their identity consistency. It addresses the shortcomings of traditional methods in preserving instance identity, allowing the model to retain key features when images are modified.
- Hint Self-Rewrite (PSR) mechanismPSR rewrites text prompts based on a language model, introducing randomness to generate diverse images. It uses discrete text sampling to allow the model to explore different interpretations while maintaining semantic consistency.
- Unified framework and interactive generationWeGen integrates multiple visual generation tasks into a single framework, based on natural dialogue and user interaction, to progressively optimize the generated results while retaining the parts that satisfy the user.
- Large-scale dataset supportWeGen is trained on a large dataset extracted from internet videos. The dataset contains rich descriptions of object dynamics and automatically labeled descriptions, which helps the model learn consistency and diversity.
WeGen's project address
- GitHub repository:https://github.com/hzphzp/WeGen
- arXiv technical paper:https://arxiv.org/pdf/2503.01115
WeGen application scenarios
- Creative DesignIt helps designers quickly generate creative concept sketches and inspire creativity, and is applicable to fields such as advertising, packaging, and architecture.
- Content creationGenerate concept art for scenes, characters, or props for the film, game, and animation industries, accelerating the creation process.
- Educational SupportGenerate images related to the teaching content to help students understand abstract concepts more intuitively.
- Personalized customizationGenerate customized design solutions based on user needs, such as clothing and home decoration.
- Virtual social interaction and entertainmentGenerate virtual avatars, scenes, or props to enhance the experience of virtual social interaction and gaming.