MUMU - Text and Image-Driven Multimodal Generative Model

What is MUMU?

MUMU is a multimodal image generation model that improves the accuracy and quality of generated images by combining text prompts and reference images. The MUMU model architecture is based on a pre-trained convolutional UNet from SDXL and utilizes the hidden states of the Idefics2 visual language model. During training, the model uses both synthetic and real data. Through a two-stage training process, MUMU better preserves the details of conditional images and demonstrates generalization ability on tasks such as style transfer and role consistency.

MUMU's main functions

• Multimodal input processingMUMU can process both text and image inputs simultaneously, and it can generate images with the same style as the reference image based on the text description.
• Style conversionMUMU can transform realistic images into cartoon or other specified styles, which is very useful in the fields of art creation and design.
• Role ConsistencyWhen generating images, MUMU can maintain the consistency of the character's features, and can maintain the character's uniqueness even when style is changed or combined with different elements.
• Details preservedMUMU is better able to preserve the details of the input image when generating images, which is crucial for generating high-quality images.
• Conditional image generationUsers can provide specific conditions or requirements, and MUMU can generate images that meet the user's needs based on these conditions.

MUMU's technical principles

• Multimodal learningThe MUMU model can handle various types of input data, including text and images. It generates images that match the text descriptions by learning the relationships between text descriptions and image content.
• Visual-Language Model EncoderThe MUMU model uses a vision-language model encoder to process input text and images. The encoder converts text into vector representations that the model can understand and transforms image content into feature vectors.
• diffusion decoderThe MUMU model uses a diffusion decoder to generate images. A diffusion decoder is a generative model that generates images by progressively adding details, thus achieving high-quality image generation.
• Conditional generationThe MUMU model considers conditional information from both text and image when generating images. This means the model generates new images based on the input text description and reference image, ensuring that the generated images meet the given conditions.

MUMU's project address

• arXiv technical paper:https://arxiv.org/pdf/2406.18790

How to use MUMU

• Prepare input data:Prepare a text description: Clearly describe the features and style of the image you want to generate.Prepare reference images: If there are specific styles or elements that need to be reflected in the generated images, one or more reference images can be provided.
• Accessing the MUMU model:Upload or input your text description and reference image using the interface or platform provided by MUMU model.
• Set generation parameters:Set the parameters for image generation as needed, such as resolution, style preference, and specific image content.
• Submit a generation request:Submit the prepared input data and parameters to the MUMU model and request the generation of an image.
• Waiting for results to be generated:The model generates the target image based on the input text and image, after a certain amount of computation time.

Application scenarios of MUMU

• Artistic CreationArtists and designers can use MUMU to generate images with specific styles and themes based on text descriptions for use in paintings, illustrations, or other visual art works.
• Advertising and MarketingBusinesses can use MUMU to quickly generate attractive advertising images that can be customized according to marketing strategies and brand style.
• Game developmentGame designers can use MUMU to generate images of characters, scenes, or props in games, accelerating the visual development process.
• Film and animation productionIn the pre-production of movies or animations, MUMU can help concept artists quickly generate visual concept art.
• Fashion DesignFashion designers can use MUMU to explore design concepts for clothing, accessories, and more, and generate fashion illustrations.