UniFluid - A multimodal image generation and understanding framework jointly developed by Google and MIT.

What is UniFluid?

UniFluid, a unified autoregressive framework jointly developed by Google DeepMind and MIT, is used for joint visual generation and understanding tasks. It processes multimodal image and text inputs based on continuous visual tagging to generate discrete text tags and continuous image tags. The framework is based on a pre-trained Gemma model, trained with paired image-text data, allowing generation and understanding tasks to mutually reinforce each other. UniFluid uses the standard SentencePiece as the text tagger, a Continuous Variational Autoencoder (VAE) as the image generation tagger, and combines the SigLIP image encoder for the understanding task. Through carefully tuned training recipes and loss weight balance, UniFluid achieves results comparable to or better than single-task baselines in both image generation and understanding tasks, demonstrating strong downstream task transfer capabilities, including image editing, visual description, and question answering.

Main functions of UniFluid

• Joint visual generation and understandingIt simultaneously handles image generation (e.g., generating images from text descriptions) and visual understanding (e.g., image description, visual question answering) tasks.
• Multimodal input processingIt supports multimodal input of images and text, embedding images and text into the same space for joint training.
• High-quality image generationIt generates high-quality images based on continuous visual markers and supports random generation order to improve the generation effect.
• Strong visual comprehension abilityIt performs well in tasks such as visual question answering and image description, and supports a variety of downstream tasks.
• Efficient downstream task migrationIt can quickly adapt to downstream tasks such as image editing and visual question answering, demonstrating good versatility and scalability.

UniFluid's technical principles

• Unified Autoregressive FrameworkUniFluid uses an autoregressive model to embed image and text inputs into the same space, and handles visual generation and understanding tasks in a unified manner based on "next label prediction".
• Continuous visual markersThe image is encoded into a continuous label using a continuous variational autoencoder (VAE), which avoids the information loss caused by discrete labels and preserves the continuity of the image.
• Modality-specific prediction headThe classification head handles text generation tasks, while the diffusion head handles image generation tasks, ensuring effective training and inference across different modalities.
• Randomly generated orderIn image generation tasks, image labels are generated based on a random order to avoid generation problems caused by a fixed order (such as raster order).
• Loss function balanceAdjusting the loss weights for image generation and text understanding tasks achieves a balance between the two tasks, allowing the model to perform well in both tasks.
• Pre-trained large language models (LLM)Based on the pre-trained Gemma model, its powerful language and visual understanding capabilities are used to improve the overall performance of the model.

UniFluid project address

• arXiv technical paper:https://arxiv.org/pdf/2503.13436

Application scenarios of UniFluid

• Image generationIn fields such as creative design, advertising production, and game development, it can quickly generate corresponding image content based on the text description input by the user.
• Image editingIn photo editing, digital art creation, and other scenarios, it allows you to add or delete objects, change styles, adjust colors, and so on.
• Visual Question Answering (VQA)In fields such as education, intelligent customer service, and assisting visually impaired individuals, it helps users better understand and interpret image information.
• Image Captioning): Automatically generates descriptive text for images, used in social media content generation, image search engine optimization, and assisting visually impaired people in understanding images.
• Multimodal content creationIt provides a more vivid visual and linguistic experience in video scriptwriting, virtual reality (VR), and augmented reality (AR) content development.