Liquid - A unified multimodal generation framework jointly launched by Huazhong Technology, ByteDance, and the University of Hong Kong.
Liquid is a minimalist unified multimodal generation framework jointly developed by Huazhong University of Science and Technology, ByteDance, and the University of Hong Kong. Based on VQGAN, it encodes images into discrete visual tokens, sharing the same vocabulary space as text tokens, enabling large-scale...
What is Liquid?
Liquid is a minimalist, unified multimodal generation framework jointly developed by Huazhong University of Science and Technology, ByteDance, and the University of Hong Kong. Based on VQGAN, it encodes images into discrete visual tokens, sharing the same lexical space as text tokens. This allows large language models (LLMs) to achieve visual generation and understanding without structural modifications. Liquid abandons traditional external vision modules, leveraging the semantic understanding capabilities of LLMs for multimodal tasks, significantly reducing training costs (by 100 times compared to training from scratch). It performs exceptionally well in visual generation and understanding tasks, outperforming some diffusion models. Liquid reveals the scale dynamics of multimodal tasks, demonstrating that as model size increases, the conflict between visual and language tasks gradually disappears, and the two can even mutually reinforce each other.
Main functions of Liquid
- Visual generationGenerates high-quality images based on text descriptions, supporting multiple resolutions and styles.
- Visual understanding: Handling image-related tasks, such as visual question answering (VQA) and image description generation.
- Multimodal fusionIt seamlessly integrates visual and language tasks, supporting simultaneous processing of text generation, image generation, and visual understanding tasks.
- High-efficiency expansionBased on existing large language models (LLM), it can be trained with a small amount of data and low cost to quickly expand multimodal capabilities.
- Language ability retentionWhile enhancing visual generation capabilities, it maintains strong language generation and comprehension capabilities, making it suitable for multimodal tasks.
Liquid's technical principles
- Image TokenizerVQGAN (Vector Quantized Generative Adversarial Network) encodes images into discrete visual tokens. Visual tokens and text tokens share the same vocabulary space, allowing images and text to be processed in a unified way.
- Unified feature spaceVisual and text tokens are learned in the same feature space and trained based on the "next token prediction" task. This allows for seamless switching and optimization of the model between visual and language tasks.
- LLM-based generationLiquid extends existing large language models (LLMs) to perform visual generation and understanding based on powerful semantic understanding capabilities. It adds embeddings of visual tokens to the LLM to handle visual tasks without requiring additional visual modules (such as CLIP or diffusion models).
- Multimodal data trainingThe model is pre-trained using mixed data (text data and image-text pairs), allowing it to learn both language and visual tasks simultaneously. The model's performance on different tasks is then optimized by adjusting the data ratios.
- Two-way promotion mechanismVisual generation and visual understanding tasks share a unified token space, have consistent optimization goals, and can mutually promote each other. Increasing the amount of data for visual generation or understanding tasks can improve the model's performance on the other task.
Liquid's project address
- Project official website:https://foundationvision.github.io/Liquid/
- GitHub repository:https://github.com/FoundationVision/Liquid
- HuggingFace model library:https://huggingface.co/Junfeng5/Liquid
- arXiv technical paper:https://arxiv.org/pdf/2412.04332
- Experience the demo online:https://huggingface.co/spaces/Junfeng5/Liquid_demo
Application scenarios of Liquid
- Creative DesignGenerate high-quality images based on text descriptions to assist in artistic creation, advertising design, and game art.
- Content creationAutomatically generate images related to text for use in social media, blogs, and news reports.
- Visual Q&AIt can understand image content and answer related questions for use in education, customer service, and intelligent assistants.
- Multimodal dialogueBy combining images and text for intelligent interaction, the naturalness and usability of the dialogue system can be improved.
- VR/AR applicationsGenerate virtual scenes and objects to enhance immersion and interactive experience.