VARGPT - A unified model for multimodal understanding and generation launched by Peking University

What is VARGPT?

VARGPT is an innovative multimodal large language model focused on visual understanding and generation tasks. Based on an autoregressive framework, it unifies visual generation and understanding within a single model, avoiding the complexity of task switching. VARGPT extends the LLaVA architecture, achieving visual understanding through next-token prediction and visual generation through next-scale prediction, efficiently handling mixed modal inputs and outputs. VARGPT employs a three-stage training strategy: a pre-training stage learns language and visual features, while a mixed visual instruction fine-tuning stage further aligns visual and textual features and enhances instruction compliance. This results in excellent performance on tasks such as visual question answering and reasoning, while also demonstrating powerful capabilities in visual generation tasks, naturally generating high-quality images.

VARGPT's main functions

• Unity of visual understanding and generationVARGPT is a multimodal large language model that can achieve visual understanding and visual generation within a single autoregressive framework. It performs visual understanding tasks, such as visual question answering and reasoning, using a next-token prediction paradigm, and achieves visual generation based on a next-scale prediction paradigm.
• Mixed-mode input and outputVARGPT supports mixed-modal input of text and images, and can output text and images simultaneously. It is more flexible and efficient in handling complex vision-language tasks.
• High-efficiency visual generationVARGPT is equipped with a dedicated visual decoder containing 2 billion parameters for high-quality visual generation. It can generate images based on text instructions, progressively building image content through an autoregressive process.
• Wide applicability of multimodal tasksVARGPT outperforms other models in benchmark tests across multiple vision centers, naturally supports instruction-to-image synthesis, and is suitable for a variety of vision-language tasks.

VARGPT Technical Principles

• Unified Autoregressive FrameworkVARGPT unifies visual understanding and generation tasks within a single autoregressive framework. For visual understanding, the model employs a next-token prediction paradigm, predicting the next text token to perform visual question answering and reasoning tasks. For visual generation, it uses a next-scale prediction paradigm, progressively predicting the next scale information of the image. This allows the model to efficiently handle both visual and language tasks within a single framework.
• Visual decoder and feature mappingVARGPT contains a dedicated visual decoder with 2 billion parameters for high-quality visual generation. The decoder consists of 30 Transformer blocks, each containing 30 attention heads with a width of 1920, and employs adaptive normalization (AdaLN).
• Multiscale labelingTo support visual generation, VARGPT uses a multi-scale variational autoencoder (VAE) architecture, similar to the VAR model. The architecture decomposes images into labels of different scales through a multi-scale quantization scheme, with a vocabulary size of 4090, and uses the OpenImages dataset for training.
• Mixed-mode input and outputVARGPT supports mixed-modal input of text and images, and can output both text and images simultaneously. The model flexibly switches between text and visual modalities through special markers and cue formats, achieving mixed-modal generation.
• Three-stage training strategy
  • Pre-training phaseLearn the mapping relationship between text and visual features.
  • Hybrid vision instruction fine-tuning stageBy constructing a visually generated instruction dataset and combining it with a multi-turn dialogue instruction dataset for hybrid training, the model's capabilities in visual question answering and instruction-to-image synthesis tasks are enhanced.

VARGPT project address

• Project official website:https://vargpt-1.github.io/
• GitHub repository:https://github.com/VARGPT-family/VARGPT
• arXiv technical paper:https://arxiv.org/pdf/2501.12327

Application scenarios of VARGPT

• Visual Question Answering and ReasoningVARGPT can handle complex visual question answering tasks by understanding image content and generating accurate text responses.
• Instruction to Image GenerationVARGPT supports generating high-quality images based on text instructions. Images of specific scenes can be generated using simple text descriptions.
• Multimodal content creationVARGPT can handle mixed-modal inputs and outputs, supporting seamless switching between text and images. Input text descriptions, and the model generates corresponding images, or generates related text content based on images.
• Creativity and EntertainmentVARGPT's image generation capabilities can be applied to the creative and entertainment fields to generate personalized artworks, virtual characters, and more.