Qwen VLo - A multimodal unified understanding and generative model launched by Tongyi Qianwen

What is Qwen VLo?

Qwen VLo is a multimodal unified understanding and generation model developed by the Tongyi Qianwen team. It represents a comprehensive upgrade from a large multimodal model, enabling it to "understand" the world and perform high-quality re-creation based on that understanding, achieving a leap from perception to generation. It accurately understands image content and generates consistent and high-quality images accordingly. Users can use natural language commands to instruct the model to perform style transfer, scene reconstruction, or detail enhancement on images, and the model can flexibly respond and generate results that meet expectations. Qwen VLo supports multilingual commands, breaking down language barriers and providing a convenient interactive experience for users worldwide. It possesses dynamic resolution training and generation capabilities, supporting image generation at any resolution and aspect ratio, making it suitable for various scenarios.

Qwen VLo's main functions

• Precise content understanding and re-creationQwen VLo can accurately understand image content and maintain a high degree of semantic consistency during the generation process. For example, a user can upload a photo of a car and request a "color change." The model can accurately identify the car model, preserve the structural features of the original image, and complete a natural color style transition.
• Open command editing and modificationUsers can submit various creative commands via natural language, such as "change the style of this painting to Van Gogh style" or "add a clear sky to this image." The model can respond flexibly to these commands, completing tasks such as art style transfer, scene reconstruction, and detail enhancement, and can complete complex commands involving multiple operations at once.
• Multilingual command supportQwen VLo supports multiple language commands, including Chinese and English, breaking down language barriers and providing a convenient interactive experience for users worldwide.
• Dynamic resolution generationThe model is trained with dynamic resolution, supporting the generation of images with arbitrary resolution and aspect ratio, and is suitable for various scenarios such as posters, illustrations, and web banners.
• Progressive generation mechanismQwen VLo generates images in a progressively clearer manner from left to right and top to bottom, allowing you to observe and adjust the generation process in real time for a more flexible and controllable creative experience.
• Image detection and annotationQwen VLo can perform annotation tasks on existing information, such as detection, segmentation, and edge detection.
• Text to Image GenerationQwen VLo supports generating images directly from text descriptions, including general images and posters in both Chinese and English.

Qwen VLo's technical principles

• Model ArchitectureThe architecture of Qwen VLo mainly consists of the following key modules:
  • Visual EncoderThe Qwen VLo architecture employs the Vision Transformer (ViT) framework, which segments the input image into multiple fixed-size patches and converts them into serialized feature vectors. To support dynamic resolution, Qwen VLo modifies ViT by removing the original absolute position embedding and introducing 2D-RoPE (Rotary Position Embedding) to capture the two-dimensional positional information of the image.
  • Input ProjectorThe visual feature sequence is compressed to a fixed length (e.g., 256) through a single-layer cross-attention module to improve processing efficiency. Simultaneously, this module integrates two-dimensional absolute position encoding to preserve positional information.
  • Large Language Models (LLM)Based on Qwen-7B, it is initialized using pre-trained weights and is responsible for handling language modal input.
  • Output Projector: Map the features generated by LLM to a feature space that the modality generator can understand, typically a simple Transformer layer or MLP layer.
  • Modality Generator: A derivative model based on LDM (Latent Diffusion Models), responsible for generating the final image output.
• Dynamic resolution mechanismQwen VLo introduces a dynamic resolution mechanism, enabling it to handle image inputs of arbitrary resolution.
  • Dynamic visual mark conversionThe model dynamically generates a variable number of visual tokens based on the resolution of the input image, avoiding information loss caused by scaling high-resolution images to low resolution.
  • Smart ResizeDuring the inference phase, the image will be adjusted to a multiple of 28 to maintain the aspect ratio as much as possible and avoid distortion.
  • Token compressionBy using a simple MLP layer, adjacent 2×2 tokens are compressed into a single token to reduce the sequence length of visual input.
• Training methodsQwen VLo training is divided into three stages:
  • Phase 1: Large-scale pre-training for a single taskThe training data is pre-trained using a large amount of text and images, with all images uniformly processed to a size of 224×224. This stage primarily trains the model's ability to align visual modalities with the language model.
  • Phase Two: Multi-task Pre-trainingUsing higher resolution (448×448) data, multiple vision and text generation tasks are introduced to improve the model's multimodal task processing capabilities.
  • Phase 3: Instruction Fine-tuning (SFT)Multimodal, multi-turn conversation data is constructed through manual annotation and model generation to improve the model's instruction compliance and dialogue capabilities.
• Progressive generation mechanismQwen VLo employs a progressive generation approach, building image content step-by-step from left to right and top to bottom. During generation, the model continuously adjusts and optimizes the predicted content to ensure consistency in the final result. It is suitable for tasks requiring fine-grained control over long text paragraphs, allowing users to observe and adjust the generation process in real time.
• Multimodal fusionQwen VLo achieves unified processing of multimodal data by fusing visual and linguistic features. The model can perform operations such as image editing, style transfer, and generation based on user-input text commands, and supports multilingual commands.

How to use Qwen VLo

• Visit Qwen ChatVisit the official website of Qwen Chat.
• Upload an image or enter textUpload the image to the platform, or enter a text command.
• Input commandEnter natural language commands as needed, such as "change the style of this painting to Van Gogh style" or "add a clear sky to this picture".
• View the generated resultsThe model will generate or edit images according to instructions and display the results.

Application scenarios of Qwen VLo

• Image Editing and GenerationQwen VLo can change the style of an image from one to another, such as changing a cartoon style to a realistic style.
• Visual Question Answering (VQA)Qwen VLo can answer questions related to image content, such as describing the scene in the image, identifying objects in the image, etc.
• Document parsingQwen VLo can parse image-based documents (such as scanned documents or image PDFs) and identify the location information of elements such as text, images, and tables.
• Text recognition and information extractionIt supports recognizing text and formulas from images, or extracting information from invoices, certificates, and forms.
• Video UnderstandingQwen VLo can analyze video content, such as locating events in the video and obtaining timestamps, or generating summaries of key time periods.
• Design and CreativityQwen VLo provides powerful tools for designers, marketers, educators, and others to quickly realize creative ideas, such as generating posters and illustrations.