Vary-toy: An open-source small visual multimodal model

What is Vary-toy?

Vary-toy is a small-scale visual language model (LVLM) proposed by researchers from Megvii, the University of Chinese Academy of Sciences, and Huazhong University of Science and Technology of China. It aims to address the challenges of training and deploying large-scale LVLMs. For researchers with limited resources, large models typically have billions of parameters, making them difficult to train and deploy on consumer-grade GPUs (such as the GTX 1080 Ti). Vary-toy's core goal is to allow researchers to experience all the functionalities of current LVLMs (document OCR, visual localization, image captioning, visual text retrieval, etc.) with limited hardware resources.

Vary-toy's main functions

• Document-level optical character recognition (OCR)Vary-toy can recognize and understand text in document images, which is very useful in scenarios such as processing scanned documents and PDF files.
• Image descriptionThe model is able to generate descriptive text for images, which is crucial for understanding image content and generating image descriptions for tasks such as VQA.
• Visual Question Answering (VQA)Vary-toy is able to answer questions about the content of an image, which involves understanding the visual information of the image as well as the textual information associated with it.
• Object detectionThrough enhanced visual vocabulary, Vary-toy possesses the ability to perceive (localize) natural objects, enabling it to identify and locate objects in images.
• Image to text conversionVary-toy can convert image content into a structured text format, such as converting PDF images to Markdown format.
• Multimodal dialogueThe Vary-toy model supports multimodal dialogue and is able to understand and generate dialogues related to image content.

Vary-toy's official website

• Official project homepage:https://varytoy.github.io/
• Arxiv research paper:https://arxiv.org/abs/2401.12503
• Demo runtime address:https://vary.xiaomy.net/
• GitHub repository:https://github.com/Ucas-HaoranWei/Vary-toy

Vary-toy's technical principles

Vary-toy works based on several key technical and design decisions that work together to improve the model's performance on visual language tasks while keeping the model small. The main components of how Vary-toy works are as follows:

1. Visual vocabulary generationVary-toy utilizes a small autoregressive model (OPT-125M) to generate a novel visual vocabulary network. This network learns how to effectively encode visual information by processing PDF image-text pairs and object detection data from natural images. Compared to the traditional Vary model, Vary-toy treats natural images as positive samples instead of negative samples when generating visual vocabulary, thus making fuller use of the network's capacity.
2. The fusion of visual vocabulary and CLIPAfter generating new visual vocabulary, Vary-toy combines it with the original CLIP (Contrastive Language-Image Pre-training) model. CLIP is a powerful visual-language model that maps images and text to a shared embedding space. In this way, Vary-toy can leverage CLIP's image understanding capabilities while enhancing the processing of textual information through a new visual vocabulary network.
3. Multi-task pre-trainingVary-toy employs a multi-task learning strategy during the pre-training phase, meaning the model processes multiple types of data simultaneously during training, such as image captioning, PDF OCR, object detection, plain text dialogue, and visual question answering (VQA). This multi-task training helps the model learn richer visual and linguistic representations, improving its generalization ability across various downstream tasks.
4. Model StructureVary-toy follows Vary's pipeline design but with structural adjustments. When an image is input, the new visual vocabulary branch resizes the image to a 1024×1024 resolution, while the CLIP branch obtains a 224×224 image through center cropping. The image features output from the two branches are merged and used as input to the Qwen-1.8B language model with 1.8B parameters.
5. Data input formatTo adapt to different tasks, Vary-toy needs to handle multiple input formats. For example, for PDF image-text pairs, the model uses a specific cue (such as "Provide the OCR results of this image.") to guide the output of the correct result. For object detection tasks, the model uses different cue templates to handle multiple objects in the image.
6. Fine-tuning (SFT)Following pre-training, Vary-toy further optimizes the model through a Supplemental Text-First-Fit (SFT) stage. This stage uses the LLaVA-80K dataset, an image dataset containing detailed descriptions and prompts generated by GPT4. This helps the model better understand and generate text related to the image content.

Through these mechanisms, Vary-toy is able to effectively handle complex visual language tasks while maintaining model miniaturization. This design makes Vary-toy a powerful tool for visual language research in resource-constrained environments.

How to use Vary-toy

1. Access the official Vary-toy demo experience address (vary.xiaomy.net)
2. Click to upload an image or select an example image on the left.
3. Input prompts such as describing image content or detecting objects in the image.
4. Wait for the model to generate the results.