SmolVLM - A lightweight visual language model from Hugging Face

What is SmolVLM?

SmolVLM is a lightweight visual language model from Hugging Face, designed specifically for on-device inference. With 2 billion parameters, it achieves efficient memory usage and fast processing speed. SmolVLM offers three versions to meet different needs: SmolVLM-Base: suitable for fine-tuning downstream tasks; SmolVLM-Synthetic: fine-tuned based on synthetic data; and SmolVLM-Instruct: an instruction-based fine-tuned version that can be directly applied to interactive applications. The model borrows from the Idefics3 concept, using SmolLM2 1.7B as the language backbone and improving visual information compression efficiency through pixel shuffling. It is trained on the Cauldron and Doctmatix datasets, optimizing image encoding and text processing capabilities.

Main functions of SmolVLM

• Device-side inferenceSmolVLM is designed for device-side inference and can run effectively in resource-constrained environments such as laptops, consumer-grade GPUs, or mobile devices.
• Fine-tuning capabilityThe model offers three versions to meet different needs:
  • SmolVLM-Base is used for fine-tuning of downstream tasks;
  • SmolVLM-Synthetic is fine-tuned based on synthesis data;
  • A finely tuned version of the SmolVLM-Instruct command that can be directly applied to interactive applications.
• Optimized architecture designDrawing inspiration from Idefics3, this approach uses SmolLM2 1.7B as the language backbone and employs a pixel shuffling strategy to improve the compression rate of visual information, thereby achieving more efficient visual information processing.
• Processing long texts and multiple imagesThe training dataset includes Cauldron and Doctrine, and SmolLM2 is context-extended to handle longer text sequences and multiple images.
• Low memory usageSmolVLM encodes a 384×384 pixel image block into 81 tokens, while Qwen2-VL requires 16,000 tokens, significantly reducing memory usage.
• High throughputIn multiple benchmark tests, SmolVLM’s prefill throughput is 3.3 to 4.5 times faster than Qwen2-VL, and its generation throughput is 7.5 to 16 times faster.
• Open source modelSmolVLM is completely open source, and all model checkpoints, VLM datasets, training recipes, and tools are released under the Apache 2.0 license.
• Training datasetSmolVLM covers Cauldron and Doctmatix, and adds contextual extensions to SmolLM2, enabling it to handle longer text sequences and multiple images.

SmolVLM project address

• Github repository:https://github.com/huggingface/blog/blob/main/smolvlm.md
• HuggingFace model library:https://huggingface.co/blog/smolvlm
• Experience the demo online:https://huggingface.co/spaces/HuggingFaceTB/SmolVLM
• Complete list of datasets:https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct/blob/main/smolvlm-data.pdf

Application scenarios of SmolVLM

• Video analysisSmolVLM has demonstrated its potential as a fundamental video analytics task under computationally limited conditions. In the CinePile benchmark, SmolVLM achieved a score of 27.14%, demonstrating its competitive ability in video understanding.
• Visual Language ProcessingSmolVLM provides developers and researchers with a powerful tool for visual language processing without the need for expensive hardware.
• Local deploymentSmall models support local deployment on browsers or edge devices, reducing inference costs and supporting user customization.
• AI popularizationThe development of SmolVLM is expected to expand the use of visual language models, making complex AI systems more accessible and widespread, and providing powerful machine learning capabilities to a wider audience.