FastVLM - Apple's efficient visual language model

What is FastVLM?

FastVLM is a high-performance visual language model (VLM) from Apple that improves the efficiency and performance of high-resolution image processing. The model introduces the novel FastViTHD hybrid visual encoder, effectively reducing the number of visual tokens and significantly lowering encoding time. While maintaining similar performance to existing VLMs, FastVLM dramatically improves processing speed; for example, in the LLaVA-1.5 setting, it reduces the time to first token generation (TTFT) by 3.2 times compared to other models. FastVLM performs excellently in various VLM benchmarks and has a smaller model size and requires less training data, demonstrating its efficiency and practicality in multimodal understanding tasks.

The main functions of FastVLM

• Efficient encoding of high-resolution imagesSupports quickly converting high-resolution images into visual tokens, reducing encoding time and the number of tokens required.
• Low latency responseIn LLaVA-1.5 settings, FastVLM's first token generation time (TTFT) is significantly shortened, resulting in a faster response speed, making it suitable for real-time interactive scenarios, such as mobile text and image Q&A assistants, which can quickly provide answers.
• Model size optimizationThe visual encoder is significantly smaller, 3.4 times smaller than similar models, making it easier to deploy on mobile devices and edge intelligence devices, reducing hardware requirements and improving model portability.
• Improve VLM performance: While significantly reducing the time to first token (TTFT), it maintains performance comparable to existing advanced models.
• Simplified model designNo additional token pruning steps are required, simplifying the design of the visual encoder.

The technical principle of FastVLM

• Hybrid Vision Encoder FastViTHDFastViTHD is the core component of FastVLM. Compared to traditional pure convolutional encoders or pure Transformer encoders (such as ViT), it combines the advantages of convolutional layers and Transformer blocks. Convolutional layers can efficiently process high-resolution images, reducing the number of tokens based on downsampling operations. Transformer blocks can further extract high-quality visual features, providing more accurate visual information for LLM. The FastViTHD architecture contains multiple stages, each with a specific depth and embedding dimension. For example, its depth is set to [2, 12, 24, 4, 2], and its embedding dimension is [96, 192, 384, 768, 1536].
• Optimized architecture designFastVLM optimizes its architecture to achieve high efficiency in high-resolution image processing. Unlike simply extending the FastViT architecture, FastVLM introduces an additional stage that downsamples the tensors before the self-attention layer. The self-attention layer only needs to process the already downsampled tensors, reducing computational cost. For example, in a typical hybrid model, the tensors processed by the self-attention layer are downsampled by a factor of 16 in each direction; in FastVLM, the input tensors processed by the widest MLP layer are downsampled by a factor of 64 in each direction, significantly reducing visual encoding latency.
• Working in conjunction with LLMFastVLM connects the visual encoder to the LLM based projection layer (also known as the connector module). The visual tokens output by the visual encoder are converted into a format suitable for LLM processing by the connector module. The LLM then uses the visual tokens and text input to fuse and understand the data, generating the corresponding output. This collaborative approach allows visual information to be effectively integrated into the language generation process, enabling the functionality of a visual language model.

FastVLM project address

• GitHub repositoryhttps://github.com/apple/ml-fastvlm
• HuggingFace model libraryhttps://huggingface.co/collections/apple/fastvlm-68ac97b9cd5cacefdd04872e
• arXiv technical paper: https://www.arxiv.org/pdf/2412.13303

Application scenarios of FastVLM

• Visual Q&AQuickly understand images and answer related questions.
• Image and text matchingDetermine whether the image and text description are consistent.
• Document UnderstandingAnalyze the text content in the image and understand its meaning.
• Image description generationAutomatically generate descriptive text for images.
• Multimodal recommendation: Make accurate recommendations by combining image and text information.