project
BlueLM-V-3B - An algorithm and system co-design method developed by ViVo in collaboration with the Chinese University of Hong Kong.
BlueLM-V-3B is an algorithm and system co-design method jointly developed by vivo AI Lab and the MMLab of the Chinese University of Hong Kong. It supports the efficient deployment of multimodal large language models (MLLMs) to mobile devices. The model has a small size (2.7B)...
What is BlueLM-V-3B?
BlueLM-V-3B is an algorithm and system co-design method jointly developed by vivo AI Lab and MMLab of the Chinese University of Hong Kong. It supports the efficient deployment of multimodal large language models (MLLM) to mobile devices. The model is characterized by its small size (2.7B language parameters and 400M visual parameters), fast speed (24.4 tokens/s generation speed), and strong performance (score of 66.1 in the OpenCompass benchmark). Based on an optimized dynamic resolution scheme and hardware-aware deployment, it improves the inference efficiency and performance of the model on mobile phones.
Main functions of BlueLM-V-3B
- Multimodal understandingIt processes and integrates various data formats such as text and images, providing richer interaction and deeper contextual understanding.
- Real-time processingEnables real-time responses on mobile devices, suitable for scenarios requiring rapid feedback, such as augmented reality or real-time translation.
- Privacy protectionIt supports local processing on the device, reducing data transfer and enhancing user privacy protection.
- High-efficiency deployment: Optimize the model to adapt to the computing and memory limitations of mobile devices, ensuring efficient operation on resource-constrained hardware.
- high performanceAchieve performance comparable to models with a larger number of parameters while having a relatively small number of parameters.
- Cross-language abilityIt supports multilingual understanding, enhancing the model's applicability in different language environments.
Technical Principles of BlueLM-V-3B
- Algorithm Design:
- Dynamic resolution processing: Redesign the dynamic image resolution strategy to adapt to high-resolution image understanding, reduce the number of image tokens, and reduce deployment complexity.
- Loose aspect ratio matching methodIntroducing parameter α allows for the selection of a more suitable aspect ratio, reducing image magnification and optimizing training and deployment efficiency.
- System Design:
- Batch Image EncodingThe parallel processing capabilities of the NPU are used to batch process image patches, accelerating the image encoding process.
- Pipeline parallel processingIn the image encoding process, a pipeline parallel processing is designed to hide the execution latency of Conv2D operations.
- Token downsamplingBased on merging and fusing information, the number of image tokens is reduced to adapt to the processing capabilities of the NPU.
- Block computationTo handle long input tokens, a block-based strategy is adopted to balance parallel processing and NPU performance.
- Model QuantizationWe quantize model weights with INT8 and INT4 precision, while maintaining the INT16 precision of LLM activation and FP16 precision of ViT activation, thus balancing computational efficiency and model accuracy.
- Overall frameworkThe ViT and LLM models are loaded simultaneously during model initialization, and processing begins immediately after the user uploads an image. Commands can be input simultaneously, improving response speed and memory usage efficiency.
BlueLM-V-3B project address
- arXiv technical paper:https://arxiv.org/pdf/2411.10640
Application scenarios of BlueLM-V-3B
- Augmented Reality (AR)Provide real-time augmented reality experiences on mobile devices, such as using a phone camera to identify objects in the real world and provide relevant information.
- Real-time translationIn cross-language communication, it translates text in speech or images in real time, helping users overcome language barriers.
- Educational SupportAs a learning tool, it helps students understand complex concepts and provides an interactive learning experience with images and text.
- Visual Question Answering (VQA)Users can ask questions about the content of images based on photos they take or upload, and the model provides accurate answers.
- Image and document understandingIn office automation, it involves understanding and processing the content of images and documents, such as automatically recognizing document information like invoices and contracts.