BGE-VL - A multimodal vector model open-sourced by the Beijing Academy of Artificial Intelligence (BAAI) in collaboration with several universities.

What is BGE-VL?

BGE-VL is a multimodal vector model developed by the Beijing Academy of Artificial Intelligence (BAAI) in collaboration with several universities. It is trained on the large-scale synthetic dataset MegaPairs. BGE-VL focuses on multimodal retrieval tasks, such as text-to-image retrieval and combined image retrieval, leveraging efficient multimodal data synthesis methods to enhance the model's generalization ability and retrieval performance. The BGE-VL series includes BGE-VL-Base and BGE-VL-Large, based on the CLIP architecture, and BGE-VL-MLLM, based on a large multimodal model architecture. The model has performed exceptionally well on multiple benchmarks, setting new benchmarks in combined image retrieval tasks and significantly improving retrieval accuracy. BGE-VL's core advantages lie in the scalability and high quality of its data synthesis method and its outstanding generalization ability on multimodal tasks.

Main functions of BGE-VL

• Image and text searchIt can retrieve the most relevant image based on the input text description, or retrieve relevant text information based on the input image.
• Combined Image RetrievalIt supports users to input both image and text commands simultaneously, and comprehensively understands the information from both to retrieve more accurate target images.
• Multimodal embeddingIt maps images and text into a unified vector space, allowing data from different modalities to be compared and retrieved using vector similarity.
• Command fine-tuningThe model is fine-tuned based on synthesized multimodal instruction data to better understand and execute complex multimodal tasks, thereby improving the model's generalization ability and task adaptability.

BGE-VL Technical Principles

• Data synthesis methods (MegaPairs):
  • Data miningThe goal is to mine diverse image pairs from massive image and text corpora and use various similarity models (such as CLIP) to find candidate images related to the query image.
  • Instruction generationBased on the Multimodal Large Language Model (MLLM) and Large Language Model (LLM), open-domain retrieval instructions are generated, the relationships between image pairs are summarized, and high-quality retrieval instructions are written.
  • Triplet constructionThis tool generates multimodal triplet data containing "query image, query statement, and target image" for model training. The data requires no manual annotation, making it efficient and scalable.
• Multimodal model architecture:
  • CLIP-based architectureBGE-VL-Base and BGE-VL-Large use a CLIP-like architecture to map images and text to the same vector space using image encoders and text encoders, and compare and learn to optimize model performance.
  • Multimodal large model architectureBGE-VL-MLLM is based on a more complex multimodal large model architecture, which can handle complex multimodal interactions and instruction understanding tasks.
  • Command fine-tuningThe model is fine-tuned based on synthesized multimodal instruction data to enhance its understanding and execution capabilities for multimodal tasks.
• Comparative learning and optimizationDuring training, the model comparatively learns and optimizes the vector representation of multimodal embeddings, making related images and text closer together in the vector space, while distancing unrelated data. Based on training on large-scale synthetic data, the model learns more generalized multimodal feature representations, demonstrating excellent performance in various multimodal tasks.

BGE-VL project address

• HuggingFace model library:https://huggingface.co/collections/BAAI/megapairs

Application scenarios of BGE-VL

• Intelligent SearchUsers can upload images or enter text to quickly find relevant content, improving search accuracy.
• Content RecommendationBased on user-uploaded content or interests, similar image and text materials are recommended to enhance the personalized experience.
• Image editing assistanceIt helps designers quickly find reference images with similar styles, improving creative efficiency.
• Intelligent Customer ServiceBy combining images and text to understand user problems, we can provide more intuitive solutions and improve service efficiency.
• Cultural Heritage ResearchBased on image and text search, it can quickly find relevant cultural relics or research materials, thus assisting in archaeological and conservation work.