Infinity-MM - A multi-modal instruction dataset with tens of millions of records launched by Zhiyuan.

What is Infinity-MM?

Infinity-MM is a multimodal instruction dataset with tens of millions of samples, released by the Beijing Academy of Artificial Intelligence (BAAI). It contains 43 million samples and a data volume of 10TB. The dataset has undergone quality filtering and deduplication to ensure high quality and diversity, improving the performance of open-source visual-language models (VLMs). BAAI has introduced a synthetic data generation method based on open-source VLMs, further expanding the dataset's scale and diversity. Based on Infinity-MM, BAAI successfully trained the Aquila-VL-2B multimodal model with 2 billion parameters, achieving state-of-the-art performance among models of similar size.

Main functions of Infinity-MM

• Improve the performance of open source modelsInfinity-MM improves the performance of open-source visual-language models (VLMs) by providing large-scale and high-quality instruction data, approaching or reaching the level of closed-source models.
• Dataset ConstructionIt contains 43 million rigorously screened and deduplicated multimodal samples, covering various types such as visual question answering, text recognition, document analysis, and mathematical reasoning.
• Synthetic data generationBased on open-source VLMs and detailed image annotations, it generates diverse instructions closely related to image content, expanding the scale of the dataset and improving data diversity.
• Model training and evaluationThe Infinity-MM dataset was used to train a VLM with 2 billion parameters, Aquila-VL-2B, and the model showed excellent performance on multiple benchmark tests.
• Promoting multimodal researchBased on providing large-scale, high-quality datasets, it promotes the research and application development in the field of multimodal AI.

Infinity-MM's technical principles

• Data collection and preprocessingInfinity-MM's data comes from multiple public datasets and undergoes deduplication and quality filtering to ensure high quality and diversity.
• Synthetic data generation methods:
  • Image and instruction labeling system: Use open-source recognition models (such as RAM++) to automatically label images, extract key information, and form the semantic basis of the images.
  • Instruction Tag SystemDesign a three-level instruction label system that covers instructions of different levels and types.
  • Establishing a correspondence between images and instruction labelsThis function tracks the correspondence between image tags and instruction tags, enabling quick retrieval of matching instruction/task tags.
• Problem generation and filteringThe indicator model generates specific questions based on the image and instruction type, and makes a reasonable judgment.
• Answer generation and filteringAfter generating the question, the system further generates corresponding instruction answers, which are then strictly filtered to ensure a match with the image content or task.
• Phased training strategyThe Aquila-VL-2B model is based on a phased training method, which gradually improves the model's ability to understand and process visual information.
• Multimodal architectureThe Aquila-VL-2B model is based on the LLaVA-OneVision architecture, combining the text tower (Qwen2.5-1.5B-instruct) and the visual tower (Siglip400m).
• Improved training efficiencyZhiyuan's self-developed FlagScale framework adapts to model training, improving training efficiency to 1.7 times that of the original DeepSpeed-based training code.

Infinity-MM project address

• HuggingFace model library:https://huggingface.co/datasets/BAAI/Infinity-MM
• arXiv technical paper:https://arxiv.org/pdf/2410.18558

Applications of Infinity-MM

• Visual Question Answering (VQA)Based on paired data of images and related questions, a model is trained to understand and answer questions about the content of images.
• Image CaptioningIt generates descriptive text for images and has wide applications in social media, content management, and image retrieval.
• Document understanding and analysisExtracts and understands visual and textual information in documents, suitable for automated office work, intelligent document processing, and information extraction.
• Mathematical and Logical ReasoningTraining models to solve mathematical problems and logical reasoning tasks is extremely useful for educational technology, automated testing, and intelligent tutoring systems.
• Multimodal Interaction SystemsCombining visual and linguistic information enhances the naturalness and efficiency of human-computer interaction, making it suitable for intelligent assistants and customer service robots.