InfiMM-WebMath-40B - A massive, multimodal dataset open-sourced by ByteDance and the Chinese Academy of Sciences.

What is InfiMM-WebMath-40B?

InfiMM-WebMath-40B is a massively multimodal dataset jointly open-sourced by ByteDance and the Chinese Academy of Sciences. It aims to improve the mixed text-image reasoning capabilities of multimodal models in the field of mathematics. Extracted from Common Crawl, the dataset has undergone rigorous screening, cleaning, and annotation, containing 24 million web pages, 85 million image URLs, and 40 billion text tags, covering a rich array of mathematical and scientific content. InfiMM-WebMath-40B significantly enhances the model's mathematical reasoning capabilities, achieving excellent results in benchmark tests such as MathVerse and We-Math.

Main functions of InfiMM-WebMath-40B

• Improve mathematical reasoning abilityInfiMM-WebMath-40B contains a wealth of math and science content, including text, formulas, symbols, and images, to help MLLMs learn math and improve their mathematical reasoning abilities.
• Understanding multimodal informationThis dataset is multimodal, containing both text and image data, helping MLLMs learn how to combine textual and image information for reasoning, and better understand complex mathematical concepts and problems.
• P2 promotes model applicationMLLMs pre-trained based on InfiMM-WebMath-40B can be better applied to mathematics-related applications, such as math problem banks, math learning tools, and math paper reading and comprehension.

Technical Principles of InfiMM-WebMath-40B

• Data sourceBased on Common Crawl, a large database containing publicly available web page content from the Internet.
• Data filteringBased on keyword matching, retain pages containing specific terms such as mathematics and formulas. Set threshold conditions, such as requiring each document to contain a certain number of LaTeX symbols. Use fastText for language filtering, retaining only Chinese and English content.
• Data extractionUse the Trafilatura library to extract text content. Analyze image URLs on web pages to extract images related to mathematical content.
• Data cleaningDeduplication is performed using techniques such as MinHash. Rule-based filtering methods are employed, such as removing short documents containing "lorem ipsum", filtering out documents containing inappropriate content, and excluding documents containing Unicode errors.
• Data labelingThe LLaMA3-70B-Instruct model was used to score mathematical content, and the fastText classifier was used for high-precision filtering.

InfiMM-WebMath-40B project address

• HuggingFace model library:https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B
• arXiv technical paper:https://arxiv.org/pdf/2409.12568

Application scenarios of InfiMM-WebMath-40B

• Math question bank and assessment toolsDevelopers train MLLMs to automatically generate math problems, evaluate student answers, and provide feedback, building intelligent math problem banks and assessment tools.
• Mathematics learning tools and platformsTo help MLLMs better understand mathematical concepts and formulas, and to develop smarter math learning tools and platforms, such as providing personalized learning suggestions, answering student questions, and assisting in solving math problems.
• Reading and understanding mathematical papersMLLMs enhance the understanding of mathematical papers by developing tools that can automatically summarize, translate, and interpret mathematical papers.
• Mathematical researchIt provides data support for mathematical research, such as for training mathematical models, conducting mathematical experiments, and analyzing mathematical data.
• Other scientific fieldsInfiMM-WebMath-40B contains science-related content, such as physics, chemistry, and biology, to train MLLMs to understand scientific concepts, formulas, and images, and to assist in scientific research and applications.