Crawl4LLM - An intelligent web crawler system jointly open-sourced by Tsinghua University and Carnegie Mellon University

What is Crawl4LLM?

Crawl4LLM is an open-source intelligent web crawler system jointly developed by Tsinghua University and Carnegie Mellon University, which improves the efficiency of Large Language Model (LLM) pre-training. Crawl4LLM prioritizes crawling high-value web pages based on intelligent evaluation of their value for LLM pre-training, achieving nearly 5 times the efficiency of traditional crawlers. Crawl4LLM supports three crawling modes: intelligent mode, random crawling mode, and link-based crawling mode. It also features features such as periodic saving of crawler status and data visualization, and can seamlessly integrate with the DCLM framework for direct use in model training.

Main functions of Crawl4LLM

• Intelligent webpage selectionThe system prioritizes crawling high-value web pages based on their value to LLM pre-training, thereby improving data quality and reducing the crawling of invalid data.
• Multiple crawling modes:
  • Smart modeBased on webpage value assessment, prioritize crawling high-value webpages.
  • Random modeRandomly crawl web pages, suitable for scenarios with non-precise requirements.
  • Based on link count patternThis tool crawls web pages based on the number of links, making it suitable for large-scale data collection.
• The crawler status is saved periodically.It supports periodically saving the crawler state, and can resume crawling from the point of interruption even if interrupted, thus avoiding data loss.
• Data browsing and visualizationIt provides data browsing tools and a visualization interface, making it easy for users to monitor the crawling progress and results in real time.
• Seamless integration with DCLM frameworkThe crawled data is used for LLM pre-training to improve the efficiency and accuracy of the data stream.

The technical principles of Crawl4LLM

• Pre-training Influence ScoreCrawl4LLM uses a pre-trained influence scorer (such as DCLM fastText) to score web pages. The scorer evaluates the contribution of web pages to the LLM pre-training based on metrics such as content quality and relevance. In each crawling iteration, newly discovered web pages are scored by the scorer, and the crawling priority is determined based on the score.
• Priority queueCrawl4LLM prioritizes web pages based on a priority queue, crawling the highest-rated pages first, replacing the traditional scheduling mechanism based on graph connectivity (such as PageRank). Using this priority queue, Crawl4LLM quickly discovers and crawls the most valuable web pages for pre-training, reducing the crawling of low-value pages.
• Multi-dimensional data evaluationCrawl4LLM considers the quality of webpage content and combines multiple dimensions such as the number of links and content length to give a comprehensive score. It analyzes the link relationships of high-scoring webpages to discover more potentially high-value webpages.
• Simulation and OptimizationLarge-scale simulation experiments were conducted on the ClueWeb22 dataset to verify its effectiveness in different scenarios. Based on these experiments, the algorithm parameters were optimized to ensure optimal pre-training results with a limited amount of crawling.
• Reduce the burden on the websiteCrawl4LLM reduces unnecessary web scraping, lowers website traffic load, and improves the compliance of scraping activities. It reduces the pressure of data scraping on websites and network resources, promoting a more sustainable approach to acquiring pre-trained data.

Crawl4LLM's project address

• GitHub repository:https://github.com/cxcscmu/Crawl4LLM
• arXiv technical paper:https://arxiv.org/pdf/2502.13347

Application scenarios of Crawl4LLM

• LLM pre-training data collectionEfficiently acquire high-quality data for pre-training large language models.
• Search Engine OptimizationImprove search result quality and optimize user experience.
• Dataset ConstructionQuickly filter and build high-quality corpora to meet research and business needs.
• Network monitoring and analysisMonitor online dynamics and analyze trending topics and information dissemination.
• Enterprise-level data acquisition: Accurately capture data in specific fields for knowledge management or market analysis.