Nemotron-CC - A large pre-trained dataset from NVIDIA

What is Nemotron-CC?

Nemotron-CC is a large-scale pre-training dataset released by the NVIDIA team, which transforms Common Crawl data into high-quality corpora suitable for long-sequence pre-training. The Nemotron-CC dataset combines classifier ensembles, synthetic data restatements, and reduced heuristic filter dependencies to achieve a better balance between data volume and quality. Nemotron-CC contains 6.3 trillion tokens, of which 4.4 trillion are globally deduplicated original tokens and 1.9 trillion are synthetically generated tokens. Nemotron-CC demonstrates superior performance in both short-term (1T tokens) and long-term (15T tokens) training, especially in tasks such as MMLU, where it significantly improves model accuracy compared to DCLM and Llama 3.1 models. The development of the Nemotron-CC dataset provides richer and more diverse data resources for training large-scale language models.

Main functions of Nemotron-CC

• Provide high-quality pre-trained datasetsNemotron-CC provides high-quality, large-scale pre-trained datasets for large language models (LLMs), especially suitable for long sequence training, such as training requirements of 15T tokens.
• Balancing data quantity and qualityWithout significantly sacrificing data quality, this method greatly increases the size of the dataset, includes more unique and real tokens, and improves the model's performance in long sequence training.
• Support model performance improvementExperiments show that the model trained by Nemotron-CC performs well on multiple benchmark tasks, especially on tasks such as MMLU, and can significantly improve the accuracy of the model compared to other existing datasets.

Nemotron-CC Technical Principles

• Optimize text extractionJustext was chosen as the HTML-to-text extractor because it performs better in extracting high-quality tokens, effectively improving the initial quality of the dataset.
• Model-based quality labeling:
  • Classifier ensembleWe construct three distinct quality classifiers, each with a different high-quality preference. Based on the ensemble of these three classifiers, we score all documents and categorize the crawled corpus into different quality levels according to these scores.
  • Quality label allocationFurther, the fine-grained quality scores are clustered into 5 downstream task quality categories. Based on continuous pre-training and task performance evaluation, quality labels that are more in line with actual performance are assigned to each category.
• Synthetic data generation:
  • Low-quality data restatementFor low-quality documents, restatement reduces noise and errors while retaining useful information. Restating low-quality documents with Wikipedia-style prompts effectively reduces errors and redundancy and improves formatting.
  • High-quality data expansionFor high-quality documents, more unique tokens can be obtained by generating diverse question-and-answer pairs, extracting and refining knowledge, and creating knowledge lists, thereby enriching the content and diversity of the dataset.
• Dataset integration:
  • Large-scale data synthesisMore than 1.8 trillion synthetic tokens were generated using the Mistral NeMo 12B model, including 336.3 billion tokens generated from low-quality documents and 1.5 trillion tokens generated from high-quality documents.
  • Dataset ConstructionThe above techniques and methods were applied to 99 snapshots of Common Crawl (CC-MAIN-2013-20 to CC-MAIN-2024-30) to construct a dataset of 6.3 trillion tokens, of which 4.4 trillion were the original tokens deduplicated globally and 1.9 trillion were the synthetically generated tokens.

Nemotron-CC project address

• Project official website:https://developer.nvidia.com/blog/announcing-nemotron-cc
• arXiv technical paper:https://arxiv.org/pdf/2412.02595

Application scenarios of Nemotron-CC

• Pre-trained large language modelsSuitable for training long sequences, such as 15T tokens, to improve the performance of models in complex tasks.
• Fine-tuning and task-specific adaptationThis facilitates the model's rapid adaptation to multi-task learning and domain-specific tasks, thereby improving task performance.
• Text generation taskUsed to generate high-quality text, such as news and stories, and to improve the naturalness of dialogue systems.
• Research and DevelopmentIt helps explore improvements in model architecture and training methods, and provides benchmark testing resources.
• Education and TrainingGenerate educational resources to support language learning and enrich educational content.