ProX - A framework for improving the quality of pre-trained data for large language models
ProX (Programming Every Example) is a framework designed to improve the quality of pre-trained data for large language models. Unlike traditional methods that rely on human experts to formulate rules, ProX treats data cleaning as a programming problem...
What is ProX?
ProX (Programming Every Example) is a framework designed to improve the quality of pre-training data for large language models. Unlike traditional methods that rely on human experts to formulate rules, ProX treats data cleaning as a programming problem, enabling models to automatically perform fine-grained operations such as string normalization and noisy line removal. Even small models (e.g., with 0.3B parameters) can demonstrate data processing capabilities comparable to human experts. Experimental results show that ProX-processed data used for model pre-training achieves performance improvements of over 2% across various downstream tasks. Another significant advantage of ProX is its broad applicability across different model sizes and pre-training corpora, including in domain-specific (e.g., mathematics) continuous pre-training, significantly improving model performance without requiring domain-specific design. ProX saves training FLOPs, providing a promising path for the efficient pre-training of large language models.
ProX's main features
- Data Refinement: ProX refines large-scale datasets based on its generation and execution processes, improving data quality for use in the pre-training of large language models.
- Automated processing: It automatically cleans and improves each data sample with fine granularity, without the need for human expert intervention.
- Performance improvements: Models pre-trained on data processed by ProX showed performance improvements of over 2% across multiple downstream tasks.
- Domain flexibility: It is applicable to various fields, including mathematics, and improves accuracy without requiring a specific field design.
- Resource saving: Compared to data synthesis methods based on large language models, ProX significantly reduces the demand for computing resources while maintaining the results.
ProX's technical principles
- Model adaptability: ProX first fine-tunes a small, basic language model on seed data to adapt it for the data refining task.
- Program generation: The adapted model generates a data processing procedure for each sample in the pre-training corpus, which includes operations such as filtering, string standardization, and noise removal.
- Program execution: The generated program is executed by a predefined executor, producing a refined corpus ready for pre-training.
- Two-stage refining: ProX comprises two phases: document-level programming and block-level programming, which respectively refine data into coarse and fine granular levels.
- Function call: ProX enhances data quality through flexible function calls, unifying specific transformation or cleaning processes.
- Computational efficiency: ProX demonstrates how to achieve more efficient pre-training by investing additional computational resources to refine the pre-training corpus with fewer pre-training computational FLOPs.
ProX project address
- Project official websitegair-nlp.github.io/ProX
- GitHub repository:https://github.com/GAIR-NLP/ProX
- HuggingFace model library:https://huggingface.co/gair-prox
- arXiv technical paper:https://arxiv.org/pdf/2409.17115
Application scenarios of ProX
- Large-scale language model pre-trainingProX can improve the dataset quality of large language models that are pre-trained on a large amount of high-quality text data.
- Data cleaning and preprocessingIn data mining, natural language processing, and other machine learning tasks, ProX automates data cleaning and preprocessing steps, reducing human intervention.
- Domain Adaptation TrainingFor applications in specific fields, such as healthcare, law, or finance, ProX can optimize datasets to better fit specialized terminology and context.
- Continuous learningIn scenarios involving continuous or incremental learning, ProX helps models adapt to new information and changes by constantly refining data.
- Data SynthesisProX can generate high-quality synthetic data to enhance existing datasets, especially in data-scarce domains.