smoltalk-chinese - OpenCSG open-source synthetic dataset designed specifically for large-scale Chinese language models

What is smoltalk-chinese?

smoltalk-chinese is an open-source synthetic dataset from OpenCSG specifically designed for Large Language Models (LLMs) in Chinese. This dataset contains over 700,000 synthetic data points, covering various task types including information retrieval, reasoning, planning, editing, programming, mathematics, role-playing, data analysis, creative writing, consultation, and brainstorming. These diverse tasks are designed to enhance the model's versatility and adaptability, resulting in better performance across different application scenarios. The dataset generation process strictly adheres to high standards, employing advanced generative models and deduplication techniques to ensure data quality and diversity.

Main functions of smoltalk-chinese

• Improve language model performanceThe dataset is designed specifically for large-scale Chinese language models (LLMs) and supports supervised fine-tuning (SFT) of the model through high-quality synthetic data, thereby improving the model's performance on a variety of tasks.
• Diverse task coverageThe dataset covers a variety of task types, including information retrieval, reasoning, planning, editing, programming, mathematics, role-playing, data analysis, creative writing, consulting, and brainstorming, enhancing the model's versatility and adaptability.
• High-quality data generationBased on advanced generative models and deduplication techniques, we ensure data quality and diversity, and avoid data duplication and redundancy.
• Supports multiple application scenariosBy simulating conversational styles in daily life and including data containing mathematical problems, the model can better adapt to real-world application scenarios.

The technical principles of smoltalk-chinese

• Data generationThe original data was synthesized using Magpie, combined with generative models such as deepseek-v2.5 and qwen2.5-72b-instruct, and the Distilabel library for data generation. This combination of tools and models ensured the richness and diversity of the generated data.
• Data filteringBased on the qwen2-7b-instruct model, the clarity and fluency of the first instruction in the dialogue data are scored, and only data with a score of 2 or above are retained to ensure data quality.
• DeduplicationThe first instruction of the dialogue data is encoded using the gte-large-zh model, and deduplication is performed based on embedding similarity (threshold set to 0.8) to ensure the uniqueness and diversity of the data.
• Data classification and statisticsThe generated data is classified and statistically analyzed to better understand its distribution and characteristics.

The project address for smoltalk-chinese

• HuggingFace model library:https://huggingface.co/datasets/opencsg/smoltalk-chinese

Application scenarios of smoltalk-chinese

• Language model fine-tuningThe dataset is designed specifically for supervised fine-tuning (SFT) of large Chinese language models, and supports the improvement of model performance on a variety of tasks through high-quality synthetic data.
• Diverse task trainingThe dataset covers a variety of task types, including information retrieval, reasoning, planning, editing, programming, mathematics, role-playing, data analysis, creative writing, consulting, and brainstorming, which can help the model better understand and generate text in these areas.
• Dialogue system optimizationBy simulating real-world user interaction scenarios, smoltalk-chinese provides rich training materials for the dialogue system, enabling it to better understand and generate natural language dialogues.
• Improve mathematical reasoning abilityThe dataset contains math problems from the Chinese version of Math23K, which helps enhance the model's ability in mathematical reasoning and problem-solving.