dots.llm1 - A large-scale text model open-sourced by Xiaohongshu Hi Lab
dots.llm1 is a medium-sized Mixture of Experts (MoE) text model open-sourced by Xiaohongshu's hi lab, with 142 billion parameters and 14 billion activation parameters. The model was tested on 11.2TB of high-quality token data...
What is dots.llm1?
dots.llm1 is a medium-sized Mixture of Experts (MoE) text model open-sourced by Xiaohongshu's hi lab, boasting 142 billion parameters and 14 billion activation parameters. The model was pre-trained on 11.2TB of high-quality token data, employing efficient Interleaved 1F1B pipelined parallelism and Grouped GEMM optimization techniques, significantly improving training efficiency. Based on a carefully designed data processing workflow and two-stage supervised fine-tuning, dots.llm1 performs exceptionally well in common Chinese and English scenarios, mathematical tasks, and coding tasks, demonstrating strong competitiveness compared to models like Qwen2.5-72B. The hi lab team open-sourced the checkpoint and instruction models for each 1TB of tokens during the pretraining phase, providing a rich research foundation for the large model community and contributing to the development of large model technology.
The main functions of dots.llm1
- Multilingual text generationIt supports the generation of high-quality Chinese and English text, and is suitable for various scenarios, such as writing assistance and content creation.
- Complex instruction followIt can understand and execute complex instructions to complete specific tasks, such as data processing and code generation.
- Knowledge Q&AIt provides accurate knowledge Q&A services to help users obtain the information they need.
- Mathematical and Code ReasoningPossesses mathematical calculation and code reasoning abilities, capable of solving mathematical problems and writing simple code.
- Multiple rounds of dialogueIt supports multi-turn dialogues, enabling natural and fluent communication with users, understanding the context, and providing appropriate responses.
Technical Principles of dots.llm1
- Mixture of Experts (MoE) Architecturedots.llm1 is a MoE model based on a Decoder-only Transformer, configured with 6-in-128 Experts, meaning that each token activates a maximum of 6 experts, for a total of 128 experts. Out of 142 billion parameters, each forward propagation activates only 14 billion parameters, using a gating mechanism to select the most relevant experts for computation, thus improving computational efficiency.
- High-efficiency data processing and training frameworkPre-training was performed using 11.2TB of high-quality token data, sourced from Common Crawl and web data crawled by our own spider. The data underwent multiple rounds of cleaning and filtering to ensure quality. Training efficiency was significantly improved through Interleaved 1F1B pipelined parallel All-to-All overlap and efficient Grouped GEMM.
- Two-stage monitoring and fine-tuningTwo rounds of basic training were conducted on the full dataset, using techniques such as sampling and dynamic learning rate adjustment to initially unleash the model's potential. For specific domains such as mathematics and code, rejection sampling fine-tuning was introduced, combined with a validator to select high-confidence important samples, further improving the model's inference performance.
- Learning rate scheduling and hyperparameter optimizationBased on the WSD learning rate scheduling method, a 3e-4 learning rate is maintained during the stabilization phase to train 10T of tokens. During the annealing phase, two phases are used to train 1.2T of tokens, gradually reducing the learning rate. During training, hyperparameters such as batch size are adjusted to ensure training stability and avoid loss spikes.
The project address for dots.llm1
- GitHub repository:https://github.com/rednote-hilab/dots.llm1
- HuggingFace model library:https://huggingface.co/rednote-hilab
- arXiv technical paper:https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf
Application scenarios of dots.llm1
- Content creationIt assists in writing copy, stories, news articles, etc., and provides creative inspiration and writing suggestions.
- Education and LearningUsed in language learning, programming education, and Q&A to help learners improve their skills.
- Commercial OfficeBuild intelligent customer service, generate data analysis and market research reports to help businesses operate.
- Programming DevelopmentGenerates code snippets and code documentation, provides debugging suggestions, and improves development efficiency.
- Personal AssistantManage schedules, plan tasks, organize information, and improve personal work and life efficiency.