Hugging Face releases a detailed "Small Model Training Guide"

The HuggingFace team's "Small Model Training Guide: The Core Principles of Building Top-Tier Language Models" is an over 200-page technical blog that systematically shares advanced training techniques.LLMThis guide provides valuable end-to-end experience. Based on the team's complete hands-on experience training the 3B parameter model SmolLM3 using 384 H100 GPUs, it offers developers a valuable resource.Large ModelTraining "panoramic maps".

The guide's greatest value lies in its extreme candor and practicality. Unlike academic papers that only showcase perfect results, this guide meticulously records the various "chaotic realities" of the training process—including the agonizing struggle of debugging data loaders in the early hours, the panic of inexplicable loss spikes, and training anomalies caused by minor tensor parallelization bugs, along with their solutions. The guide's unreserved sharing becomes...Large ModelA "pitfall avoidance guide" in the training field.

Training Compass – Deep Thinking Before Decision Making

Before investing millions of computing resources, the guidelines require teams to conduct rigorous self-examination. The quality of decisions made at this stage directly determines the success or failure of the entire project.

In-depth analysis of the reasons for incorrect training

The guide uses a detailed cost calculation model to show that the real cost, from data collection and cleaning, model architecture design, training infrastructure setup to final service deployment, far exceeds the value of "idle computing power." A typical 3B model training project requires 10 person-months of investment just for data preparation, a dedicated team for infrastructure maintenance, and model optimization and deployment are bottomless pits.

The pitfalls of "following the trend" training were verified through the analysis of 10 real-world failure cases. One of the cases involved a company that saw Chat...GPTAfter achieving success, he blindly devoted himself to training himself.Large ModelAs a result, due to the lack of clear application scenarios, although the final model had good technical indicators, it failed to generate value in the business. The guide provides a risk assessment checklist, which includes 37 assessment items across three dimensions: technical risk, market risk, and talent risk.

A rigorous system of standards worthy of training

At the research needs level, the guidelines distinguish between exploratory research and confirmatory research. Exploratory research, such as the design of novel attention mechanisms, requires greater room for trial and error; confirmatory research, such as optimizer improvements, requires rigorous controlled experimental designs.

Regarding production needs, the guidelines specifically emphasize the quantitative assessment of domain specialization. For example, in the legal field, it is necessary to evaluate the performance gap of existing models in sub-tasks such as legal interpretation, case reasoning, and contract analysis. Custom training will only be considered when the accuracy gap in key tasks exceeds 20%.

Experimental Validation – Driving Decision Making with Scientific Methods

The guidelines establish a complete experimental methodology to ensure that every decision is supported by data. The core of this system is to transform subjective experience into objective data through systematic ablation experiments.

Complete Engineering Practice of Ablation Experiments

Baseline selection is no longer a singleSimpleThe decision was a complex one based on multi-dimensional evaluation. The team compared the performance of three mainstream architectures—Llama, Qwen, and Gemma—under the same training configuration, focusing not only on the final evaluation metrics but also on training stability, scalability, and inference efficiency. For example, on some architectures, training stability significantly decreases when the model size scales from 1B to 3B; this characteristic needs to be understood early in the project.

Regarding experiment design, the guidelines provide detailed resource configuration templates. For architecture exploration experiments, it is recommended to train a full-size model on 100B tokens; for data recipe experiments, it is recommended to use a target-size model for parallel testing on multiple data mixing schemes. Each experiment needs clearly defined key performance indicators, including technical indicators such as MMLU score and GSM8K accuracy, and engineering indicators such as training throughput and memory usage efficiency.

Innovative Construction of Evaluation System

Traditional evaluation methods often fail to provide effective signals in the early stages of training. This guide develops an early evaluation system that, by inserting specific probing tasks during training, can make fairly accurate predictions of final performance even when the model has only been trained on 10% of the data. These probing tasks include assessments of fundamental abilities such as vocabulary mastery, grammatical comprehension, and basic reasoning skills.

Architecture Design – Evidence-Based Component Selection

Deep Engineering Analysis of Attention Mechanisms

During the design of SmolLM3, the team conducted rigorous empirical comparisons of the three attention mechanisms. MHA theoretically possesses the strongest expressive power, but its memory consumption becomes a significant bottleneck during long sequence inference. Specific test data shows that when the sequence length reaches 8192, MHA's KV cache requires 4.2GB of memory, while GQA only requires 1.1GB.

GQA's practical validation reveals a delicate balance in the number of groups. Through extensive ablation experiments, the team found that setting the number of groups to 8 achieves the optimal balance between model performance and inference efficiency. Further analysis shows that different attention heads do indeed learn different types of attention patterns, some focusing on local dependencies and others on global relationships. GQA maintains this diversity to some extent through grouping.

Systems Engineering with Long Context Processing

Implementing document-level masks involves much more than just a technical skill. The team discovered that without document-level masks during the training data packaging process, the model learns spurious cross-document relationships, severely impacting its ability to understand long documents. Through comparative experiments, the model using document-level masks demonstrated a 15.3% performance improvement on long document QA tasks.

The choice of positional encoding is a story of technological evolution. Standard RoPE performs excellently on short sequences, but its performance collapses when extrapolating long sequences. The team tested various improvements, including Linear RoPE, YaRN, and NoPE, and ultimately chose a hybrid strategy. This strategy uses RoPE at the lower level to maintain performance on short sequences, while using NoPE at the higher level to enhance extrapolation capabilities, achieving the best of both worlds.

Data Management – A Decisive Factor in Model Capability

Scientific Principles and Practices of Data-Driven Formulation

The theoretical basis for multi-stage training stems from a deep understanding of learning dynamics. In the early stages of training, the model needs diverse data to establish a general foundation for language understanding; in the later stages of training, when the model has mastered basic capabilities, high-quality specialized data can help it overcome capability bottlenecks.

A complete pipeline has been established for data quality control. Deduplication algorithms include exact matching and semantic similarity detection, using techniques such as MinHash and SimHash to identify and remove semantically duplicated content. Quality filtering employs a multi-level filtering strategy, ranging from basic character-level filtering to complex semantic quality assessment, with clear quantitative standards at each level.

Methodological innovation in data experimentation

The design of the zero-ablation experiment demonstrates engineering ingenuity. The team discovered that for data formulation experiments, testing with a model of the target size is crucial. This is because models of different sizes have varying sensitivities to data distribution; formulations effective on smaller models will yield different results.Large ModelThe above may be completely invalid. Each data experiment is tested on a uniform evaluation set to ensure the comparability of results.

The innovation of the annealing experiment lies in its methodology for timing. By monitoring changes in the model's performance on the validation set, the team can accurately determine the optimal time to introduce new data. For example, when the model's mathematical capabilities plateau, it signals the need to introduce high-quality mathematical data.

Training Marathon – A System Engineering Execution Over a Long Period

Military-grade standards for pre-training preparation

Infrastructure validation established a comprehensive checklist. Each GPU underwent a 72-hour stress test to ensure no performance degradation occurred under prolonged high load. Network performance testing measured bandwidth and, more importantly, tested actual performance in a many-to-many communication mode, the primary communication mode for distributed training.

The monitoring system is built using a layered design. The bottom layer is hardware monitoring, which tracks the temperature, power consumption, and memory usage of each GPU in real time. The middle layer is system monitoring, which focuses on training throughput and data loading speed. The top layer is algorithm monitoring, which tracks loss curves and evaluates changes in metrics. The monitoring data from these three layers is displayed through a unified dashboard, supporting correlation analysis.

Problem-solving system during training

A systematic troubleshooting process was established for diagnosing throughput degradation. From basic data loading speed checks to intermediate network communication status analysis, and then to complex kernel performance analysis, each level has corresponding tools and methods. The team has also built a common problem knowledge base, which includes various anomaly patterns encountered historically and their solutions.

Analyzing loss anomalies requires extensive experience. The guide details different types of loss anomaly patterns: sudden spikes usually indicate data problems, slow rises may mean an excessive learning rate, and plateaus suggest a need to adjust the training strategy. For each pattern, corresponding diagnostic and remedial solutions are provided.

Post-training – Meticulous craftsmanship from base to product

A quantitative framework for post-training decision-making:

Requirements analysis no longer relies on subjective judgment but establishes a complete quantitative evaluation system. By testing the performance of the base model on different tasks, the performance gap on each task is accurately calculated, and the priority of subsequent training and resource allocation are determined.

The cost-benefit analysis model considers multiple dimensions: direct computational cost, time cost, opportunity cost, and expected performance improvement and business value. This model helps teams make the optimal choice among multiple post-training options.

Best practices in engineering for technology implementation:

The data formulation design in the SFT phase is a delicate balancing act. The instruction data needs to cover a diverse range of task types while avoiding over-representation of certain task types. The team employs a task-based hierarchical sampling strategy to ensure that each task category receives appropriate training.

Technology selection during the preference learning phase is based on extensive comparative experiments. The DPO (Dual Point of Interest) is...SimpleWhile the model performs steadily on tasks, more sophisticated reward design is needed for complex reasoning tasks. The team developed a reward model evaluation system that can accurately predict the performance of the reward model in preference learning.

Infrastructure – The Engineering Cornerstone of Large-Scale Training

In-depth optimization practices of hardware systems:

The GPU cluster's architecture is designed with the specific needs of training tasks in mind. In addition to the compute GPUs, dedicated nodes are configured for data preprocessing and checkpoint storage. The network topology employs a hybrid approach, using high-bandwidth InfiniBand between compute nodes and traditional Ethernet for management nodes.

The storage architecture design is a culmination of experience. Training data is stored using a distributed file system, supporting high-concurrency reads; checkpoints are saved using high-performance object storage, ensuring data availability even during training interruptions.fastRecovery; logs and monitoring data use a time-series database, supporting complex analytical queries.

Performance monitoringintelligentsystem:

System health monitoring not only collects data, but more importantly, establishes an early warning mechanism. Through...Machine LearningBy analyzing historical data, the algorithm enables the system to predict potential hardware failures and issue warnings before problems occur. For example, analyzing GPU temperature trends can predict fan failures; analyzing network packet loss rates can predict network card aging.

Resource estimation is based on accurate modeling of the training process. In addition to theoretical FLOPs calculations, various overheads in actual training must be considered: data loading time, gradient synchronization overhead, checkpoint saving time, etc. Real-world factors can increase training time by 20-30% compared to theoretical calculations.

In-depth analysis of practical SmolLM3 case studies:

During SmolLM3 training, the infrastructure team began cluster preparation two weeks in advance. Each server underwent 72 hours of stress testing, network performance was tuned for a week, and the storage system was specifically optimized to support high-frequency checkpointing. The initial investment paid off during training, with no interruptions due to infrastructure issues throughout the entire training cycle.

During training, the monitoring system captured 187 abnormal events, 12 of which triggered [a specific event/mechanism].automaticThe repair mechanism required manual intervention five times. The most serious incident involved an intermittent NVLink failure on a compute node, causing system...automaticThe computational tasks on this node are migrated to other nodes to ensure the continuity of the training task.

This guide concludes that the core of building high-performance large language models lies in a systematic methodology rather than simply piling on technologies. Through the complete practice of the SmolLM3 project, the team extracted the core principles that run through pre-training and post-training: making scientific decisions using a "training compass" framework, insisting on verifying each change through controlled experiments, following the "single-variable tuning" principle to avoid complex interference, and always maintaining a pragmatic attitude driven by use cases. In the pre-training stage, a reliable ablation experiment process needs to be established to cope with the challenges of scaling; in the post-training stage, attention should be paid to data balance and detailed debugging. Finally, the authors encourage developers to deepen their understanding through practical exploration, source code study, and cutting-edge research, emphasizing that behind every excellent model lies countless nights of debugging and refinement, which is precisely...open sourceA true reflection of the scientific spirit.

Original addresshttps://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook

All Categories