Profiling Data - Performance analysis data from the DeepSeek open-source training and inference framework

What is Profiling Data?

Profiling Data is performance analysis data from DeepSeek's open-source training and inference framework. Based on detailed information captured during program execution by PyTorch Profiler, it's used to analyze and optimize software performance. After downloading, it can be directly used in Chrome or Edge browsers. chrome://tracing or edge://tracing Open the documentation for visualization analysis. Profiling data records key metrics such as program execution time, resource utilization, and communication patterns. In deep learning frameworks, profiling data helps developers understand computational and communication overlap strategies during model training and inference, the usage of different hardware resources, and potential performance bottlenecks. Based on data analysis, developers can optimize code implementation, adjust parallelization strategies, and improve the overall efficiency of the system.

Main functions of Profiling Data

• Performance bottleneck identificationBased on recording the time consumption and resource usage during program execution, it helps developers quickly locate performance bottlenecks, such as which functions or modules consume too much time or resources.
• Resource Utilization AnalysisIt analyzes the usage of hardware resources such as CPU, GPU, and memory to help optimize resource allocation and avoid resource waste.
• Communication mode analysisIn distributed systems, we analyze the communication patterns and latency between nodes to optimize communication strategies and reduce communication overhead.
• Optimization GuideIt provides data support for code optimization, helping developers choose appropriate optimization strategies, such as parallelization and caching optimization.

The reasoning process of profiling data

• PrefillingBased on EP32 and TP1 (consistent with the actual online deployment of DeepSeek V3/R1), the hint length is 4K, and the batch size per GPU is 16K tokens. Two micro-batches are used for overlapping computation and fully connected communication to ensure balanced attention computation load between the two micro-batches.
• DecodingBased on EP128 and TP1, the hint length is 4K, and the batch size per GPU is 128 requests. Similar to pre-padding, decoding also uses two micro-batches for overlapping computation and fully connected communication. Unlike pre-padding, the fully connected communication during the decoding phase does not occupy GPU SMs (Streaming Multiprocessors). After the RDMA (Remote Direct Memory Access) message is issued, all GPU SMs are released, and the system waits for the fully connected communication to complete after computation.

Project address for Profiling Data

• GitHub repository:https://github.com/deepseek-ai/profile-data

Application scenarios of Profiling Data

• Identify performance bottlenecksQuickly identify the most time-consuming or resource-intensive parts of a program.
• Guide performance optimizationAdjust strategies based on data to improve computing and communication efficiency.
• Analysis of resource utilizationAssess the usage of hardware resources (such as CPU, GPU, and memory) to avoid waste.
• Optimize system architectureAdjust the configuration and communication strategies of the distributed system based on the runtime data.
• Detection performance regressionCompare performance data from different versions to ensure that optimizations are effective and do not result in performance degradation.