DeepEP - DeepSeek's open-source expert parallel communication library, designed specifically for MoE training and inference.

What is DeepEP?

DeepEP is DeepSeek's first open-source EP (Expert Parallel) communication library specifically designed for training and inference of hybrid expert models (MoE). It provides high-throughput, low-latency all-to-all GPU kernels, supporting intra-node and inter-node NVLink and RDMA communication. DeepEP is specifically optimized for the group-restricted gating algorithm in the DeepSeek-V3 paper, supports FP8 data format scheduling, and introduces a hook-based communication-computation overlap method that does not consume GPU computing resources. The low-latency kernel performs exceptionally well in the inference and decoding stages, with latency as low as 163 microseconds. DeepEP is compatible with the Hopper GPU architecture and requires Python 3.8, CUDA 12.3, and PyTorch 2.1 or later.

DeepEP's main functions

• High-efficiency communication kernelDeepEP provides high-throughput and low-latency all-to-all GPU kernels suitable for MoE dispatch and combine operations.
• Low-precision calculation supportIt supports low-precision data formats such as FP8 and BF16, significantly improving computing efficiency and reducing memory requirements.
• Optimized communication mechanismDeepEP provides an optimized kernel for the group-restricted gating algorithm proposed in the DeepSeek-V3 paper, supporting asymmetric bandwidth forwarding from NVLink to RDMA, suitable for training and inference pre-filling tasks.
• Low-latency inference decodingIt provides a low-latency kernel with pure RDMA, which is particularly suitable for latency-sensitive inference and decoding scenarios, with latency as low as 163 microseconds.
• Communication and computing overlap: Introducing a Hook-based communication-computation overlap method that does not occupy GPU streaming multiprocessor (SM) resources and maximizes computational efficiency.
• Flexible resource managementIt supports flexible GPU resource management, allowing users to control the number of SMs used, adapting to different workloads.
• Network configuration optimizationDeepEP has been fully tested on the InfiniBand network and supports traffic isolation via Virtual Channels (VLs) to prevent interference between different types of traffic.

DeepEP's project address

• Github repository:https://github.com/deepseek-ai/DeepEP

DeepEP's performance

• High throughput kernelDeepEP was tested on an H800 GPU and a CX7 InfiniBand 400 Gb/s RDMA network card, demonstrating excellent throughput performance.
  • Internal node communicationIn intra-node communication using NVLink, the bottleneck bandwidth for distribution and merging operations reaches [values missing]. 153 GB/s and 158 GB/s.
  • Cross-node communicationIn cross-node communication using RDMA, the bottleneck bandwidth for distribution and merging operations reaches [values missing]. 43-47 GB/s.
• Low-latency kernelDeepEP's low-latency kernel is designed specifically for inference decoding, using pure RDMA technology to significantly reduce latency.
  • In processing 8 experts At that time, the delay of the distribution operation is 163 microsecondsThe delay of the merge operation is 318 microsecondsRDMA bandwidth is 46 GB/s.
  • As the number of experts increased, delays rose slightly, but... 256 experts At that time, the delays for distribution and merging operations are respectively 194 microseconds and 360 microseconds.
• System compatibilityDeepEP is primarily compatible with InfiniBand networks, but also supports running on Converged Ethernet (RoCE). It requires a Hopper architecture GPU, Python 3.8 or later, CUDA 12.3 or later, and PyTorch 2.1 or later.

DeepEP System Requirements

• Hardware Requirements:
  • GPUs that support the Hopper architecture (such as H100 and H800) may support more architectures in the future.
  • Devices that support GPUDirect RDMA are required; please refer to the NVSHMEM hardware specifications for specific requirements.
  • Intra-node communication requires NVLink, while inter-node communication requires an RDMA network.
• Software Requirements:
  • Python 3.8 and above.
  • CUDA 12.3 and above.
  • PyTorch version 2.1 and above.
  • A modified version of NVSHMEM needs to be installed. Please refer to the relevant documentation for specific installation instructions.
  • We recommend installing GDRCopy (v2.4 and above) for low-latency GPU memory copying.
• Network requirements:
  • The primary testing environment was an InfiniBand network, compatible with RDMA over Converged Ethernet (RoCE).
  • It supports traffic isolation via Virtual Channels (VLs) to prevent interference between different workloads.
• Other requirements:
  • In a containerized environment, it is necessary to ensure that the host has loaded the necessary kernel modules (such as...). gdrdrvThe relevant DEB packages were correctly installed.
  • After installation, you need to set environment variables (such as...). NVSHMEM_DIR(For use by DeepEP)

Application scenarios of DeepEP

• Large-scale model trainingDeepEP provides efficient parallel communication support, suitable for training hybrid expert models (MoE), and significantly improves training efficiency.
• Reasoning taskSuitable for latency-sensitive inference and decoding scenarios, it can significantly reduce latency and improve inference throughput.
• High-performance computingIt supports multiple hardware platforms, including the Hopper GPU architecture, and optimizes communication performance for NVLink and RDMA networks.
• Intelligent Customer ServiceBy optimizing the reasoning process, DeepSeek's intelligent customer service system can quickly respond to user questions and improve service efficiency.
• Financial sectorIt is used for risk assessment, automated report generation, etc., and predicts the probability of default by analyzing corporate financial reports and public opinion data.