MT-TransformerEngine - A high-efficiency training and inference optimization framework open-sourced by Moore's Threads.

What is MT-TransformerEngine?

MT-TransformerEngine is an open-source, high-efficiency training and inference optimization framework from Moore's Threads, designed specifically for Transformer models. The framework leverages operator fusion and parallel acceleration techniques to fully utilize the computational potential of Moore's Threads' full-featured GPUs, significantly improving training efficiency. It supports FP8 mixed-precision training, further optimizing performance and stability by utilizing the GPU's native FP8 computing power. MT-TransformerEngine works in conjunction with MT-MegatronLM to achieve efficient hybrid parallel training, suitable for large models such as BERT and GPT.

Main functions of MT-TransformerEngine

• High-efficiency training accelerationBy integrating multiple computationally intensive operations, it reduces memory access and computational overhead, significantly improving training efficiency. It supports data parallelism, model parallelism, and pipeline parallelism, fully utilizing the computing resources of GPU clusters.
• Inference optimizationOptimize the inference phase of the Transformer model to reduce latency and improve throughput. Reduce memory usage during inference by optimizing memory management.
• Collaboration with ecological tools
  • Synergy with MT-MegatronLMIt supports integration with the MT-MegatronLM framework to achieve more efficient hybrid parallel training.
  • Integration with MT-DualPipeCombined with MT-DualPipe, it further optimizes the utilization of computing resources.
  • Support Torch-MUSAIt relies on the Torch-MUSA deep learning framework and MUSA software stack, and is compatible with a variety of algorithms.
• Multimodal model supportIt supports training multimodal Transformer models, making it suitable for complex tasks involving multiple modalities such as text and images.
• Communication optimizationBy optimizing communication strategies, communication latency between GPUs is reduced, thereby improving overall training efficiency.

Technical principles of MT-TransformerEngine

• Operator fusionMT-TransformerEngine incorporates various operator fusion optimizations tailored to the characteristics of Transformer models. It integrates normalization layers with QKV horizontal fusion, self-attention computation fusion, and residual connections with fully connected layers, reducing memory accesses and CUDA kernel startup time during computation, thereby improving model inference performance.
• Parallel acceleration strategy
  • Hybrid Parallel TrainingIt supports multiple parallel strategies, including data parallelism, tensor parallelism, and pipelined parallelism. Tensor parallelism achieves distributed computing across multiple GPUs by splitting the matrix operation dimensions of the attention layer and the multilayer perceptron; pipelined parallelism divides the model into multiple stages and improves throughput through micro-batch delivery.
  • Reduce bubble rateBy integrating with technologies such as MT-DualPipe and DeepEP, it significantly reduces "pipeline bubbles" and further improves parallel training efficiency.
• FP8 Mixed Precision TrainingThis deeply integrates the FP8 mixed-precision training strategy natively supported by Moore's Threads GPUs. During training, low-precision computation based on FP8 accelerates training while maintaining numerical stability through specific techniques, reducing memory usage and significantly improving training efficiency.
• High-performance operator libraryIt integrates the high-performance operator library muDNN, which is deeply optimized for GPUs to improve computational efficiency.

The project address for MT-TransformerEngine

• Github repository:https://github.com/MooreThreads/MT-TransformerEngine

Application scenarios of MT-TransformerEngine

• Large-scale language model trainingMT-TransformerEngine is suitable for training large-scale language models such as GPT, BERT, and T5. Through its efficient parallelization technology and FP8 mixed-precision training strategy, it can efficiently train models with billions or even hundreds of billions of parameters on large-scale GPU clusters.
• Multimodal model trainingThe framework supports training multimodal Transformer models and can handle data containing multiple modalities such as text, images, and videos. For example, Meta-Transformer uses an Encoder with frozen weights to perform multimodal awareness even without paired multimodal training data.
• Real-time inferenceIn real-time inference scenarios requiring low latency, the MT-TransformerEngine can significantly improve inference speed through its optimized inference engine and FP8 accuracy, making it suitable for tasks such as natural language processing and image recognition.