AB
AB
AiBoss

All Categories

project

HPC-Ops - Tencent Hunyuan's open-source industrial-grade high-performance large model inference operator library

HPC-Ops is an open-source industrial-grade high-performance large model inference operator library from Tencent Hunyuan AI Infra team. It covers core modules such as Attention, MoE, GEMM, sampling and communication fusion, and natively supports BF16/FP8 and mixed precision.

What is HPC-Ops?

HPC-Ops is an open-source, industrial-grade, high-performance large-model inference operator library from Tencent Hunyuan AI Infra team. It covers core modules such as Attention, MoE, GEMM, sampling, and communication fusion, and natively supports BF16/FP8 and mixed precision. HPC-Ops is deeply optimized for GPUs such as the NVIDIA H20, with dynamic scheduling and operator fusion resulting in an end-to-end QPM improvement of over 30%. Many performance metrics significantly surpass mainstream baselines such as vLLM, FlashInfer, and SGLang, and it already supports Tencent Hunyuan's large-scale production-level inference services.

Main functions of HPC-Ops

  • Dynamic Attention SchedulingFor online inference loads with mixed long and short requests, runtime tile-level dynamic task scheduling is adopted to achieve CTA-level load balancing, with a maximum speedup of 2.95 times for long text scenarios.
  • Router GEMMIt uses a combination of dual BF16 GEMMs to simulate FP32 precision computation, and offline weight splitting and inference stage fusion into a single kernel, which balances high precision in numerically sensitive scenarios with high throughput of Tensor Cores.
  • FusedMoEThe five stages of routing index, Gate-Up GEMM, activation quantization, Down GEMM, and Top-K weighted aggregation are reconstructed into a bubble-free pipeline, eliminating the overhead of memory transfer and kernel startup.
  • Fused AllReduce+NormIt deeply integrates cross-GPU communication, residual addition, and RMSNorm normalization, and achieves integrated communication and computing based on NVLink multicast and P2P technology, alleviating the bottleneck of tensor parallelism.
  • Sampler Large Operator FusionIt integrates more than 10 kernels, including repetition penalty, temperature scaling, Top-K, Top-P, Softmax, and random sampling, into two CUDA kernels to fill the gap in post-processing at the end of inference.
  • GroupGEMM FP8Supports expert parallelism and grouped expert matrix multiplication, natively supports per-tensor and block-wise FP8 quantization, and is adapted for efficient inference of MoE models.

The technical principle of HPC-Ops

  • Attention Dynamic SchedulingTraditional static split-kv cannot handle dynamic loads with mixed long and short requests. HPC-Ops splits all requests at a uniform tile granularity, evenly distributes the size of each CTA task through the total global tile count, and then uses a greedy bucketing algorithm to achieve the most even distribution. The Task Assign module generates a dedicated task mapping table before each inference. Each layer of Attention Kernel accurately picks up tasks according to the table, and finally the Combine Kernel merges the results to achieve load balancing throughout the process.
  • Router GEMM Dual BF16 Analog FP32For precision-sensitive scenarios such as MoE routing, the FP32 weights are split offline into high-order BF16 and low-order residual BF16 (scaling factor 1/256). During the inference phase, the BF16 Tensor Core GEMM is executed twice and linearly combined, and the activation value remains BF16 throughout. The dual-path computation is merged into a single kernel, the input is only moved once, the intermediate results are cached by dual register accumulators, and the results are written out after one FMA correction in the Epilogue phase, with no HBM round-trip overhead throughout the process.
  • FusedMoE pipeline refactoringThe routing phase uses shared memory block statistics, reserving a continuous video memory output range for each expert; Gate-Up GEMM directly reads the original input through the routing index, eliminating the need for independent Gather transport; Warp Specialization is eliminated, and data transport and calculation are completed by the same Warp Group, upgrading the memory access latency masking logic from the software pipeline within the CTA to cross-CTA hardware scheduling; PDL technology connects the entire link for bubble-free execution.
  • Fused AllReduce + Norm Communication Computing FusionBased on CUDA multimem and P2P technology, it is encapsulated into a native NVLink unified operation. RMSNorm(AllReduce(x) + residual, weight)The high-throughput version relies on the NVSwitch multicast mechanism to adapt to the Prefill scenario, while the low-latency version is based on the Lamport P2P mechanism to achieve dual-kernel overlapping execution through PDL, adapting to the Decode scenario.

How to use HPC-Ops

  • Hardware environmentPrepare a server equipped with an NVIDIA SM90 architecture GPU (such as H20) and install Python 3.8+, CUDA 12.8+, and C++17 compiler.
  • Cloning repositoryClone the HPC-Ops source code repository from GitHub to your local machine.
  • Compile and packageEnter the project directory and execute the compile and package command to generate the wheel installation file.
  • Installation and DeploymentUse the pip command to install the generated wheel file and complete the environment deployment.
  • Import ModuleImport the hpc module into your Python script.
  • Prepare dataPrepare input tensors and configuration parameters that meet the accuracy requirements based on the target operator type.
  • Calling Operator: Call the corresponding Python API (such as Attention, FusedMoE, Sampler, etc.) to perform high-performance inference.
  • Test and verificationRefer to the test scripts in the tests directory to verify the correctness and specific usage of the operators.
  • Framework integration: Seamlessly embed HPC-Ops into mainstream inference frameworks such as vLLM and SGLang through the provided Python API.
  • Performance benchmarkUse the scripts in the benchmark directory to run performance reproduction and comparison tests on the target hardware.

The core advantages of HPC-Ops

  • End-to-end end-to-end optimizationIt covers the entire inference chain from Attention → Router → MoE → Communication Convergence → Sampler, resulting in an end-to-end QPM improvement of 30%+, rather than single-point operator optimization.
  • Dynamic load balancing eliminates long tailsRuntime Tile-level dynamic scheduling and greedy bucketing algorithm can accelerate long text by up to 2.95 times in scenarios with mixed long and short requests, and improve end-to-end QPM by 17%.
  • High-precision Router GEMMThe dual BF16 combination simulates FP32 precision with an error of only 1/36 of TF32, and its performance is up to 3.22 times faster than cuBLAS FP32, balancing numerical sensitivity and computational power release.
  • MoE bubble-free production lineFusedMoE reconstructs the five stages of routing, Gate-Up, quantization, Down, and Top-K aggregation into a single execution link, eliminating the overhead of memory transfer and kernel startup, resulting in a 1.2x~1.6x improvement over vLLM/SGLang.
  • Deep integration of communication and computingFused AllReduce+Norm is encapsulated as a native NVLink integrated operation, breaking through the bottleneck of tensor parallelism and achieving up to 1.68 times faster speedup compared to NCCL/FlashInfer.

HPC-Ops project address

  • GitHub repository:https://github.com/Tencent/hpc-ops

HPC-Ops Comparison with Similar Products

Dimension HPC-Ops FlashInfer
Project nature Tencent's Hunyuan open-source and long-term maintained industrial-grade operator library has been validated in large-scale production. A community-driven, open-source high-performance operator library that is widely integrated into frameworks such as vLLM and SGLang.
Attention Dynamic Scheduling Supports runtime tile-level dynamic task allocation and greedy bucketing, achieving maximum speedup for long text in scenarios with mixed long and short requests. 2.95xMixed batch acceleration 1.59x~1.76x The primary scheduling method is static split-k, which exhibits stable performance under uniformly long loads, but is prone to CTA-level long-tail latency when dealing with mixed long and short loads.
Attention Standard Performance BF16 Prefill (Highest) 1.33xDecode 2.22xFP8 Decode 2.0x(Relative to FlashInfer baseline) As a mainstream baseline in the community, it exhibits excellent performance, but there is room for optimization under dynamic loads and sparse scenarios.
Sparse Attention FP8 block sparse prefill supports pre-computed block masks to skip irrelevant KV tiles, offering the highest performance relative to baselines like FlashInfer. 3.16x It supports block sparse mode, but HPC-Ops offers a more refined implementation of sparse scheduling and tile quantization at FP8 precision.
MoE Fusion FusedMoE reconstructs the routing index → Gate-Up → Quantization → Down → Top-K aggregation into a bubble-free pipeline, achieving superior performance. 1.2x~1.6x It provides basic MoE support, but lacks full-module pipeline integration, and incurs overhead from multi-stage memory relocation and kernel startup.
Router GEMM Unique dual BF16 analog FP32 precision, with an error only 1/36th that of TF32, superior performance. 3.22xSuitable for numerically sensitive routing modules There is no dedicated high-precision Router GEMM optimization; it requires cuBLAS FP32 or acceptance of BF16/TF32 accuracy loss.
Communication and computing convergence Fused AllReduce+Residual+RMSNorm is encapsulated into a native NVLink unified operation, offering the highest performance relative to FlashInfer. 1.68x~1.76x It provides basic communication operators, but lacks native integration of AllReduce, residual, and normalization.

Application scenarios of HPC-Ops

  • High-throughput online inference serviceDynamic Attention scheduling combined with Sampler significantly reduces long-tail latency, making it suitable for real-world business loads with mixed long and short requests.
  • MoE Large Model InferenceFusedMoE is a deep optimization for MoE models such as DeepSeek-V3 and Qwen3-235B, improving inference efficiency in TP/EP parallel scenarios.
  • Tensor Parallel Distributed ReasoningFused AllReduce + Norm breaks through the communication bottleneck of multiple GPUs, making it suitable for large-scale model deployments on a single node with multiple GPUs.
  • Precision-sensitive inferenceRouter GEMM achieves FP32 precision with BF16 computing power, meeting the needs of numerically sensitive modules such as MoE routing and sparse attention.
Back