openPangu 2.0 - An upgraded version of Huawei's open-source Pangu large model

What is openPangu 2.0?

openPangu 2.0 is a major upgrade to Huawei's open-source Pangu large model, providing a Pro version with up to 505B parameters and a Flash version with 92B parameters. The model is uniformly equipped with a 512K ultra-long context, and natively optimized for Ascend and Kirin chips. The model structure, weights, technical reports, inference code, pre-training code, post-training code and training operators will be gradually released across the entire chain starting from June 30.

Main features of openPangu 2.0

• Dual version matrixThe Pro version is designed for extreme performance, while the Flash version is designed for lightweight deployment.
• 512K Extremely Long ContextIt provides unified support for understanding extremely long texts, analyzing complex code, and cross-application agent interactions.
• Ascend native optimizationThe single-card inference throughput is twice that of mainstream open-source models in the industry, with lower latency.
• HarmonyOS Agent AdaptationTasks are completed faster and with higher accuracy, while token consumption and resource usage are significantly reduced.
• End-side entry modelThe model is a Kirin affinity model with 30B total parameters / 2B activation, supporting local inference on mobile phones and IoT devices.

Technical principles of openPangu 2.0

• DSA+SWA Independent Layered Hybrid ArchitectureThe industry's first architecture to adopt an independent layered hybrid architecture, which separately deploys Dynamic Sparse Attention (DSA) and Sliding Window Attention (SWA) to achieve an ultra-sparse attention mechanism. While maintaining the ability to understand 512K long contexts, the architecture significantly reduces redundant attention overhead through layered computation, resulting in a non-linear decrease in computational power consumption for ultra-long sequence inference.
• Extremely Sparse Activation and Expert RoutingEmploying a 28:1 sparsity ratio, the Pro version, with a total of 505B parameters, only requires 18B to complete inference. Through expert-optimized routing and precise computational allocation, the model achieves consistency exceeding [a certain level] during both training and inference phases. 99%This avoids the training-inference distribution offset problem common in traditional MoE models, and significantly reduces inference cost and memory usage.
• Ascend native full-stack optimizationFrom algorithm architecture to training and inference, it is fully adapted to Ascend chips. On the training side, supernode affinity optimization is implemented, improving training efficiency by 30% and increasing throughput for 512K long sequence training. 50%The inference side integrates high-precision native operators such as mHC, Muon, and ModAttn, achieving a single-card throughput that is twice that of mainstream open-source models in the industry, thus achieving a balance between high throughput and low latency.
• End-side quantitative pruning and expert reuseThe on-device model undergoes native quantization pruning specifically for the Kirin chip, resulting in a 50% speedup inference and a 20% reduction in memory usage. An expert reuse loss mechanism is introduced, reducing expert switching frequency by 50%. Combined with activated expert prediction technology, the on-device model throughput is increased by 5 times, enabling high-performance local inference on mobile phones and IoT devices.

How to use openPangu 2.0

The model weights, inference code, and training code can be downloaded and used from the official website on June 30th.

The core advantages of openPangu 2.0

• Native Computing AffinityDeeply compatible with Ascend and Kirin chips, allowing you to unleash peak hardware performance without additional conversion.
• Extreme sparsityThe sparse ratio of 28:1 significantly reduces inference load, enabling large model capabilities with smaller activation parameters.
• Extremely long context unificationA 512K context window covers long documents, multi-turn dialogues, and complex agent tasks.
• Training pipeline fully openIt is one of the few complete open-source pre-training and post-training code sources in the industry, which lowers the threshold for secondary development.
• End-to-end cloud collaborative coverageThe cloud-side Ascend + edge-side Kylin dual architecture enables full-scenario deployment from data centers to mobile devices.

Comparison of openPangu 2.0 with similar competing products

Application scenarios of openPangu 2.0

• Enterprise Knowledge BaseIn-depth analysis of ultra-long legal contracts, research papers, and financial reports using 512K context processing.
• HarmonyOS Intelligent Agent: Build a cross-application scheduling agent in the HarmonyOS ecosystem to achieve automated device control and multimodal interaction.
• Ascend Cloud ReasoningDeploy cost-effective large-model API services based on Ascend clusters to reduce the inference cost of domestic computing power.
• End-side privacy computing: Enables local document processing and offline smart assistant on mobile devices through the 30B input model.
• Vertical Industry Customization: Leveraging open-source training code, quickly fine-tune specialized models for fields such as manufacturing, healthcare, and government.