Chitu - A large-scale model inference engine jointly open-sourced by Tsinghua University and Tsinghua Jizhi.

What is Chitu?

Chitu is a high-performance large-model inference engine jointly open-sourced by the Institute of High-Performance Computing at Tsinghua University and Tsinghua Jizhi. It is specifically designed to address the high cost and low efficiency issues in the inference phase of large models, boasting strong hardware adaptability. It supports multiple NVIDIA GPUs and domestically produced chips, breaking the dependence on specific hardware (such as the NVIDIA Hopper architecture). In terms of performance, when deploying DeepSeek-R1-671B on an A800 cluster, compared to some foreign open-source frameworks, GPU usage is reduced by 50%, and inference speed is increased by 3.15 times. It supports deployment across all scenarios, from pure CPU to large-scale clusters, meeting the inference needs of different scales and scenarios.

Chitu's main functions

• Multi-dimensional computing power adaptationIt supports multiple series of NVIDIA GPUs, from the latest flagship to older models, while also providing optimized support for domestic chips, breaking the dependence on NVIDIA's Hopper architecture.
• Scalable across all scenariosFrom pure CPU deployment and single GPU deployment to large-scale cluster deployment, Chitu Engine can provide scalable solutions to meet inference needs of different scales and scenarios.
• Low latency optimizationFor latency-sensitive scenarios, such as financial risk control, optimize model inference speed and reduce response time.
• High throughput optimizationIn high-concurrency scenarios, such as intelligent customer service, increase the number of requests processed per unit of time.
• Small memory optimizationReduce single-card memory usage, enabling enterprises to achieve higher inference performance with fewer hardware resources.
• Long-term stable operationThe Chitu engine can be applied to real-world production environments, and its stability is sufficient to handle concurrent business traffic.
• Ready to use right out of the boxQingcheng Jizhi has launched an inference all-in-one machine based on Chitu, providing out-of-the-box deployment solutions and professional operation and maintenance services, further simplifying the process of enterprise AI implementation.

Chitu's technical principles

• Underlying technological innovationChitu Engine, through underlying technological innovation, has for the first time enabled native execution of FP8 precision models on non-NVIDIA Hopper architecture GPUs and various domestically produced chips. This breaks the past dependence of FP8 models on NVIDIA H-series high-end GPUs, allowing enterprises to efficiently deploy large models on a wider range of hardware types.
• Operator-level optimizationChitu has performed instruction-level optimizations on key operators (such as GeMM and MoE), directly processing FP8 data instead of simple quantization. This ensures lossless model accuracy while significantly improving inference speed. In tests on the A800 cluster, compared to some foreign open-source frameworks, the Chitu engine still achieved a 3.15x speed improvement in inference speed with a 50% reduction in GPU usage.
• Full-scenario performance optimizationThe Chitu engine supports low latency, high throughput, and small video memory optimization, providing optimal solutions based on different scenario requirements, hardware configurations, and system environments. Chitu can find the best balance between GPU utilization, memory efficiency, and network transmission based on system resource conditions.
• Parallel computing and compiler optimizationThe Chitu engine embodies years of experience in parallel computing and compiler optimization from a Tsinghua University team. Through intelligent compilation technology, the team is able to accelerate the development process of high-performance operators and achieve optimization for different hardware architectures in a shorter time.

Chitu's project address

• Github repository:https://github.com/thu-pacman/chitu

Chitu's application scenarios

• Risk identification and early warningChitu Engine can quickly process massive amounts of transaction data, monitor potential risks in real time and issue timely warnings, helping financial institutions improve risk management efficiency.
• Intelligent customer service and customer experience optimization: By leveraging a large-scale intelligent knowledge base, we can quickly respond to customer needs and enhance the service experience.
• Disease Diagnosis AssistanceThe Chitu engine can quickly process medical data, improve the speed and accuracy of disease diagnosis, and enhance the patient's medical experience.
• Traffic flow optimizationThe Chitu engine can process traffic data in real time, optimize traffic flow, and alleviate urban congestion.
• Scientific data analysisThe Chitu engine can efficiently process scientific research data and accelerate the scientific research process.