KTransformers - Tsinghua University's open-source framework for optimizing large language model inference

What are KTransformers?

KTransformers is an open-source project launched by the KVCache.AI team at Tsinghua University in collaboration with Qujing Technology. It optimizes the inference performance of large language models and lowers the hardware requirements. Based on a GPU/CPU heterogeneous computing strategy and leveraging the sparsity of the MoE architecture, KTransformers supports running DeepSeek-R1 and V3 (671B full-power version) on a single GPU with only 24GB of VRAM. Preprocessing speeds reach up to 286 tokens/s, and inference generation speeds reach up to 14 tokens/s. The project significantly improves inference speed through computationally intensive offload strategies, high-performance operators, and CUDA Graph optimizations.

Main functions of KTransformers

• Support for local inference of very large modelsIt supports running full-power large models with 671B parameters such as DeepSeek-R1 on a single graphics card with only 24GB of video memory, breaking the traditional hardware limitations.
• Improve reasoning speedThe preprocessing speed can reach up to 286 tokens/s, and the inference generation speed can reach 14 tokens/s.
• Compatible with multiple models and operatorsSupports DeepSeek series and other MoE architecture models, provides a flexible template injection framework, supports users to switch quantization strategies and kernel replacement, and adapts to different optimization needs.
• Lowering the hardware thresholdIt significantly reduces the memory requirements of large models, enabling ordinary users and small and medium-sized teams to run models with hundreds of billions of parameters on consumer-grade hardware, achieving "home-like" deployment.
• Support long sequence tasksIntegrating the Intel AMX instruction set, the CPU prefill speed can reach 286 tokens/s, which is 28 times faster than traditional solutions, reducing the processing time of long sequence tasks from "minutes" to "seconds".

The technical principles of KTransformers

• MoE architectureThe sparse MoE matrix is offloaded to the CPU/DRAM for processing, while the dense part is retained on the GPU, which greatly reduces the demand for video memory.
• Offload strategyTasks are allocated to GPUs and CPUs based on computational intensity: high-intensity tasks (such as MLA operators) are preferentially allocated to GPUs, while low-intensity tasks are allocated to CPUs.
• High-performance operator optimization:
  • CPU side: Using llamafile as the CPU core, combined with optimizations such as multithreading, task scheduling, and load balancing, improves CPU inference efficiency.
  • On the GPU side: The Marlin operator is introduced to optimize quantization matrix calculations, achieving a speedup of 3.87 times compared to traditional libraries (such as Torch).
• CUDA Graph OptimizationLeveraging CUDA Graph reduces Python call overhead, minimizes CPU/GPU communication breakpoints, and enables efficient heterogeneous computing collaboration. Each decode operation requires only one complete CUDA Graph call, significantly improving inference performance.
• Quantization and storage optimizationEmploying 4-bit quantization technology further reduces model storage requirements, requiring only 24GB of video memory to run a 671B parameter model. Simultaneously, the KV cache size is optimized to reduce storage overhead.
• Template Injection FrameworkIt provides a YAML-based template injection framework, allowing users to flexibly switch quantization strategies, kernel replacements, and other optimization methods to adapt to the needs of different scenarios.

KTransformers project address

• GitHub repository:https://github.com/kvcache-ai/ktransformers

Application scenarios of KTransformers

• Individual development and small to medium-sized teamsRun large models on consumer-grade hardware to develop text generation, question-answering systems, etc., reducing costs.
• Long sequence tasksIt efficiently processes long texts and performs code analysis, reducing processing time from minutes to seconds.
• Enterprise applicationsLarge models can be deployed locally for intelligent customer service, content recommendation, etc., saving cloud service costs.
• academic researchTo explore and optimize the MoE architecture model on ordinary hardware, thereby accelerating the research process.
• Education and TrainingAs a teaching tool, it helps students practice large-scale model applications and understand optimization techniques.