Mooncake - Kimi's Large-Scale Inference Framework, developed in collaboration with Tsinghua University and other institutions.

What is a mooncake?

Mooncake is the dark side of the moon.KimiMooncake is a large-model inference architecture jointly open-sourced with Tsinghua University and other institutions. It employs a distributed architecture centered around a KVCache, fully utilizing underutilized CPU, DRAM, and SSD resources in GPU clusters by separating pre-filling and decoding clusters to achieve efficient KVCache caching. Mooncake's core advantage lies in its ability to significantly improve the throughput of large-model inference while reducing computational overhead, handling high-load scenarios while maintaining service level objectives (SLOs) related to service latency. The architecture performs exceptionally well in long-context scenarios, significantly improving throughput, and supports prediction-based early rejection strategies to optimize resource allocation under overload conditions. The Mooncake project is open-source on GitHub, driving the development of efficient inference platforms for large-model technology.

Main functions of Mooncake

• High-efficiency large model inferenceMooncake optimizes the inference process for large models through its distributed architecture, significantly improving inference throughput, especially when dealing with long contextual data.
• KVCache centralized designCentered on KVCache, Mooncake achieves efficient data caching and reuse, reducing reliance on GPU resources and lowering computing power overhead.
• Pre-filling and Decoding SeparationThe architecture separates the prefill and decode stages, allowing resources to be optimized for the computational characteristics of different stages.
• Resource optimizationThrough its split design, Mooncake can more effectively utilize CPU, DRAM, and SSD resources, thus improving resource utilization.
• Load balancingMooncake implements a cache-based load balancing strategy, which improves cache hit rate and system load balancing through an automatic hotspot migration scheme.
• Overload ManagementWhen faced with high load conditions, Mooncake employs a prediction-based early rejection strategy to optimize resource allocation and reduce unnecessary computation.
• High-performance transmissionBased on RDMA technology, Mooncake achieves high-speed KVCache transmission across nodes, reducing latency.
• Standardized InterfaceMooncake provides a standard interface for creating new high-performance memory semantic storage for the era of large models, and offers a reference implementation scheme.
• Cost reductionBy optimizing the inference process and resource utilization, Mooncake helps reduce the cost of inference for large models, making AI technology more economical and efficient.

The technical principle of mooncake

• Distributed architectureBy utilizing the CPU, DRAM, and SSD resources in the GPU cluster, distributed storage and transmission of KVCache are achieved, which improves cache capacity and transmission bandwidth and reduces dependence on a single GPU resource.
• Global Scheduler (Conductor): Responsible for scheduling requests based on the current KVCache distribution and workload, and deciding whether to copy or swap KVCache blocks to optimize overall throughput and meet service level objectives (SLO).
• Chunked Pipeline ParallelismFor long context requests, the input tokens are divided into multiple blocks and processed in parallel on different nodes to reduce latency.
• Layer-wise prefillAsynchronous loading and storage of KVCache reduces VRAM usage through overlapping transfers and computations.
• Cache-aware schedulingMooncake's scheduling algorithm takes into account KVCache reuse, prefilling time, and instance load queuing time to achieve efficient request scheduling.

Mooncake's project address

• Github repository:https://github.com/kvcache-ai/Mooncake
• arXiv technical paper:https://arxiv.org/pdf/2407.00079

Applications of Mooncake

• Natural Language Processing (NLP)Mooncake can be used to support various NLP tasks, such as language translation, text summarization, sentiment analysis, question answering systems, and chatbots.
• Content recommendation systemIn recommendation systems, Mooncake can be used to process user behavior data and content features to provide personalized recommendations.
• Search enginesMooncake can be used to improve search engine query understanding and document ranking by understanding complex query intent and document content, thus providing more accurate search results.
• Speech recognition and generationIn the field of speech recognition, Mooncake can be used to improve the accuracy of speech-to-text conversion; in the field of speech generation, it can generate more natural and fluent speech output.
• Image and video analysisEfficient reasoning ability can also assist in image and video analysis tasks, such as image annotation and video content understanding.
• Intelligent customer service and virtual assistantMooncake offers robust backend support, and its intelligent customer service and virtual assistant can handle complex conversations and tasks.