3FS - DeepSeek's open-source high-performance distributed file system

What is 3FS?

3FS (Fire-Flyer File System) is a high-performance distributed file system from DeepSeek, designed specifically for AI training and inference tasks. Utilizing modern SSDs and RDMA network technology, 3FS aggregates the throughput of thousands of SSDs and the network bandwidth of hundreds of storage nodes based on a discrete architecture, delivering a read throughput of up to 6.6 TiB/s. 3FS provides strong consistency guarantees and a universal file interface, eliminating the need to learn new storage APIs. 3FS excels in large-scale data processing and inference optimization, achieving a throughput of 3.66 TiB/min in GraySort tests and a KVCache read throughput of up to 40 GiB/s.

3FS main functions

• High-performance data accessIt aggregates the throughput of thousands of SSDs and the network bandwidth of hundreds of storage nodes, providing a read throughput of up to 6.6 TiB/s. It supports high-throughput parallel read and write in large-scale clusters, optimizing data loading efficiency in AI training and inference tasks.
• Strong consistency guaranteeImplement Chained Replication and Assigned Query (CRAQ) technology to ensure strong data consistency and simplify application development complexity.
• General file interfaceIt provides stateless metadata services and supports transactional key-value stores (such as FoundationDB), so users do not need to learn new storage APIs.
• Optimize AI workloads:
  • Data preparationIt efficiently manages a large number of intermediate outputs and supports hierarchical directory structures.
  • Data loadingIt supports random access across compute nodes without prefetching or dataset shuffling.
  • Checkpoint supportProvides high-throughput parallel checkpointing capabilities for large-scale training.
  • KVCacheIt provides a high-throughput, high-capacity caching alternative for inference tasks, optimizing inference efficiency.
• High scalability and flexibilityIt supports large-scale cluster deployment and is suitable for diverse application scenarios ranging from single nodes to thousands of nodes.

3FS technical principles

• Decoupled architectureBased on a design that separates compute and storage, storage resources are centrally managed, and high-speed networks (such as RDMA) are used to achieve efficient data transmission. This allows applications to access storage resources in a "location-independent" manner, simplifying resource management.
• Chained Replication and Assignment Query (CRAQ)To achieve strong consistency, 3FS is based on CRAQ technology. Chained replication ensures data consistency across multiple replicas, and allocation queries optimize read performance and reduce latency.
• Stateless metadata service3FS introduces a stateless metadata service based on a transactional key-value store (such as FoundationDB), which improves system scalability and reduces the complexity of metadata management.
• Direct I/O and RDMA optimizationDirect I/O allows direct access to the SSD, avoiding file caching, reducing CPU and memory overhead, and RDMA technology enables efficient data transfer, further improving performance.
• KVCache technologyIn inference tasks, KVCache caches key intermediate results, avoiding redundant computations and significantly improving inference efficiency. Combining high throughput and large capacity, KVCache is a low-cost alternative to DRAM caching.
• Data locality optimizationBased on optimized data layout and access patterns, it reduces data transmission latency and bandwidth consumption, and performs particularly well in large-scale distributed training and inference tasks.

3FS performance

• Large-scale read throughputIn a cluster consisting of 180 storage nodes, each node is equipped with 2 x 200Gbps InfiniBand network interface cards (NICs) and 16 x 14TiB NVMe SSDs. Approximately 500+ client nodes were used for read load testing, each configured with a 1 x 200Gbps InfiniBand NIC. Under the background traffic of the training job, the final aggregate read throughput reached approximately 6.6 TiB/s.
• GraySort Performance Test3FS performed exceptionally well in the GraySort benchmark test. GraySort is a large-scale data sorting test used to measure the data processing capabilities of distributed systems. The test cluster consisted of 25 storage nodes (each node with 2 NUMA domains, each domain with 1 storage service, and 2 x 400Gbps NICs) and 50 compute nodes (each node with 192 physical cores, 2.2 TiB of memory, and 1 x 200Gbps NIC). In this test, 3FS successfully sorted 110.5 TiB of data across 8192 partitions in just 30 minutes and 14 seconds, achieving an average throughput of 3.66 TiB/min.
• KVCache Inference OptimizationKVCache is a caching technology designed by 3FS to optimize the inference process of LLM (Large Language Model). It caches key vectors and value vectors in the decoding layer, avoiding redundant computations. In performance tests, KVCache achieved a peak read throughput of 40 GiB/s, significantly improving inference efficiency. KVCache's garbage collection (GC) operations also exhibit high IOPS performance, ensuring efficient cache management and updates.

3FS project address

• GitHub repository:https://github.com/deepseek-ai/3FS

3FS application scenarios

• Large-scale AI trainingIt efficiently supports the rapid reading and writing of massive amounts of data, improving training speed.
• Distributed data processingOptimizes data loading and management, supports random access, and eliminates the need for prefetching or shuffling.
• Inference optimizationBy caching intermediate results using KVCache, redundant calculations are reduced, and inference efficiency is improved.
• Checkpoint supportIt provides high-throughput parallel checkpointing capabilities to ensure the stability and recoverability of training tasks.
• Multi-node computing environmentSeamlessly integrates into large-scale clusters, supports flexible expansion, and meets the needs of AI applications of different sizes.