xLLM - JD.com's open-source intelligent inference framework

What is xLLM?

xLLM is a high-efficiency intelligent inference framework open-sourced by JD.com, optimized for domestically produced chips and supporting integrated edge-cloud deployment. The framework uses a service-engine separation architecture, with the service layer responsible for request scheduling and fault tolerance, and the engine layer focusing on computational optimization, featuring multi-stream parallelism, graph fusion, and dynamic load balancing. xLLM supports various scenarios including large models, multimodal models, and generative recommendations, providing high-performance, low-cost inference services to facilitate the efficient implementation of intelligent customer service, real-time recommendations, content generation, and other businesses, and promoting the large-scale application of large language models on domestically produced chips.

Main functions of xLLM

• Full-map execution / multi-layer pipeline orchestrationBy implementing asynchronous decoupling scheduling at the framework layer, asynchronous parallel computation and communication at the model layer, and deep pipeline optimization at the operator kernel layer, multi-layer pipeline execution orchestration is achieved, reducing computational cavitation and improving overall inference efficiency.
• Graph execution optimization for dynamic ShapesDynamic size adaptation is achieved by using parameterization and multi-image caching methods. Combined with a managed memory pool and custom operator integration, the flexibility of static images is improved and the safe reuse of memory is ensured, thus optimizing the performance of dynamic input processing.
• MoE operator optimizationFor the MoE model, we optimize the GroupMatmul and ChunkedPrefill operators to improve computational efficiency and the ability to process long sequence inputs, thereby enhancing the model's inference performance.
• High-efficiency memory optimizationIt adopts a mapping management of discrete physical memory and contiguous virtual memory, allocates memory space on demand, intelligently schedules memory page reuse, reduces memory fragmentation and allocation latency, adapts to domestic chip operators, and improves the efficiency of video memory utilization.
• Global multi-level key-value cache managementThis enables intelligent offloading and prefetching of key-value pairs in a multi-level cache, constructs a distributed storage architecture centered on the key-value cache, optimizes intelligent transmission routing of key-value pairs between multiple nodes, and improves caching efficiency and data transmission performance.
• Algorithm optimizationBy optimizing speculative inference and using MoE expert dynamic load balancing, multi-core parallelism is achieved to improve efficiency, dynamically adjust expert distribution, optimize algorithm performance, and enhance inference throughput and load balancing capabilities.

How to use xLLM

• Environmental preparation:
  • Download imageChoose the appropriate Docker image based on the hardware device (e.g., A2, A3, etc.) and architecture (x86 or ARM). For example, for an A2 device (x86 architecture), you can download... xllm/xllm-ai:0.6.0-dev-hb-rc2-x86 Mirror. If the download fails, try an alternative source. quay.io/jd_xllm/xllm-ai:0.6.0-dev-hb-rc2-x86.
  • Create a containerWhen creating and starting a container, it is necessary to mount the required devices and directories to ensure that the container can access hardware resources and data. This includes device files (such as...). /dev/davinci0,/dev/davinci_manager (etc.), model file path, driver path, etc.
• Installation and compilation:
  • Install dependencies:
    • Cloning repositoryAfter entering the container, clone the official xLLM repository and initialize the submodules.
    • Configure vcpkgIf vcpkg is not pre-installed in the image, manually clone the vcpkg repository and set the environment variables. VCPKG_ROOT Points to the installation path of vcpkg.
    • Install Python dependenciesInstall the Python dependencies required for xLLM using Tsinghua University's Python mirror source, and upgrade them. setuptools and wheel.
  • Compilation:
    • Compile to generate executable file: Run the compilation command to generate the executable file. By default, the compilation target is the A2 device. If you need to compile for other devices (such as A3 or MLU), specify the device type by adding parameters.
    • Generate whl packageIf you need to generate a Python whl package, run the corresponding compilation command. The generated whl package will be saved in [location missing]. dist/ In the directory.
• Model loading:
  • Prepare model filesPrepare the model files and place them in a path accessible to the container, for example... /mnt/cfs/9n-das-admin/llm_models.
  • Loading ModelUse the interface provided by xLLM to load the model. After the model is loaded, inference tasks can be performed.
• Inference callThe inference interface provided by xLLM allows you to input text or other data to obtain inference results. The inference interface performs calculations based on the loaded model and returns the results.

xLLM project address

• Project official websitehttps://xllm.readthedocs.io/
• GitHub repositoryhttps://github.com/jd-opensource

Application scenarios of xLLM

• Intelligent Customer ServiceWe respond quickly to user inquiries, provide accurate answers and suggestions, and improve customer satisfaction and customer service efficiency.
• Real-time recommendationsBased on user behavior data, personalized recommendations are generated in real time to improve user engagement and conversion rates.
• Content generationxLLM can generate high-quality text content, such as news, articles, and creative copy, thus aiding content creation.
• Multimodal applicationsIt supports multimodal models (such as text + image) and can be used in scenarios such as image description generation and visual question answering.
• Generative RecommendationBy combining generative technologies, we can generate richer and more personalized recommendation results, thereby improving the user experience.