AB
AB
AiBoss

All Categories

project

UltraMem - A brand new ultra-sparse model architecture launched by ByteDance's Doubao Big Model Team.

UltraMem is a novel ultra-sparse model architecture proposed by ByteDance's Doubao Big Model team, addressing the high memory access costs associated with traditional MoE architectures during inference. By optimizing memory access and computational efficiency, the architecture significantly reduces inference costs...

What is UltraMem?

UltraMem is a novel ultra-sparse model architecture proposed by ByteDance's Doubao Big Model Team, addressing the high memory access costs of traditional MoE architectures during inference. By optimizing memory access and computational efficiency, the architecture significantly reduces inference costs, achieving inference speeds 2-6 times faster than MoE and cost reductions of up to 83%. UltraMem's core technologies include: improved multi-layer structure by splitting large memory layers into multiple smaller memory layers distributed across the Transformer layer, adding skip-layer operations for parallel computation; optimized value retrieval using Tucker Decomposition Query Key Retrieval (TDQKR) to improve retrieval accuracy; and implicitly extended sparse parameters (IVE), reducing GPU memory and deployment costs through the concepts of virtual and physical memory. Experiments show that UltraMem exhibits significant performance advantages across different scales of activation parameters, and its scalability surpasses MoE as the sparse parameters increase.

UltraMem's main functions

  • Reduce reasoning costsUltraMem significantly reduces memory access requirements during inference by optimizing memory access mechanisms, reducing inference costs by up to 83%.
  • Improve reasoning speedCompared to the traditional MoE architecture, UltraMem offers 2-6 times faster inference speed, and at common batch sizes, its memory access cost is almost comparable to that of a Dense model with the same computational load.
  • Optimize memory managementUltraMem reduces memory dependence during inference by employing sparse computation and selective parameter activation strategies, effectively avoiding memory bottlenecks.
  • Support large-scale modelsThis architecture opens up new paths for building value or expert models with billions of data points and has excellent scalability.
  • Maintain model performanceWith the same parameters and activation conditions, UltraMem significantly reduces inference costs and outperforms MoE in terms of model performance.

UltraMem's technical principles

  • Sparse computation and parameter decouplingUltraMem decouples computation and parameters through sparse computation. It activates only the parameters most relevant to the current task, rather than activating all experts during inference like traditional MoE, significantly reducing memory access requirements.
  • Optimized memory access mechanismUltraMem introduces a large-scale ultra-sparse memory layer, which avoids memory bottlenecks during inference by selectively activating a small number of parameters. This mechanism makes UltraMem's memory access cost almost comparable to that of a Dense model with the same computational cost at common batch sizes.
  • Parallel computing mechanismUltraMem enables multiple inference tasks to be performed simultaneously by optimizing the computation graph and employing advanced algorithms. Parallel computing mechanisms improve resource utilization and further accelerate inference speed.
  • Tucker Decomposed Query Key Retrieval (TDQKR)UltraMem employs a more sophisticated multiplication method—Tucker Decomposed Query Key Retrieval (TDQKR)—to optimize the value retrieval process. This method improves retrieval accuracy and efficiency by decomposing the interaction between the query and the key.
  • Implicitly Extended Sparse Parameters (IVE)UltraMem introduced the Implicit Extended Sparse Parameters (IVE) technique, which implicitly extends sparse parameters through the concepts of virtual memory and physical memory. This improves model performance and scalability without increasing the GPU memory burden.
  • Multi-layer structure designUltraMem breaks down large memory layers into multiple smaller memory layers, distributed at fixed intervals within the Transformer layers. This allows the model to perform memory access operations in the memory layers and computations in the Transformer layers in parallel, improving overall efficiency.

UltraMem's project address

Application scenarios of UltraMem

  • Real-time reasoning scenariosUltraMem is suitable for inference scenarios with high latency requirements, such as code completion and intelligent customer service. In these scenarios, the model needs to respond to user requests quickly, and UltraMem's low latency can significantly improve the user experience.
  • Large-scale model deploymentUltraMem's efficient inference capabilities and low memory access costs make it ideal for deploying large-scale language models (LLMs). When building value or expert models with billions of values, UltraMem can achieve high performance with limited computing resources.
  • Financial sectorIn the financial sector, UltraMem can be used for credit asset management and risk prediction. By more accurately identifying customer intent and risk characteristics, UltraMem can help financial institutions reduce costs and increase efficiency.
  • energy industryIn the energy sector, UltraMem can be applied to equipment operation and maintenance, electricity marketing and customer service, and renewable energy power forecasting. Through its efficient data processing and inference capabilities, UltraMem can optimize resource allocation and improve energy efficiency.
  • Industrial AutomationIn industrial automation, UltraMem can be used for equipment failure prediction and production process optimization. Its efficient inference capabilities can quickly analyze large amounts of industrial data, enabling intelligent production management.
Back