AB
AB
AiBoss

All Categories

project

Engram - A conditional memory module jointly developed by DeepSeek and Peking University

Engram, a conditional memory module developed by the DeepSeek team in collaboration with Peking University, enhances the performance of Large Language Models (LLMs). Engram stores static knowledge in dedicated memory by introducing a modern gram embedding table...

What is Engram?

Engram, a conditional memory module developed by the DeepSeek team in collaboration with Peking University, enhances the performance of Large Language Models (LLMs). By introducing a modern -gram embedding table, Engram stores static knowledge in independent memory, enabling efficient knowledge retrieval using sparse retrieval and context-aware gating techniques. Complementing Hybrid Expert Models (MoEs), Engram significantly improves model performance on knowledge-intensive tasks, long context processing, and reasoning capabilities without increasing computational cost. It also overcomes GPU memory limitations and supports large-scale parameter expansion.

Engram's main functions

  • Knowledge RetrievalIt enables quick retrieval of fixed knowledge through statically embedded tables, reducing reliance on computing resources.
  • Long context processingThis frees up the capacity of the attention mechanism, allowing it to focus on the global context and improve reasoning capabilities for long texts.
  • Decoupling computation and storageIt supports asynchronous prefetching of data from host memory, breaking through GPU memory limitations and enabling large-scale parameter expansion.
  • Performance improvementWithout increasing computational cost, it significantly improves the model's performance on knowledge-intensive tasks, general reasoning, coding, and mathematical tasks.

Engram's technical principles

  • Word segmenter compressionThe original Token ID is mapped to a normalized ID by a pre-computed surjective function, reducing vocabulary redundancy and improving the coverage efficiency of -gram.
  • Multi-head hash searchIt solves the problem of directly parameterizing -gram combinations by using hashing methods and multi-head mechanisms to map local contexts to embedded tables.
  • Context-aware gatingThe current hidden state is used as the dynamic query, and the retrieved embeddings are used as the key and value. Noise is suppressed through a gating mechanism to ensure semantic consistency.
  • Sparse allocationGiven a parameter budget, the MoE expert parameters are reallocated to the Engram memory to discover the U-shaped trade-off between computation and memory, and to find the optimal allocation ratio.
  • System ImplementationBy leveraging the deterministic retrieval characteristics of Engrams, computation and storage are decoupled, and data transmission latency is masked through prefetching and overlap mechanisms, thus overcoming the limitations of GPU memory.

Engram's project address

  • Technical Papers: https://github.com/deepseek-ai/Engram/blob/main/Engram_paper.pdf

Engram application scenarios

  • Knowledge Q&AEnggram can quickly retrieve factual information from the knowledge base, improving the response speed and accuracy of the question-answering system.
  • Text generationThrough efficient knowledge retrieval, Engram can quickly match fixed phrases and patterns in text generation, improving generation efficiency.
  • Long text processingEngram can optimize long context modeling and enhance the model's reasoning ability for long texts, making it suitable for long text analysis in fields such as law and academia.
  • Code assistanceEngram can quickly retrieve common patterns and structures in code, improving the efficiency of code generation, completion, and understanding.
  • Mathematical reasoningEngram can quickly retrieve mathematical formulas and theorems, assisting in solving mathematical problems and improving reasoning efficiency.
Back