Titans - Google's new neural memory architecture

What are Titans?

Titans is a novel neural network architecture introduced by Google that overcomes the memory bottleneck of Transformers when processing long sequences of data. Titans introduces a neural long-term memory module, mimicking the human brain's memory mechanism to enhance the ability to remember unexpected events. The Titans architecture includes three variants: MAC (memory as context), MAG (memory as gates), and MAL (memory as layers), each integrating the memory module in different ways. Experiments show that Titans outperforms Transformers and modern linear RNN models in tasks such as language modeling, commonsense reasoning, and time series prediction, especially demonstrating superior performance in handling long sequences with over 2 million context windows. It also possesses parallel computing capabilities, improving training efficiency.

Titans' main functions

• Long sequence data processingTitans can effectively handle long sequence data with more than 2 million context windows. In long sequence tasks, it maintains high accuracy. For example, in the "needle in a haystack" task, even if the sequence length increases from 2k to 16k, the accuracy remains at around 90%.
• Memory ManagementBased on neural long-term memory modules, remembering information from long ago is crucial for tasks requiring long-term dependencies (such as language modeling and common-sense reasoning). It combines attention mechanisms to process short-term memory, focusing on direct dependencies within the current context.
• Task diversity:It performs well in a variety of tasks, including language modeling, commonsense reasoning, time series prediction, and genomics modeling, demonstrating broad applicability.
• Training efficiencyTitans' neural long-term memory module supports parallel computing, significantly improving training efficiency and enabling it to process large-scale data faster. During the inference phase, it quickly retrieves and utilizes long-term memory, improving the model's response speed.

Titans' technical principles

• Neural Long-Term Memory Module
  • Memory encodingThis approach leverages an online meta-model to learn how to remember and forget specific data during testing. The model encodes past information into the parameters of the neural network, avoiding the need to remember irrelevant details from the training data.
  • Surprise MeasurementDrawing inspiration from the principles of human brain memory, the "surprise level" of an input is determined by measuring its gradient. The larger the gradient, the more unexpected the input is, and the easier it is to remember.
  • Momentum mechanismIntroducing a momentum mechanism allows short-term surprises to accumulate into long-term memories, enabling the model to better process information flow in sequences.
  • Forgetting mechanismBased on the forgetting mechanism, the model can erase old memories that are no longer needed, prevent memory overflow, and manage limited memory capacity.
• Architecture Design
  • MAC (Memory as Context)The attention mechanism is fed together with long-term and persistent memory as context for the current input, allowing the model to consider both historical information and the current context simultaneously.
  • MAG (Memory as a Door)Gating fusion is implemented on two branches: the memory module and the sliding window attention. By combining the advantages of long-term and short-term memory, a gating mechanism is used to dynamically adjust the information flow.
  • MAL (Memory as Layer)The memory module is treated as a separate layer, and historical information is compressed before being input into the attention mechanism. This hierarchical information processing improves the model's expressive power.
• Parallel trainingOptimized based on matrix operations (matmuls), it supports parallel computing and significantly improves training efficiency.

Titans' project address

• arXiv technical paper:https://arxiv.org/pdf/2501.00663v1

Application scenarios of Titans

• Language Modeling and Text GenerationGenerate coherent, high-quality long texts, such as articles and stories, while maintaining content consistency and logic.
• Common Sense Reasoning and Question Answering SystemIt can understand and reason about complex questions in long contexts, provide accurate answers, and is suitable for question-answering tasks that require background knowledge.
• Time series forecastingIt can predict financial markets, weather changes, traffic flow, etc., capture long-term trends, and improve forecast accuracy.
• Genomics and BioinformaticsAnalyzing DNA sequences, predicting protein structures, processing long-sequence data in the biomedical field, and aiding in scientific discovery.
• Video and music processing: To understand and generate video content, create music, and maintain coherence and stylistic consistency in long sequences.